Some of the earliest roots of business intelligence technology can be found in relatively simple reporting and analysis technologies. Today’s BI, on the surface, also resembles the elegant, but ultimately inefficient executive information systems of the mid- to late-1980s. But the practice of BI, and particularly the “best practice” of BI that is starting to emerge in leading-edge organizations now, relies on a much more comprehensive and flexible portfolio of software tools for data, process, and metadata management, data extraction, data quality, integration, analysis, modeling, forecasting and visualization.
Eric Rogge, BI research director at Ventana Research, San Mateo, Calif., puts it this way, “BI is fundamentally a data distribution technology today, but it is not just historical data. It includes strategic plans and predictive insights.”
Increasingly, the BI technology portfolio also includes prepackaged analytical applications and sophisticated application development capabilities — though commodity products like Excel spreadsheets remain important elements of the BI solution set because of their ubiquity and ease of use. However, Rogge notes that while “we all benefited dramatically from the desktop revolution, and it gave us independence, it also led to permutations of the truth. The challenge now for organizations is creating a centrally managed system that has consistency so that we have trust, and we can avoid fraud.”
Furthermore, best-practice BI practitioners not only use their portfolio of BI tools and solutions to monitor business operations and business events and to measure performance, they also leverage their BI investment to drive business process changes. Certainly monitoring and measurement improve understanding and add insight, but understanding and insight without change or reaction has limited value.
Optimization of those better-understood processes and alignment of revised processes with core strategic initiatives, is where best practice BI is evolving. Fortunately the technologies that are available today to support corporate decision making are far more flexible than what was available earlier, making the re-mapping of monitoring and measuring software practical.
In an analysis published in 2004, Framingham, Mass.-based International Data Corp. estimated that the worldwide market for business analytics software (which includes both BI tools and BI analytical applications) for 2004 would be approximately $14.35 billion. Of that total, IDC estimated that approximately 70% would consist of BI tools and the remaining 30% would be made up of BI analytical applications. Moving into 2005, however, as IDC digests final 2004 financial results, early indications are that the market actually was slightly stronger than expected. In a recent conversation with Software Magazine, Dan Vesset, an IDC analyst focused on the BI market, indicated that 2004 results might come in at closer to $14.7 billion — with the same relative contribution between BI tools and analytical applications.
Significantly, Vesset also noted, “Adoption is increasing faster than the dollar denominated market sizing would suggest.” Competition and pricing pressure is driving unit costs down, but demand for the output of BI systems appears to be growing at a faster rate. In fact, even with pricing pressures in place, the IDC analysts are considering a modest upward revision to their projected dollar denominated market growth rates for the segment, though it is too soon to tell.
Either way, from a market standpoint, something does appear to be different — and better — than expected. The question is, “What are the drivers?” One perspective is that the nature of demand in the BI market is changing due to a combination of rising competitive pressures, corporate concerns about risk management and liability, and a range of government or regulatory mandates such as Sarbanes-Oxley.
IDC’s Vesset believes that a general need for information, which includes both a compliance-oriented element and a competitive element, are the primary demand-side drivers for BI investment. On the supply side, the entry of large enterprise application and infrastructure software suppliers — such as Microsoft with SQL Server Reporting Services — is contributing to market expansion.
“The BI systems that are in place are imperfect by definition,” observes Ventana’s Rogge. “BI systems are alive…Since business problems evolve over time, BI systems have to be constantly refreshed and updated over time, and the infrastructures purchased several years ago may no longer be adequate.”
Awareness of the gap between what companies have and what they need in their BI technology portfolio is helping drive the market. Customers continue to upgrade to and extend early Web-based BI investments they’ve made, and add new analytical functionality or applications, as well as retire older, difficult-to-maintain homegrown BI applications.
Stamford, Conn.-based Gartner characterizes the drivers this way: First, Gartner believes enterprise applications are creating a virtual tsunami of operational data that corporations are struggling to interpret and understand. Historically, management’s efforts to leverage that flow of data have been frustrated by a combination of inflexible technology and a lack of corporate consensus on what should be measured and how.
Second, Gartner notes that management’s focus on performance management is growing, as well as its awareness of the need to develop and measure a corporation’s key performance indicators.
Finally, regulatory reporting requirements are intensifying management’s sense of the gap between what their current BI implementation can deliver and the more accurate, timely and consistent data that they need to maintain compliance.
Fragmented Market Moves Toward Consolidation
The market for BI tools is still fragmented — and many BI customers still have isolated silos of BI information and analysis. On both the customer and vendor side, there is pressure to reduce that fragmentation. “The BI world is shifting. Instead of buying three or four or five best-of-breed products and stitching them all together, now customers are really trying to consolidate,” says Mark Murphy, managing director and analyst at First Albany Capital in San Francisco.
With customers increasingly focused on consolidation, market dynamics are starting to shift. And those changing dynamics have some significant ramifications for buyers and sellers alike. “Enterprises selecting business intelligence solutions have become more concerned with ease-of-integration above all other factors,” says Doug Laney, founder and CEO of Evalubase in San Diego. “We believe this is due to the variety of data sources these solutions must plug into, combined with the increasing need for BI solutions to be embedded into existing business applications,” he says. (See Fig. 1.)
“Preliminary findings [as of April 11, 2005] from Evalubase’s ongoing study of the enterprise technology market show that within the data warehouse segment (including data integration, data quality and business intelligence), solutions from Actuate, SAS, Firstlogic and Ascential have an early lead over those from other vendors,” Laney notes.
Given that the market for BI products includes a collection of software tools for data integration, analysis, delivery and management, as well as a set of analytical applications, it isn’t surprising that integration is challenging at many levels. From a technology point of view, “Previous attempts to drive widespread adoption of BI tools both inside and outside the firewall failed due to Web infrastructures not being able to perform and scale, and the lack of user/developer skills,” says Bill Hostmann, vice president and analyst at Gartner. “Today the Web infrastructure can scale and perform but the user/developer skills are still lacking.”
Better information in the hands of the right people at the right time and in the right form is still the ultimate goal of BI practitioners, but implementation remains challenging. “BI needs to be an integral part of the business processes that [include] measuring and providing insight, ” says Gartner’s Hostmann. (See Fig. 2.)
The problem in the past was a classic catch-22. BI is designed to help managers gain insight, allow them to make better decisions about what to change and when and where to make changes. However, if management makes changes to the underlying business process or to the performance indicators being monitored or measured, IT might be unable to keep up with the requirements to modify its BI infrastructure. Without those changes, BI will fall out of synch with operational systems, and the company might find itself right back where it started — at least as far as its BI infrastructure goes. In that scenario, why bother with BI at all?
Significant consolidation going on in the BI vendor community as well. “We had one wave of consolidation … I expect at least one of the top 10 BI vendors will be acquired,” notes IDC’s Vesset. “But for every acquisition, a start-up pops up to take its place.” Vesset notes that there are some interesting innovations occurring in the areas of data visualization and data integration, as well as growing interest in real-time analysis.
But even with consolidation and the entrance of large enterprise application suppliers such as Microsoft, Oracle, SAP, Siebel and others, there will likely be room for BI “pure-plays.”
“The majority of organizations have heterogeneous applications, and as long as they have heterogeneous applications they need BI tools that can support those multiple applications and data sources,” says Gartner’s Hostmann. Typically, Gartner sees about a third of the companies interested in BI building their own applications, a third buying BI applications and the final third using a combination of both.
One way to approach implementation is to adopt a framework-based method. The framework, according to Gartner’s Hostmann, not only includes specific technology elements, but also specifies a set of methods and defines specific business objectives. Collectively, the technologies for building BI applications make up the BI platform, and BI suites, or applications, represent the delivery layer for BI content. “The platform is something an application developer is going to use to build a BI application,” Hostmann says. “It will provide the needed analytical functions. A platform may also provide data models and have some specific data management functionality.”
As of November 2004, when it last updated its “Magic Quadrant” for BI platform providers, Gartner had not identified any vendor as a market leader, though several vendors were positioned on the borders of the challenger/leader or visionary/leader boundaries. Microsoft and SAS Institute fall in the former category and Hyperion Solutions falls in the latter. (See Fig. 3.) Looking at the Magic Quadrant for enterprise BI suites however, Gartner finds several leaders, including Business Objects, Cognos and Information Builders.
BI as a Business Process
Technology improvements have certainly helped BI practitioners get a better grip on the challenges businesses face today — and in helping BI transform itself from an analytical technology focused on historical information to becoming inseparable from the processes of running a business itself.
On the technology front, Ventana’s Rogge notes several evolutionary improvements. Web delivery of information was first and foremost the biggest improvement, the ability to manipulate and organize more data more rapidly was another, and the hard work vendors have done to create better metadata structures was yet another. “There are more and more refined abstraction layers — of which metadata is one,” he says. Better user interfaces for manipulation of information, greater functionality — such as on-the-fly aggregation, sorting, time series calculations and the ability to perform large-scale deployments for lots of users with richer visualization — all make BI a more critical element of better business performance than it was in the past.
Gartner also notes that the most successful BI practitioners have established BI competency centers to manage and support BI initiatives in their firms. Citing a survey of about 900 BI buyers, Hostmann indicates that approximately 30% of the responding organizations — generally Global 2000-class size entities — have such a center in place. “Typically, we see 38% to 39% saying they have or plan to have [a BI competency center] in 12 months,” he said. “But some 50% to 60% say they don’t have one and don’t plan to have one.” (See Fig. 4.)
Sometimes the barrier to adoption or implementation is more cultural or political than technological. Hostmann observes that there are some key differences in how best practice BI practitioners treat the information they get from their BI systems. “BI becomes a core part of the underlying business process,” he says. “Once you get the right set of insights, you can recognize some opportunities to re-org.”
“The interesting thing these days is that BI is serving business operations more than ever before,” says Ventana’s Rogge. “We’re enabling fully operational departments, but not just for decision making. BI is for managing by exception. We are inundated by data, and the real challenge is sifting through it all.”
But taking BI beyond decision making to “alignment” or management by exception, as desirable as it may be from a strategic standpoint, initiates yet another set of changes in both operational and analytical systems. To continue to benefit from the BI investment, companies have to re-map their BI systems to the new processes they define and implement on the operational side. And the cycle is never-ending.
In fact, that view of the BI investment dictates a major change in how BI ought to be approached. Instead of treating BI as a project, with a defined beginning and end, it becomes, in effect, a process itself, and one that is inseparable from the act of running a business effectively.
IDC’s Vesset concurs, noting that barriers to BI success are “usually nontechnical.” In one sense the lack of common ground, the lack of a common language or definitions between and among the various BI constituencies — including IT, business unit management, finance, and so on — become insurmountable hurdles. “How do we define key performance indicators, measure, or share data, or go through the politics of an organization?” are some of the key questions Vesset identifies. “There is a gap between IT and the business community, and there is room to improve IT productivity or to improve flexibility for end users.”
