The origins of today’s OLAP products

by Nigel Pendse, Principal of OLAP Solutions
and Co-author of the OLAPreport.com

OLAP products and applications have been around for much longer than most people think

Multidimensional analysis, the basis for OLAP, is not new. In fact, it goes back to 1962, with the publication of Ken Iverson’s book, A Programming Language. The first computer implementation of the APL language was in the late 1960s, by IBM. APL is a mathematically defined language with multidimensional variables and elegant, if rather abstract, processing operators. It was originally intended more as a way of defining multidimensional transformations than as a practical programming language, so it did not pay attention to mundane concepts like files and printers. In the interests of a succinct notation, the operators were Greek symbols. In fact, the resulting programs were so succinct that few could predict what an APL program would do. It became known as a ‘Write Only Language’ (WOL), because it was easier to rewrite a program that needed maintenance than to fix it.

Unfortunately, this was all long before the days of high resolution GUI screens and laser printers, so APL’s Greek symbols needed special screens, keyboards and printers. Later, English words were sometimes used as substitutes for the Greek operators, but purists took a dim view of this attempted popularization of their elite language. APL also devoured machine resources, partly because early APL systems were interpreted rather than being compiled. This was in the days of very costly, under-powered mainframes, so applications that did APL justice were slow to process and very expensive to run. APL also had a, perhaps undeserved, reputation for being particularly hungry for memory, as arrays were processed in RAM.

However, in spite of these inauspicious beginnings, APL did not go away. It was used in many 1970s and 1980s business applications that had similar functions to today’s OLAP systems. Indeed, IBM developed an entire mainframe operating system for APL, called VSPC, and some people regarded it as the personal productivity environment of choice long before the spreadsheet made an appearance. Even today, there are OLAP products that use APL internally.

However, APL was simply too elitist to catch on with a larger audience, even if the hardware problems were eventually to be solved or become irrelevant. It did make an appearance on PCs in the 1980s (and is still used, sometimes in a revamped form called “J”) but it ceased to have any market significance after about 1980. Although it was possible to program multidimensional applications using arrays in other languages, it was too hard for any but professional programmers to do so, and even technical end-users had to wait for a new generation of multidimensional products.

By 1970, a more application-oriented multidimensional product, with academic origins, had made its first appearance: Express. This, in a completely rewritten form and with a modern code-base, has become a widely used contemporary OLAP offering, but the original 1970’s concepts still lie just below the surface. Even after 30 years, Express remains one of the major OLAP technologies, although Oracle has struggled and failed to keep it up-to-date with the many newer client/server products. Oracle announced in late 2000 that it will build OLAP server capabilities into Oracle9i starting in mid 2001. The Oracle9i OLAP Services includes both a version of the Express engine, called the analytic workspace, and a new ROLAP engine. This will mean that the Express engine lives on well into its fourth decade, and perhaps even its fifth.

More multidimensional products appeared in the 1980s. Early in the decade, Stratagem appeared, and in its eventual guise of Acumate (now owned by Lucent), this too was still marketed to a limited extent till the mid 1990s. However, although it is a distant cousin of Express, it has never had Express’ market share, and is now little used. Along the way, Stratagem was owned by CA, which was later to acquire two ROLAPs, the former Prodea Beacon and the former Information Advantage DecisionSuite, both of which soon died.

Comshare’s System W was a different style of multidimensional product. Introduced in 1981, it was the first to have a hypercube approach and was much more oriented to end-user development of financial applications. It brought in many concepts that are still not widely adopted, like full non-procedural rules, full screen multidimensional viewing and data editing, automatic recalculation and (batch) integration with relational data. However, it too was heavy on hardware and was less programmable than the other products of its day, and so was less popular with IT professionals. It is also still used, but is no longer sold and no enhancements are likely. Although it is now available on Unix, it is not a client/server product and was never promoted by the vendor as an OLAP offering. In the late 1980s, Comshare’s DOS One-Up and later, Windows-based Commander Prism (now called Comshare Planning) products used similar concepts to the host-based System W. Hyperion Solution’s Essbase product, though not a direct descendant of System W, was also clearly influenced by its financially oriented, fully pre-calculated hypercube approach. Ironically, Comshare subsequently licensed Essbase (rather than using any of its own engines) for the engine in some of its modern OLAP products.

Another creative product of the early 1980s was Metaphor. This was aimed at marketing professionals in consumer goods companies. This too introduced many new concepts that only became popular in the 1990s, like client/server computing, multidimensional processing on relational data, workgroup processing and object oriented development. Unfortunately, the standard PC hardware of the day was not capable of delivering the response and human factors that Metaphor required, so the vendor was forced to create totally proprietary PCs and network technology. Subsequently, Metaphor struggled to get the product to work successfully on non-proprietary hardware and right to the end it never used a standard GUI.

Eventually, Metaphor formed a marketing alliance with IBM, which went to acquire the company. By mid 1994, IBM had decided to integrate Metaphor’s unique technology (renamed DIS) with future IBM technology and to disband the subsidiary, although customer protests led to the continuing support for the product. The product continues to be supported for its remaining loyal customers, and IBM relaunched it under the IDS name but hardly promoted it. However, Metaphor’s creative concepts have not gone and the former Information Advantage, Brio, Sagent, MicroStrategy and Gentia are examples of vendors covered in The OLAP Report that have obviously been influenced by it.

One other surviving Metaphor tradition is the unprofitability of independent ROLAP vendors: no ROLAP vendor has ever made a cumulative profit, as demonstrated by Metaphor, MicroStrategy, MineShare, WhiteLight, STG, IA and Prodea. The natural market for ROLAPs seems to be just too small, and the deployments too labor intensive, for there to be a sustainable business model.

By the mid 1980s, the term EIS (Executive Information System) had been born. The first explicit EIS product was Pilot’s Command Center though there had been EIS applications implemented by IRI and Comshare earlier in the decade. This was a cooperative processing product, an architecture that would now be called client/server. Because of the limited power of mid 1980s PCs, it was very server-centric, but that approach came back into fashion again with products like EUREKA:Strategy and Holos and the Web. Command Center is no longer on sale, but it introduced many concepts that are recognizable in today’s OLAP products, including automatic time series handling, multidimensional client/server processing and simplified human factors (suitable for touch screen or mouse use). Some of these concepts were re-implemented in Pilot’s Analysis Server product, which is now also at the end of its life, as is Pilot, which changed hands in August 2000, almost certainly for the last time, when it was bought by Accrue Software.

By the late 1980s, the spreadsheet was already becoming dominant in end-user analysis, so the first multidimensional spreadsheet appeared in the form of Compete. This was originally marketed as a very expensive specialist tool, but the vendor could not generate the volumes to stay in business, and Computer Associates acquired it, along with a number of other spreadsheet products including SuperCalc and 20/20. The main effect of CA’s acquisition of Compete was that the price was slashed, the copy protection removed and the product was heavily promoted. However, it was still not a success, a trend that was to be repeated with CA’s other OLAP acquisitions. For a few years, the old Compete was still occasionally found, bundled into a heavily discounted bargain pack. Later, Compete formed the basis for CA’s version 5 of SuperCalc, but the multidimensionality aspect of it was not promoted.

Lotus was the next to attempt to enter the multidimensional spreadsheet market with Improv. Bravely, this was launched on the NeXT machine. This at least guaranteed that it could not take sales away from 1-2-3, but when it was eventually ported to Windows, Excel was already too big a threat to 1-2-3 for Improv’s sales to make any difference. Lotus, like CA with Compete, moved Improv down market, but this was still not enough for market success, and new development was soon discontinued. It seems that personal computer users liked their spreadsheets to be supersets of the original 1-2-3, and were not interested in new multidimensional replacements if these were not also fully compatible with their old, macro driven worksheets. Also, the concept of a small multidimensional spreadsheet, sold as a personal productivity application, clearly does not fit in with the real business world. Microsoft went this way, by adding PivotTables to Excel. Although only a small minority of Excel users take advantage of the feature, this is probably the single most widely used multidimensional analysis capability in the world, simply because there are so many users of Excel. Excel 2000 includes a more sophisticated version of PivotTables, capable of acting as both a desktop OLAP, and as a client to Microsoft Analysis Services. However, the OLAP features in Excel 2000 are inferior to those in OLAP add-ins, so there is a good opportunity for third-party options as well.

By the late 1980s, Sinper had entered the multidimensional spreadsheet world, originally with a proprietary DOS spreadsheet, and then by linking to DOS 1-2-3. It entered the Windows era by turning its (then named) TM/1 product into a multidimensional back-end server for standard Excel and 1-2-3. Slightly later, Arbor did the same thing, although its new Essbase product could then only work in client/server mode, whereas Sinper’s could also work on a stand-alone PC. This approach to bringing multidimensionality to spreadsheet users has been far more popular with users. So much so, in fact, that traditional vendors of proprietary front-ends have been forced to follow suit, and products like Express, Holos, Gentia, MineShare, PowerPlay, MetaCube and WhiteLight now proudly offer highly integrated spreadsheet access to their application servers. Ironically, for its first six months, Microsoft OLAP Services was one of the few OLAP servers not to have a vendor-developed spreadsheet client, as Microsoft’s (very basic) offering only appeared in June 1999 in Excel 2000. However, the (then) OLAP@Work Excel add-in filled the gap, and still (under its new snappy name, BusinessQuery MD for Excel) provides much better exploitation of the server than does Microsoft’s own Excel interface.

A few users demanded multidimensional applications that were much too large to be handled in multidimensional databases, and the relational OLAP tools evolved to meet this need. These presented the usual multidimensional view to users, sometimes even including a spreadsheet front-end, even though all the data was stored in an RDBMS. These have a much higher cost per user, and lower performance than specialized multidimensional tools, but they are a way of providing this popular form of analysis even to data not stored in a multidimensional structure.

Other vendors expanded into what is now called desktop OLAP (even though, in Web implementations, the cubes are usually resident on the server): small cubes, generated from large databases, but downloaded to PCs for processing. These have proved very successful indeed, and the one vendor that sells both a relational query tool and a multidimensional analysis tool (Cognos, with Impromptu and PowerPlay) reports that the latter is much more popular with end-users than is the former.

Now, even the relational database vendors have embraced multidimensional analysis, with Oracle, IBM, Microsoft, Informix, CA and Sybase all developing or marketing products in this area. Ironically, having largely ignored multidimensionality for so many years, it now looks like Oracle, Microsoft and IBM might be the new ‘OLAP triad’, with large OLAP market shares, based on selling multidimensional products they did not invent.

So, what lessons can we draw from this 35-year history?

1. Multidimensionality is here to stay. Even hard to use, expensive, slow and elitist multidimensional products survive in limited niches; when these restrictions are removed, it booms. We are about to see the biggest-ever growth of multidimensional applications.

2. End-users will not give up their general-purpose spreadsheets. Even when accessing multidimensional databases, spreadsheets are the most popular client platform. Multidimensional spreadsheets are not successful unless they can provide full upwards compatibility with traditional spreadsheets, something that Improv and Compete failed to do.

3. Most people find it easy to use multidimensional applications, but building and maintaining them takes a particular aptitude — which has stopped them from becoming mass market products. But, using a combination of simplicity, pricing and bundling, Microsoft now seems determined to prove that it can make OLAP servers almost as widely used as relational databases.

4. Multidimensional applications are often quite large and are usually suitable for workgroups, rather than individuals. Although there is a role for pure single-user multidimensional products, the most successful installations are multi-user, client/server applications, with the bulk of the data downloaded from feeder systems once rather than many times. There usually needs to be some IT support for this, even if the application is driven by end-users.

5. Simple, cheap OLAP products are much more successful than powerful, complex, expensive products. Buyers generally opt for the lowest cost, simplest product that will meet most of their needs; if necessary, they often compromise their requirements. Projects using complex products also have a higher failure rate, probably because there is more opportunity for things to go wrong.

OLAP milestones

Year

Event

Comment

1962 Publication of A Programming Language by Ken Iverson First multidimensional language; used Greek symbols for operators. Became available on IBM mainframes in the late 1960s and still used to a limited extent today. APL would not count as a modern OLAP tool, but many of its ideas live on in today’s altogether less elitist products, and some applications (eg Adaytum e.Planning Analyst and Cognos Finance, the former Lex 2000) still use APL internally. The family connection continues with Ken Iverson’s son, Eric, now working for Adaytum Software, still in connection with APL and APL replacement projects.
1970 Express available on timesharing (followed by in-house versions later in the 1970s) First multidimensional tool aimed at marketing applications; now owned by Oracle, and still one of the market leaders (after several rewrites and two changes of ownership). Although the code is much changed, the concepts and the data model are not. The modern version of this engine is now shipping as the MOLAP engine in Oracle9i Release 2 OLAP Option.
1982 Comshare System W available on timesharing (and in-house mainframes the following year) First OLAP tool aimed at financial applications. No longer marketed, but still in limited use on IBM mainframes; its Windows descendent is marketed as the planning component of Comshare MPC. The later Essbase product used many of the same concepts, and like System W, suffers from database explosion.
1984 Metaphor launched First ROLAP. Sales of this Mac cousin were disappointing, partly because of proprietary hardware and high prices (the start-up cost for an eight-workstation system, including 72Mb file server, database server and software was $64,000). But, like the Mac, Metaphor users remained fiercely loyal.
1985 Pilot Command Center launched First client/server EIS style OLAP; used a time-series approach running on VAX servers and standard PCs as clients.
1990 Cognos PowerPlay launched This became both the first desktop and the first Windows OLAP and now leads the “desktop” sector. Though we still class this as a desktop OLAP on functional grounds, most customers now implement the much more scalable client/server and Web versions.
1991 IBM acquired Metaphor The first of many OLAP products to change hands. Metaphor became part of the doomed Apple-IBM Taligent joint venture and now lingers on as IDS, but there are few remaining sites.
1992 Essbase launched First well-marketed OLAP product, which went on to become the market leading OLAP server by 1997.
1993 Codd white paper coined the OLAP term This white paper, commissioned by Arbor Software, brought multidimensional analysis to the attention of many more people than ever before. However, the Codd OLAP rules were soon forgotten (unlike his influential and respected relational rules).
1994 MicroStrategy DSS Agent launched First ROLAP to do without a multidimensional engine, with almost all processing being performed by multi-pass SQL — an appropriate approach for very large databases, or those with very large dimensions, but suffers from a severe performance penalty. The modern MicroStrategy 7i has a more conventional three-tier hybrid OLAP architecture.
1995 Holos 4.0 released First hybrid OLAP, allowing a single application to access both relational and multidimensional databases simultaneously. Many other OLAP tools are now using this approach. Holos is now owned by Crystal Decisions, but is fading from the market.
1995 Oracle acquired Express First important OLAP takeover. Arguably, it was this event that put OLAP on the map, and it almost certainly triggered the entry of the other database vendors. Express has now become a hybrid OLAP and competes with both multidimensional and relational OLAP tools.
1996 BusinessObjects 4.0 launched First tool to provide seamless multidimensional and relational reporting from desktop cubes dynamically built from relational data. Early releases had problems, now largely resolved.
1997 Microsoft announced OLE DB for OLAP This project was code-named Tensor, and became the ‘industry standard’ OLAP API before even a single product supporting it shipped. Many third party products now support this API, which is evolving into the more modern XML for Analysis.
1998 IBM DB2 OLAP Server released This version of Essbase stored all data in a form of relational star schema, in DB2 or other relational databases, but it was more like a slow MOLAP than a scalable ROLAP. IBM later abandoned its “enhancements”, and now ships the standard version of Essbase as DB2 OLAP Server. Despite the name, it remains non-integrated with DB2.
1998 Hyperion Solutions formed Arbor and Hyperion Software ‘merged’ in the first large consolidation in the OLAP market. Despite the name, this was more of a takeover of Hyperion by Arbor than a merger, and was probably initiated by fears of Microsoft’s entry to the OLAP market. Like most other OLAP acquisitions, this went badly.
1999 Microsoft OLAP Services shipped

This project was code-named Plato and then named Decision Support Services in early pre-release versions, before being renamed OLAP Services on release. It used technology acquired from Panorama Software Systems in 1996. This soon became the OLAP server volume market leader through ease of deployment, sophisticated storage architecture (ROLAP/MOLAP/Hybrid), huge third-party support, low prices and the Microsoft marketing machine.

1999 CA starts mopping up failed OLAP servers CA acquired the former Prodea Beacon, via Platinum, in 1999 and renamed it DecisionBase. In 2000 it also acquired the former IA Eureka, via Sterling. This collection of failed OLAPs seems destined to grow, though the products are soon snuffed-out under CA’s hard-nosed ownership.
2000 Microsoft renames OLAP Services to Analysis Services Microsoft renamed the second release of its OLAP server for no good reason, thus confusing much of the market. Of course, many references to the two previous names remain within the product.
2000 XML for Analysis announced This initiative for a multi-vendor, cross-platform, XML-based OLAP API is led by Microsoft (later joined by Hyperion and then SAS Institute). It is, in effect, an XML implementation of OLE DB for OLAP.
2001 Oracle begins shipping the successor to Express Six years after acquiring Express, which has been in use since 1970, Oracle began shipping Oracle9i OLAP, expected eventually to succeed Express. However, the first release of the new generation Oracle OLAP was incomplete and unusable. The full replacement of the technology and applications is not expected until well into 2003, some eight years after Oracle acquired Express.
2001 MicroStrategy abandons Strategy.com Strategy.com was part of MicroStrategy’s grand strategy to become the next Microsoft. Instead, it very nearly bankrupted the company, which finally shut the subsidiary down in late 2001.
2002 Oracle ships integrated OLAP Oracle9i Release 2 OLAP Option shipped in mid 2002, with a MOLAP server (a modernized Express), called the Analytical Workspace, integrated within the database. This was the closest integration yet between a MOLAP server and an RDBMS. But it is still not a complete solution, lacking front-end tools and applications.

You can contact Nigel Pendse, the author of this paper, by e-mail at NigelP@OLAPReport.com if you have any comments or observations. Nigel Pendse is principal of OLAP Solutions and has had over twenty-five years experience as a user, vendor and independent consultant in the areas now known as OLAP and Business Intelligence. He is a frequent speaker at conferences, and as a consultant, advises both users and vendors on OLAP product issues. He has degrees in mechanical and nuclear engineering, and was international marketing director of a well-known software firm until 1994. He is based in London, England.

Nigel Pendse provided permission on behalf of OLAPReport.com to archive this article and feature it at DSSResources.COM on Saturday, 20 July 2002. This article, "The origins of today’s OLAP products", was posted at DSSResources.COM on October 6, 2002. All information copyright 2002, Business Intelligence Ltd, all rights reserved. This version was last updated on July 20, 2002. The current version should be online at http://olapreport.com/origins.htm.