Data warehousing projects are frequently side-tracked or derailed completely by non-technical factors, in particular the political treaty lines within the firm, and the politicized nature of data itself. Because data warehouses are infrastructure for sociotechnical systems (STSs) within the firm, politics and the exercise of power are inherent in data warehousing projects, and data warehouse designers have to adopt work practices and methods from non-technical disciplines, think of themselves in new ways, and employ some fairly sophisticated qualitative sociological methods in order to optimize the chances for successful deployment of data warehouses.
Where Are The Failed Data Warehousing Projects?
The signal-to-noise ratio in the data warehousing market is the lowest it has ever been. As more and more vendors enter the marketplace, as more and more professional conference-making companies create more and more professional conferences, as the number of articles that cross our desks each week climbs and climbs, I canít help feeling that we are missing something in the loud and often pointless public discourse on data warehousing: something fairly important. Where are the failed data warehousing projects?
I believe -- although I have only my own experience as a consultant, and quiet talk with some of my peers, to support this belief -- that the data warehousing marketplace woefully underreports both actual cases of failed data warehouse projects, and normative and prescriptive information for warehousing practitioners on techniques for avoiding project failure.
Why actual cases of failure are underreported will be obvious to anyone whoís ever written or approved a capital appropriation request for a million dollars: misspending that much money is not something you talk about with PC Week if you intend to remain employed or get another job in the IT business.
Whatís not so obvious to me is why the normative or prescriptive information about failure-avoidance in data warehousing projects is so scanty. If we mark the founding of the data warehousing discipline from Inmonís seminal definition in 1990 (and surely the discipline is older than that by any practical standard, particularly if we give credit to Commander and CommandCenter as the progenitors of OLAP), we have at least a decade of collective experience in the discipline, multiplied by, say, 10,000 practitioners worldwide, each working a nominal 8 hour day for 220 days: 176,000,000 person-hours of experience. That ought to be more that enough collective guild practice for some clear prescriptive models for failure avoidance to have emerged by now.
Yet the information in the IT trade press -- our disciplinary best-practices exchange mechanism -- is almost nonexistent. For example, using a common CD-ROM based data base of about 70,000 IT journal articles from 110 periodicals covering the period July 1995 to July 1996, the query:
returned less than 100 articles, while the query
returned 279 articles, and the query
returned 1283 articles.
Assuming I wasnít being tricked by the disclosure of failure avoidance tips under the headings of "how to succeed" articles, I looked into the 100 articles from the first query, and found some of what I was looking for. Most articles contained throw-away, unimplementable advice:
Such incisive analysis...
Some of the articles were more substantive, however. Larry Greenfield, one of the more astute observers on the data warehousing scene, lists these ten warnings:
All true, I think, and all important considerations. But articles of this type were the exception, not the norm, which looked more like this:
"A data warehouse is a complex undertaking." Thereís rocket science-class analysis for you. "Create a metadata model" -- you can guess what this consultant is selling as the cure-all for data warehousing. All in all, we donít seem to be telling one another very much about what kinds of complexity weíre dealing with, why data warehousing projects fail, and what we can do, as practitioners, to prevent project failures.
Modeling Data Warehousing Project Failure
It seems to me that all the advice in the trade press, all my personal experience, and what I know of other practitionersí experiences, leads to a pretty simple four-category model of warehousing project failures. Data warehousing, data marting and other kinds of decision support systems (DSS) projects fail because of:
Of these four categories, only the last -- the sociotechnical factors -- are significantly new for most IT organizations. Design factors have always been in play in complex IT projects; data warehousing introduces new design models and disciplines, but doesnít change the fundamental playing field. Similarly, technology factors have always played a role in IT project success and failure; all data warehousing projects do is aggravate this historical problem area by (a) increasing the number of separately-purchased components involved in the project and (b) raise significantly the integration burden, in terms of development and testing activities, imposed on the IS organization. Procedural factors are also familiar territory, though the presence of a large data warehouse platform on the data center floor frequently means new database technologies, different backup and recovery processes, and sometimes foreign systems management and tuning tools and processes.
But the sociotechnical factors are largely new; this area was pretty much submerged in classical OLTP design. People and politics were less of an issue, or no issue at all. The target user community held little organizational power, was rarely seriously consulted during design, and could in most cases be compelled to use the system once it was deployed (though there are those interestingly complex unionized situations). And the projectís sponsor was almost invariably after a rock-solid, measurable business objective: process or task routinization, cost containment, workforce reduction.
Itís this last area -- people and politics -- that Iím really interested in, because the most painful, indirect and ultimately revelatory discussions I have with my clients are centered around the politics of data warehousing. Sometimes these conversations begin under the heading of "engaging with the business unit"; other times they are framed by the question "How do we cost-justify a data warehouse?" or "How do we get management buy-in on a data warehouse?" Infrequently, they are frank enough that other, more central and more overtly political topics surface: how do we get the business unitsí IT organizations (or end-user communities) to trust us?
I have had these conversations quite often in the last couple of years, particularly when my firm is rebuilding one of our customersí data warehousing projects after another firm has cratered the customerís project and beat a hasty retreat. Although I canít prove it, I am convinced that the most common set of factors contributing to data warehousing project failure are not design factors or technology factors or procedural factors, but sociotechnical factors: people and politics.
Talking About Politics
If we were honest with ourselves, as professionals, we would admit what Rosabeth Moss Kanter suggested in 1979 in a famous Harvard Business Review article: that
Power is also the last dirty word in data warehousing. And itís crippling the discipline, in my view. Anyone whoís actually done a warehousing project knows very well that most of their organizational energy was spent dealing with political issues of a few particular sorts, and yet the query
against the article database I mentioned earlier returned only 29 articles from the database described above. The vast majority of these articles are notable for the vacuity of their comments on the political factors in data warehousing, resorting to pat formulae like:
and this gem, found in a value-added reseller trade publication in an article on how "hot" the data warehousing market is for VARs:
Apparently at least one vendor community recognizes the sociotechnical factors in the market, if only as an obstacle to quick and easy revenue.
A couple of well-put, pointed articles did emerge, particularly two by Julia Vowler. But there was no systematic discussion of the politics of data warehousing to be found anywhere in the almost 1300 articles in the database.
This fact, and a couple of recent conversations with clients, led me to ask myself a few questions:
Politics In Data Warehousing
Data warehousing projects are always potentially political because:
Politics, Part One: Treaty Line Violations
Any sensible data warehouse design is a part of a larger architectural model designed to deliver data from the points of capture (inside or outside the firm) to the points of use, probably transforming the data elements in the process. A warehouse, in other words, is the key data consolidation and pumping station in a complex data distribution system that begins with the firmís production applications and external data syndicates, wholesalers and enrichers of various sorts, and ends on the intelligent desktops of managers, analysts, customer care personnel and the like. That network always crosses treaty lines: invisible boundaries within the firm that mark both "turf" and "domains of control".
Some of these treaty lines are functional (with the inevitable tensions that are an in-built feature of the functional organization as an organizational form), but the most insidious treaty lines are almost never functional. Consider, for example, the near-invisible treaty line that is drawn across the backplane of every PC in the modern corporation. Much like the residence jack in the American telephone system, the desktop treaty line marks a boundary between the "provider" and the "consumer" and, just as we are free to choose our telephones, PC users feel themselves imperatively to be in rightful possession of personal computers , the tool configuration of which is a private affair. When a warehousing project mandates toolsets or impinges on the desktop in other ways, the treaty line is crossed, and warning bells often ring out.
Politics, Part Two: Data Ownership and Data Access
If there is any rule that does apply across organizations regardless of their market focus or structure, it is this: power accrues to those who:
The irony in data warehouse projects is that, all too often, these laws are working against the very organization they used to work for: the corporate IS organization. As often as not, the data we canít get for a data warehouse is the data controlled by semi-autonomous divisional or departmental IS functions that came into being because the historically stingy policies of the corporate IS organization with respect to data access hamstrung a division, department or business unit so badly that they built their own IS function to regain the autonomy they needed for marketplace effectiveness. The dangers of centralized data control constitute, at some fundamental level, the raison díetre of too many distributed IS organizations, and as a result their willingness to collaborate with the adversary is minimal at best.
But the real political problem with data warehousing is not the loss of data ownership that such projects imply, for every organization asked to contribute to the warehouse, a loss of control over access to the raw data itself, something frightening for any group with:
which, in my experience, is pretty much every commercial organization operating today. Inasmuch as divisions, departments and business units have gently cooked their dirty, ambiguous and unflattering data for years in the interest of keeping things clear and simple (and in the legitimate interest of not allowing bad or ambivalent data to get in the way of clarity about the real state of the business at whatever level), these organizations are understandably leery about exposing the uncooked data to inspection by other groups that may have a vested interest or historical reason to point out the data quality issues.
Politics, Part Three: Work Practice Integration
We are comfortable with the notion that production and service workers are obliged, by the terms and conditions of their employment, to submit to the discipline of information technology. When I was an accounts receivable data entry clerk for IBM in the early 1980s, I did my work as the system told me to do it: I stepped through tasks the IT presented to me, in the order it presented them to me, on a 327x terminal screen, for six hours a day, every day. That was my job: to be disciplined by the machine.
As Iíve walked up the hierarchy over the last fifteen years, from production worker to service worker to knowledge worker, my IT has backed off, and then come back not as a disciplining force, but a facilitating force. That is so in part because the firm is not permitted, by the terms and conditions of employment, to inspect or discipline the work practices of knowledge workers. Attempts to do this Ė even relatively benign ones such as TQM process definition initiatives or restatements of the obvious fact that electronic mail is company property Ė are met with resistance on a grand scale, muttering about invasion of privacy, and the magic word: "professionalism." Professionals, it seems, are above needing external discipline: because they practice a profession, whatever discipline they need with respect to what they do and how they do it is supplied by the profession, not by IT.
One of the things knowledge workers do, early and often, in a mostly unscrutinized way, is make decisions. Small and large, important and inconsequential, decisions get made every day about all kinds of things. And the plain fact seems to be that most of these decisions are data-free or data-poor, made based on "experience" or "gut feel" or some other intangible, unmeasurable quality that is decidely human, decidely part of these magic professional disciplines and decidely not something a computer can frame, direct or do (see Treaty Lines above).
The work practice of decision-making has been done historically outside the IT infrastructure of the firm. Data warehousing projects threaten this long-standing practice. And they create, in knowledge workers, what Thorsten Veblen called, in another context,"the conscious withdrawal of efficiency": passive-aggressive behavior on the part of knowledge worker communities that includes
Sensing The Political: 10 Warning Signs
How can a project team, comprised mostly of IS personnel, know before they start that their data warehousing project is, or is likely to become, politicized? While your mileage may vary, there are common, clear signs that your project is or will become politicized:
Mastering The Political: 10 Countermeasures
Getting Into Politics
Information technology is, for better or for worse, social these days. The good old days of batch and online transaction processing systems design and deployment are gone; we buy those things now, from independent software vendors. The systems we have to build Ė decision support systems, computer-supported collaborative work environments, workflow systems, intranets, extranets, whathaveyou-nets Ė are all deeply and inextricably social applications of IT: computing applied to groups of people with power, status and a network of relationships.
That means, for better or for worse, that politics is an integral part of IT projects from here on in. Or out, depending on your perspective. And that, in turn, means IT professionals have no choice but to get into organizational politics, understand the forms, shapes and paths organizational politics takes, and become astute at navigating in a political environment. Not because politics is cool, or fun, but because politics is a feature of the landscape: the beast standing between us and the gate marked "successful project conclusion."
About the Author
Marc Demarest is the CEO of Noumenal, Inc., a private consultancy focused on business turn-around and go-to-market services for intellectual property-based companies in technical and commercial markets. Prior to Noumenal, Marc was Chairman and CEO of DecisionPoint Applications, Inc., the industry's leading provider of packaged data warehousing software for financial, human resources, manufacturing and distribution analysis. Marc has also held executive positions with The Sales Consultancy, Inc., a high-technology sales and marketing consulting firm based in Dallas, Texas and the United Kingdom, and Sequent Computer Systems, Inc. in Portland, Oregon, where he led the development of worldwide sales and marketing process development and created an APQC award-winning knowledge management technology environment. Widely known as the father of data marting, Marc publishes and speaks regularly on topics ranging from knowledge management and decision support systems to IT technology futures and information ethics. He holds BA degrees in Political Science and English Literature from Bucknell University in Lewisburg, PA., an MA from the University of South Carolina, and is a graduate of the Stanford Institute's high-technology business management program.
Demarest, M., "The Politics of Data Warehousing", DSSResources.COM, 07/23/2004.
Marc Demarest provided permission to archive articles from the Noumenal website at DSSResources.COM on Friday, December 5, 2003 and he provided permission to archive and feature this specific article on Friday, July 9, 2004. This article was posted at DSSResources.COM on July 23, 2004.