What staffing is needed for DW/BI/DSS?
by Dan Power
Some Ask Dan! questions are more difficult for me to answer than others. On July 6, 2006, Aileen MacKay sent me some especially hard questions and she followed up with some additional questions on September 29, 2006. Over the past 4 months, I've done some searching and thinking trying to figure out how to answer her questions. In this Ask Dan!, I'll focus only on her staffing questions.
Aileen writes "I would like to find out more information on staffing and skill levels for adequately maintaining a Data Warehouse and a DSS. What staffing level are most companies maintaining to effectively manage their Data Warehouse and DSS? Are they flexing using consultants? or do they have a core DW Team? What is the ideal staffing level? What is the "best practice" organizational structure that is used today?
Someone has likely conducted a survey to answer these questions quantitatively. Once that data was collected, it was probably grouped by the size of the responding organization and possibly by how long the organization had had a data warehouse and perhaps by some measure of scope like the number of data marts or the amount of data stored in the warehouse. All of these situational factors impact staffing. In searching the Web and trade publications however, I haven't found the results of that survey.
What did I find? Info-Tech Research Group (ITRG) recently surveyed 1,600 IT decision-makers about their IT budget and staffing patterns. ITRG prepared reports (June 2006) for nine industries. One 2006 report costs $990, "each additional industry report may be purchased for just $500." These reports are probably too general to answer Aileen's questions and the cost is a barrier. Foote Partners has a Quarterly Salary and Skills Pay Report for Data Warehousing/Business Intelligence for 20 selected U.S. cities. The job titles included in the report are: Vice President/Director – Data Warehousing/BI; Data Warehouse/BI Project Manager – Processing; Data Warehouse/BI Project Manager – Decision Support Services; Data Warehouse/BI Information Security Manager; Data Warehouse/BI Architect; Data Warehouse/BI Engineer; Data Warehouse/BI Developer; Data Warehouse/BI Auditor; Sr. Decision Support System(DSS) Analyst; Sr. BI Analyst; Sr. DSS/BI Engineer; Data Warehouse/BI Administrator; and Data Warehouse/BI Management Specialist. Again this report won't answer Aileen's questions and the $800 cost is a barrier. It is interesting to examine the job titles included in the report.
So based on case studies, anecdotes and my past readings, I'll give my opinions. If you are familiar with a survey related to these questions, please send me an email.
My DSS book (Power, 2002) notes "A complex DSS built using either an SDLC or a prototyping approach requires a team development approach. Once the system is developed a group may also be needed to maintain the system. Some large-scale DSS are built with teams of 2-3 people or with a larger group of 10 or more. ... The composition of the DSS team will change over the development cycle so the project manager needs to provide direction and motivation for the DSS team."
The data warehousing project that I am most familiar with was at Ertl (Power and Roth, 2003). In 1996 "the Ertl IS organization consisted of 23 employees, including six full time developers, three contract programmers, three managers of systems development, one database administrator, one business analyst, one manager of end user computing, two help desk specialists, and one network specialist. At the peak of data warehouse development efforts 21 contract programmers had been employed." Ertl is best classified as a medium-sized manufacturing organization.
I'm familiar with data warehouse projects that involved a single internal staff person and projects that have been completely outsourced. At an insurance company, a data warehouse consultant was hired to develop a feasibility prototype for the senior actuary. He designed a star schema, established dimensions and an ETL interface. He used Cognos Enterprise PowerPlay to provide the BI mechanism for the Actuary. At a publishing and direct mail company an internal staff person built a financial data warehouse for corporate management in about 9 months. Many small-scale decision support projects can be completed by a Senior Decision Support System Analyst or Senior BI Analyst.
Let's look at some examples from cases at DSSResources.com. Peter Barton (2003) reported the George Washington University data warehouse project team "consists of 1 full-time project manager and 5 project team members." The Iowa Department of Revenue data warehouse was initially built beginning in 1999 with extensive help from Teradata Professional Services staff. In March 2003, both the Web-based Audit Support Component and the Business Objects interface were completed. In 2004 after expanding its technical services staff, "one Teradata Professional Services staff member works full time alongside the revenue department staff, writing queries and ensuring the logical data model is fully operational" (Teradata Staff, 2004). Anissa Stevens, AVANCO International, worked with a small consulting team to build a data mart for Redland Genstar (cf., Stevens, 2004).
Recently, Computer Sciences Corp. received a two year, $21.5 million contract to develop the Blue Health Intelligence data warehouse for the Blue Cross and Blue Shield Association. "CSC is designing, developing, deploying and hosting the warehouse. It will store clinical records for 20 Blue Plan operations and 79 million people, and is expandable to house records for 100 million people. The data warehouse is being tested and implemented this year, and plans to be fully operational in 2007." Check the August 4, 2006 release at DSSResources.com.
What staffing level are most companies maintaining to effectively manage their Data Warehouse and data-driven DSS? Once a data-driven DSS has been built, at a minimum an organization needs a Data Warehouse/BI Management Specialist. It is possible a data base administrator with responsibility for a transaction system, can maintain the actual data warehouse, but someone needs to be concerned with extracting and loading additional data. This "bare bones" approach probably won't lead to any new capabilities. My best guess is that a staff of 3-5 people is a realistic minimum for maintaining any significant data mart/data warehouse/BI/data-driven DSS program.
Are they flexing using consultants? or do they have a core DW Team? In general, once a data-driven DSS capability has been built, I think it is best to rely on a core team of permanent staff. Finding the right people may be difficult, so training and internal reassignment may be the best way to create a more permanent DW core team. The Iowa Department of Revenue case shows that external professional services staff from vendors or consulting groups may be involved for many years. If one checks the resumes of major data warehouse consultants, the tasks they perform varies tremendously from project to project. For example, Sid Adelman is a well know data warehouse writer and consultant. He worked with a large bank "in the development and implementation of a large marketing data mart." Sid helped in architecting, planning and staffing. He assisted in developing a scope agreement that documented the function, schedule and responsibilities to be delivered in each phase of implementation. He helped develop a metadata strategy. He assisted in the evaluation and selection of a RDBMS and of query tools. Also, Sid worked with the bank to create a capacity plan for a multi-terabyte data warehouse." Someone in an organization must have the knowledge to perform these tasks to build a sophisticated data warehouse. The knowledge needed to maintain the warehouse is much less and quite different.
What is the "best practice" organizational structure that is used today? Lou Agosta in a Forrester/GIGA report in 2003 concluded "The common organizational structures for data warehousing staffing are centralized, decentralized, project-based and cross-functional. Giga knows of examples of successful data warehousing development and operations with each one of these types." I think that once large-scale, enterprise-wide data warehouses have been built, the technology system should be maintained and enhanced by a centralized unit in the IT organization with a Vice President/Director of Data Warehousing/BI/DSS. The organization structure for decision support is an important issue that must be addressed. A centralized structure for the decision support group can lead to problems in a large, decentralized organization. A project-based structure works well for development, but not as well for ongoing operation and enhancement of the decision support capabilities. Finally, a cross-functional structure may reduce some of the political problems, but I think that the dual reporting relationships and "turf" issues create an inherent long-run instability for this structure.
In a recent column (DSS News, August 13, 2006), I discussed organizing for BI in a University setting. In that column, I argued that "how a DSS task group is organized depends upon the maturity of current data-driven DSS applications and upon how much maintenance they require and the aspirations for developing novel DSS applications in performance monitoring, budgeting or enrollment forecasting."
In general, staffing for data-driven DSS should depend upon the magnitude of the business problem you are trying to solve. Structure of a decision support unit should reflect its functions and the other structures in the organization. As always, your comments and feedback are appreciated.
Agosta, L., "Organizational Design for Data Warehousing and Data Mart Support," Jan. 7, 2003, http://www.forrester.com .
Barton, P., "The George Washington University Data-Driven Decision Support Project", 2003, posted at DSSResources.COM August 15, 2003.
Computer Sciences Corporation developing IT data warehouse for Blue Cross and Blue Shield Association, 08/04/2006, DSSResources.com, URL http://www.dssresources.com/news/1680.php .
Power, D.J., Decision Support Systems: Concepts and Resources for Managers, Westport, CT: Quorum/Greenwood, 2002.
Power, D.J., "What is the process for designing decision support software?" DSS News, Vol. 7, No. 17, August 13, 2006.
Stevens, A., "Implementing the Redland Genstar Data Mart", posted at DSSResources.COM July 2, 2004.
Teradata Staff, "Closing the Tax Gap in Iowa", posted at DSSResources.COM February 21, 2004.
Last update: 2006-10-22 17:10
Author: Daniel Power
You cannot comment on this entry