What staffing is needed for DW/BI/DSS?
by Dan Power
Editor, DSSResources.com
Some Ask Dan! questions are more difficult for me to answer than
others. On July 6, 2006, Aileen MacKay sent me some especially hard
questions and she followed up with some additional questions on
September 29, 2006. Over the past 4 months, I've done some searching
and thinking trying to figure out how to answer her questions. In
this Ask Dan!, I'll focus only on her staffing questions.
Aileen writes "I would like to find out more information on staffing
and skill levels for adequately maintaining a Data Warehouse and a
DSS. What staffing level are most companies maintaining to
effectively manage their Data Warehouse and DSS? Are they flexing
using consultants? or do they have a core DW Team? What is the ideal
staffing level? What is the "best practice" organizational structure
that is used today?
Someone has likely conducted a survey to answer these questions
quantitatively. Once that data was collected, it was probably grouped
by the size of the responding organization and possibly by how long
the organization had had a data warehouse and perhaps by some measure
of scope like the number of data marts or the amount of data stored in
the warehouse. All of these situational factors impact staffing. In
searching the Web and trade publications however, I haven't found the
results of that survey.
What did I find? Info-Tech Research Group (ITRG) recently surveyed
1,600 IT decision-makers about their IT budget and staffing patterns.
ITRG prepared reports (June 2006) for nine industries. One 2006 report
costs $990, "each additional industry report may be purchased for just
$500." These reports are probably too general to answer Aileen's
questions and the cost is a barrier. Foote Partners has a Quarterly
Salary and Skills Pay Report for Data Warehousing/Business
Intelligence for 20 selected U.S. cities. The job titles included in
the report are: Vice President/Director – Data Warehousing/BI;
Data Warehouse/BI Project Manager – Processing; Data
Warehouse/BI Project Manager – Decision Support Services; Data
Warehouse/BI Information Security Manager; Data Warehouse/BI
Architect; Data Warehouse/BI Engineer; Data Warehouse/BI Developer;
Data Warehouse/BI Auditor; Sr. Decision Support System(DSS) Analyst;
Sr. BI Analyst; Sr. DSS/BI Engineer; Data Warehouse/BI Administrator;
and Data Warehouse/BI Management Specialist. Again this report won't
answer Aileen's questions and the $800 cost is a barrier. It is
interesting to examine the job titles included in the report.
So based on case studies, anecdotes and my past readings, I'll give
my opinions. If you are familiar with a survey related to these
questions, please send me an email.
My DSS book (Power, 2002) notes "A complex DSS built using either an
SDLC or a prototyping approach requires a team development approach.
Once the system is developed a group may also be needed to maintain
the system. Some large-scale DSS are built with teams of 2-3 people
or with a larger group of 10 or more. ... The composition of the DSS
team will change over the development cycle so the project manager
needs to provide direction and motivation for the DSS team."
The data warehousing project that I am most familiar with was at Ertl
(Power and Roth, 2003). In 1996 "the Ertl IS organization consisted of
23 employees, including six full time developers, three contract
programmers, three managers of systems development, one database
administrator, one business analyst, one manager of end user
computing, two help desk specialists, and one network specialist. At
the peak of data warehouse development efforts 21 contract
programmers had been employed." Ertl is best classified as a
medium-sized manufacturing organization.
I'm familiar with data warehouse projects that involved a single
internal staff person and projects that have been completely
outsourced. At an insurance company, a data warehouse consultant was
hired to develop a feasibility prototype for the senior actuary. He
designed a star schema, established dimensions and an ETL interface.
He used Cognos Enterprise PowerPlay to provide the BI mechanism for
the Actuary. At a publishing and direct mail company an internal
staff person built a financial data warehouse for corporate
management in about 9 months. Many small-scale decision support
projects can be completed by a Senior Decision Support System Analyst
or Senior BI Analyst.
Let's look at some examples from cases at DSSResources.com. Peter
Barton (2003) reported the George Washington University data
warehouse project team "consists of 1 full-time project manager and 5
project team members." The Iowa Department of Revenue data warehouse
was initially built beginning in 1999 with extensive help from
Teradata Professional Services staff. In March 2003, both the
Web-based Audit Support Component and the Business Objects interface
were completed. In 2004 after expanding its technical services staff,
"one Teradata Professional Services staff member works full time
alongside the revenue department staff, writing queries and ensuring
the logical data model is fully operational" (Teradata Staff, 2004).
Anissa Stevens, AVANCO International, worked with a small consulting
team to build a data mart for Redland Genstar (cf., Stevens, 2004).
Recently, Computer Sciences Corp. received a two year, $21.5 million
contract to develop the Blue Health Intelligence data warehouse for
the Blue Cross and Blue Shield Association. "CSC is designing,
developing, deploying and hosting the warehouse. It will store
clinical records for 20 Blue Plan operations and 79 million people,
and is expandable to house records for 100 million people. The data
warehouse is being tested and implemented this year, and plans to be
fully operational in 2007." Check the August 4, 2006 release at
DSSResources.com.
What staffing level are most companies maintaining to effectively
manage their Data Warehouse and data-driven DSS? Once a data-driven
DSS has been built, at a minimum an organization needs a Data
Warehouse/BI Management Specialist. It is possible a data base
administrator with responsibility for a transaction system, can
maintain the actual data warehouse, but someone needs to be concerned
with extracting and loading additional data. This "bare bones"
approach probably won't lead to any new capabilities. My best guess
is that a staff of 3-5 people is a realistic minimum for maintaining
any significant data mart/data warehouse/BI/data-driven DSS program.
Are they flexing using consultants? or do they have a core DW Team?
In general, once a data-driven DSS capability has been built, I think
it is best to rely on a core team of permanent staff. Finding the
right people may be difficult, so training and internal reassignment
may be the best way to create a more permanent DW core team. The Iowa
Department of Revenue case shows that external professional services
staff from vendors or consulting groups may be involved for many
years. If one checks the resumes of major data warehouse consultants,
the tasks they perform varies tremendously from project to project.
For example, Sid Adelman is a well know data warehouse writer and
consultant. He worked with a large bank "in the development and
implementation of a large marketing data mart." Sid helped in
architecting, planning and staffing. He assisted in developing a
scope agreement that documented the function, schedule and
responsibilities to be delivered in each phase of implementation. He
helped develop a metadata strategy. He assisted in the evaluation and
selection of a RDBMS and of query tools. Also, Sid worked with the
bank to create a capacity plan for a multi-terabyte data warehouse."
Someone in an organization must have the knowledge to perform these
tasks to build a sophisticated data warehouse. The knowledge needed
to maintain the warehouse is much less and quite different.
What is the "best practice" organizational structure that is used
today? Lou Agosta in a Forrester/GIGA report in 2003 concluded "The
common organizational structures for data warehousing staffing are
centralized, decentralized, project-based and cross-functional. Giga
knows of examples of successful data warehousing development and
operations with each one of these types." I think that once
large-scale, enterprise-wide data warehouses have been built, the
technology system should be maintained and enhanced by a centralized
unit in the IT organization with a Vice President/Director of Data
Warehousing/BI/DSS. The organization structure for decision support
is an important issue that must be addressed. A centralized structure
for the decision support group can lead to problems in a large,
decentralized organization. A project-based structure works well for
development, but not as well for ongoing operation and enhancement of
the decision support capabilities. Finally, a cross-functional
structure may reduce some of the political problems, but I think that
the dual reporting relationships and "turf" issues create an inherent
long-run instability for this structure.
In a recent column (DSS News, August 13, 2006), I discussed
organizing for BI in a University setting. In that column, I argued
that "how a DSS task group is organized depends upon the maturity of
current data-driven DSS applications and upon how much maintenance
they require and the aspirations for developing novel DSS
applications in performance monitoring, budgeting or enrollment
forecasting."
In general, staffing for data-driven DSS should depend upon the
magnitude of the business problem you are trying to solve. Structure
of a decision support unit should reflect its functions and the other
structures in the organization. As always, your comments and feedback
are appreciated.
References
Agosta, L., "Organizational Design for Data Warehousing and Data Mart
Support," Jan. 7, 2003, http://www.forrester.com .
Barton, P., "The George Washington University Data-Driven Decision
Support Project", 2003, posted at DSSResources.COM August 15, 2003.
Computer Sciences Corporation developing IT data warehouse for Blue
Cross and Blue Shield Association, 08/04/2006, DSSResources.com, URL
http://www.dssresources.com/news/1680.php .
Power, D.J., Decision Support Systems: Concepts and Resources for
Managers, Westport, CT: Quorum/Greenwood, 2002.
Power, D.J., "What is the process for designing decision support
software?" DSS News, Vol. 7, No. 17, August 13, 2006.
Stevens, A., "Implementing the Redland Genstar Data Mart", posted at
DSSResources.COM July 2, 2004.
Teradata Staff, "Closing the Tax Gap in Iowa", posted at
DSSResources.COM February 21, 2004.
Last update: 2006-10-22 17:10
Author: Daniel Power
Print this record
Show this as PDF file
You cannot comment on this entry
