DSS Project Evaluation Process
Managers evaluate many types of projects, but technology projects are usually the most difficult to evaluate. Evaluating DSS projects is especially difficult and it must be an on-going process for large-scale projects. Evaluation activities should be commensurate or proportionate to the scope, complexity, and cost of a proposed DSS project. Project sponsors and project managers must decide what amount and type of evaluation is appropriate and necessary in their company’s Information Technology management environment.
A discussion of the process of evaluating DSS projects generates many questions. For example, when should DSS projects be evaluated? Should we examine in-house capabilities when we evaluate DSS projects? When does vendor evaluation occur? Once we have a project team, what is their role in DSS project evaluation? Who should do the evaluation? Is a feasibility analysis always needed? How many go-no go opportunities do managers have? Most managers can add to this list. I will try to offer suggestions about some of the most commonly asked questions.
Figure 12.1 portrays the evaluation process as a multi-stage cycle of development and evaluation. The scale of the project and the development approach determines what activities will actually occur and in what sequence. Evaluation should be performed periodically from the initial idea stage to the final post-implementation project evaluation. A DSS project can be revised or even canceled at any stage. The resources that have been expended on the project are "sunk" costs. One should not continue a bad project because of the money and resources that have already been spent on it. Managers need to know when to stop spending money and divert funds to more feasible projects. Managers also need to know when despite setbacks it is desirable to continue an important project.
Figure 12.1 – On-Going DSS Project Evaluation Process
First, there are several possible times and ways to evaluate a DSS project. At different times in the development of a DSS, a different type of evaluation may be needed. An initial DSS idea often needs nurturing. So at the initial idea stage we probably want a positive, developmental evaluation. At some point in the design stage for a DSS project a feasibility study needs to be completed. Even small-scale end user built DSS projects need to be evaluated. Often a feasibility study actually improves the understanding of a proposed Decision Support System. The extent of the feasibility study should be a function of the scope of a proposed DSS and the type of proposed DSS. A feasibility study for a Web-Based, Data-Driven DSS should be much more extensive than a study for a small-scale Model-Driven DSS on a single personal computer. What should be included in a data warehouse feasibility analysis?
Managers should conduct some type of evaluation of a large-scale DSS project at each step in the Systems Development Life Cycle (SDLC) or after each major change in a prototype. Prior to implementation an enterprise-wide or inter-organizational DSS must be carefully evaluated. Managers should not hesitate to delay implementation if problems are encountered. A DSS project will fail dramatically if problems are encountered doing the initial roll-out for the DSS. A DSS is usually best introduced in stages. The initial user group is critical to the overall long-term success of the project. After implementation, DSS should be regularly reviewed and evaluated. Technologies and user needs change and a process should be in place to insure that an obsolete DSS is not hurting decision-making rather than enhancing it.
We should always examine in-house capabilities when we evaluate DSS projects. In general, DSS projects should be implemented by in-house Information Systems staff. We want to enhance expertise with DSS tools among company employees and in general DSS applications should be treated as key business capabilities.
DSS are built using software development tools and so vendor evaluation will be part of many innovative and large-scale DSS. When does vendor evaluation occur? Usually vendors are selected once a feasibility analysis is completed. Vendor capabilities and software should be considered in a feasibility study.
Once we have a project team, what is their role in DSS project evaluation? Who should do the evaluation? Is a feasibility analysis always needed? How many go-no go opportunities do I have? The DSS project team has the major role in evaluating DSS projects. The team should do the feasibility study and managers need to recognize that a major DSS project should be evaluated regularly. A feasibility study is needed, but it may be very limited in the topics addressed (see Chapter 4). Canceling a project is always difficult, but managers can avoid this by actively managing DSS projects and by critically evaluating the feasibility of proposed projects.
Large-scale DSS projects can be expensive. A data warehouse project can cost $1 million to $2 million and take 1 to 3 years to complete. Business benefits can however be extensive. An IDC study of 62 firms found an average 3-year ROI of 401% and the payback period can be very short.
Large-scale DSS can become a key strategic weapon for a company. Project evaluation helps increase the chance of success and cost effective implementation. Recall our goal is to provide managers with the right information, in the right format, at the right time and at the right cost. Let's now examine some tools that can help evaluate DSS projects.