What evaluation criteria are used for decision support projects?
by Dan Power
One can categorize computerized decision support project evaluation concerns into 4 general criteria areas: 1) economic, 2) operational, 3) schedule, and 4) technical. In any computerized decision support project evaluation, these general criteria must be considered. Lets examine the 4 criteria in more detail.
These criteria are discussed by a number of authors in the Information Systems literature, including Whitten, Bentley and Barlow (1994). The following definitions and explanations are consistent with the general information systems usage:
Economic criterion a measure of the net benefits of a decision support project or solution. This is often called a cost-benefit analysis (CBA). Cost-benefit analysis is grounded in finance and accounting theory and terminology and closely tied to the budget process. Although CBA is difficult to use for decision-support, infrastructure, and strategic Information Technology projects, it is important to examine the economic impact of a proposed system. Project leaders and sponsors should identify both tangible and intangible costs and benefits.
Operational criterion a measure of how well the specific solution will work in the organization. It is also a measure of how people feel about the computerized decision support proposal. Applying this criterion involves reviewing assumptions about whether targeted users really need and want the proposed solution.
Schedule criterion a measure of the reasonableness of the project timetable. The estimates of project effort are often overly optimistic. An assessment of this criterion should focus on the consequences of the worst case schedule, especially negative consequences from schedule delays and the impact on project costs.
Technical criterion a measure of the practicality of a specific technical solution and the availability of technical resources and expertise. In some computerized decision support proposals technical issues are the major risk concern. The stability and maturity of the proposed technology should be carefully reviewed. Using emerging technologies for large scope, poorly structured projects increases risks, but such projects may have large strategic payoffs. Technology optimism is always a danger and IT managers should evaluate emerging technologies carefully.
Which of the criteria should be the focus at various project evaluation stages? The initial project evaluation should focus on the project need and the anticipated benefits. The focus should be on the economic and operational test. As the project evaluation continues more feasibility issues need to be evaluated and the costs and benefits need to be assessed more carefully to insure that project advocates are not inflating benefits and minimizing problems. The economic issues should be revisited a number of times, and all four criteria should be a major part of a feasibility analysis for a large scope, enterprise-wide project.
Decision support projects have various levels of risk associated with them. When DSS projects have ambiguous objectives and low structure, the projects have higher levels of risk because the costs and scope of work of the project are hard to define. The schedule and technical tests are very important for high risk projects. Also, because the objectives of the project are ambiguous, it can be difficult to assess the return on the investment. When economic returns are hard to assess more qualitative economic analyses should also be used.
DSS projects with a higher degree of structure and more clearly defined objectives generally are lower risk. More detailed planning is possible for projects with specific objectives. The economic criterion is easier to apply and also tests should be more concrete and detailed in the feasibility analysis. The size or scope of a DSS project in terms of the number of users served and the size of databases developed also impacts the risk of the assessed projects. Small DSS projects in terms of scope or dollar expenditures tend to be of lower risk than large projects. Finally, the sophistication of the technology and the experience of the developers using the technology influence the overall project risk. The ultimate decision to invest in a computerized decision support project should be based on an assessment of all four criteria and not solely on project risk. Innovative DSS projects that are most likely to result in a competitive advantage are often the riskiest project.
In general, evaluation activities and the application of the economic, operational, schedule and technical tests should be proportionate to the scope, complexity, and cost of a proposed DSS project. In narrow scope DSS projects that are highly structured, the amount of analysis and evaluation will often be limited, but as the project scope increases and the amount of structure is reduced for DSS projects, project risk increases and hence more frequent and more elaborate evaluation is needed. For large scope, low structure DSS projects, multiple detailed evaluations are probably needed and justified.
In all DSS evaluations, one needs to consider the longer term consequences and not solely immediate cost savings and decision process time improvements. DSS may reduce some costs, but that is not usually the motivating factor for a new system. No DSS project decision should be made in isolation from the overall IT project portfolio. Even small projects can sometimes have major business impacts. It is important to broadly examine DSS project impacts. Once a DSS project is completed managers need to follow-up and periodically evaluate what is working well with the system and why and what problems are being encountered.
As always, your comments, suggestions and feedback are welcomed.
Keen, Peter G. W. "Value Analysis: Justifying Decision Support Systems." MIS Quarterly. March, 1981. pp. 1-15.
Power, D. J. "Justifying a Data Warehouse Project", Part I and II. DS*, The On-Line Executive Journal for Data-Intensive Decision Support, February 3 and 10, 1998: Vol. 2, Nos. 5 and 6, URL http://dssresources.com/papers/justifydw/index.html .
Power, D. J. Decision Support Systems Hyperbook. Cedar Falls, IA: DSSResources.COM, HTML version, 2000, accessed on (today's date) at URL http://dssresources.com/subscriber/password/dssbookhypertext.
Whitten, Jeffrey L., L. D. Bentley, and V.M. Barlow. Systems Analysis and Design Methods (3rd ed.), Burr Ridge, IL: Irwin, 1994.
Last update: 2007-11-28 10:14
Author: Daniel Power
You cannot comment on this entry