What criteria should be used for evaluating DSS software packages?
Some people might be quick to list criteria like ease of use, cost of the package, capabilities, vendor reputation and ease of installation. Although such factors are important and need to be considered in most situations, this question should be framed more broadly and some other issues should be addressed before specific criteria are discussed.
First, it is important to determine that buying a package versus assembling and customizing software for a DSS project is the appropriate response. It is often hard to know where the dividing line is between "buying off-the-shelf" and "building" because once the customization for the DSS becomes significant then buying a package has been transformed into a development project (even though that may not have been intended). So it is important to recognize that DSS software selection is a sequential decision situation. Begin the process by specifying requirements and ask "What functions and tasks will managers perform with the DSS? When and how will it be used?"
Second, if the decision is to buy "off-the-shelf", then one must determine what products might meet the need. It is important to recognize that one must identify comparable software packages -- or they can't be compared. "Off-the-shelf" is often appropriate for task specific or vertical market DSS software like healthcare scheduling software, collaboration and groupware software, Web-based reporting software, enterprise portal software, data mining software and competitive intelligence software.
Third, once comparable products are identified then one can ask "which one is best in this particular situation?" At this point criteria can be specified and products can be compared. Evaluators need to recognize that dominant alternatives and dominating criteria exist in situations. Sometimes one criterion is so important in making a choice that all other criteria take on a secondary role. For example, the cost of the package may be so important that relatively high cost packages have no chance of being selected. In the same vein, some software packages may be so appropriate and be such a "good fit" with the perceived need that other packages receive little consideration. For example, a manager developing a small-scale, model-driven DSS may almost without explicit evaluation "choose" to develop the application in Microsoft Excel. Prescreen the list of possible DSS products to eliminate those that do not meet constraints like the need to "fit" with other software or with existing processes, or the need to meet special regulatory or legal requirements. Also, eliminate products that don't meet technical constraints in terms of operating systems or infrastructure.
Fourth, if a dominant alternative doesn't exist and if no one criterion dominates all others, then approximately five major criteria should be identified and weighted for evaluating the comparable DSS packages. Criteria should be generally independent of each other. Some criteria that should be considered and tailored to the situation include the following:
1. Capabilities - examine the functions that a DSS product can perform and how important they are to the decision support need of targeted users. Determine if the package can be customized and in what ways. Does it meet the need? Does it provide the desired support?
2. Cost of the Package - examine the total cost of ownership including acquisition costs, implementation and training costs, maintenance costs, and any annual software license costs.
3. Ease of use - the ease of learning and using the capabilities of a product to accomplish tasks. Ease of use is in the mind of the user so ask users to assess this criterion.
4. Ease of installation and operation - how easy is it to configure, deploy and control use of a product. Is it easy to transfer information to and/or from other company information systems? Are there potential technical implementation problems?
5. Performance - what is the speed or capacity of the product when performing its functions. Also, part of the performance criterion should be software reliability.
6. Vendor reputation and reliability - the vendor matters, but in emerging product areas this criterion can be difficult to assess. What kind of vendor and technical support is needed and is available?
One hopes that comparable packages will have similar benefits, but in some cases it is necessary to rate and compare the anticipated benefits.
Over the years, the most common task that I have helped with is software evaluation. In many of those situations, political and bureaucratic issues have impacted what happened. In evaluating new fundraising software, it was clear the technical people wanted "anyone other than the current vendor". In another situation, everyone wanted to upgrade capabilities, but it was clear the staff was psychologically "locked in" to their current vendor who was pushing a major new product. At a credit union, a group was evaluating data mining software, but cost was the dominant criterion. In many cases, a company is much better off to make no purchase than to be swayed solely or primarily by the cost of the product.
AVOID buying shelfware, software that gathers dust on a shelf instead of being used for its intended purpose. I have some shelfware that I bought with good intentions, but I have had no time to implement it. I also have some old "obsolete" versions of software that still meet my needs; I have had no real reason to upgrade or no funds to make a discretionary upgrade.
Recognize that vendors can create the perception of need and a vendor representative may even be willing to help identify the requirements. Also, recognize the impact of personal relationships with vendor representatives on software evaluations.
If you prepare a Request for Proposals (RFP) for a DSS project, it should focus on goals, capabilities and the vision for the project. Set up the RFP so it is easy to compare proposals. Get the same information from each vendor. Use the internet and sites like DSSResources.COM to identify vendors.
You may also want to read Power (1997) for more evaluation suggestions and my book titled Decision Support Systems: Concepts and Resources for Managers.
For a major project, you may want to hire a consultant to support your organization's software selection process. Also, it is often helpful to purchase evaluation reports from groups like Gartner and The OLAP Report. Try packages before you buy! Summarize your evaluation for each of the major alternative packages!!
One would like to think that evaluations of DSS software packages are primarily rational, technical decisions, but personal biases, company political and bureaucratic considerations can and often do impact software evaluations.
REMEMBER -- The ultimate responsibility for making a good software purchase decision rests with you, the buyer.
Power, D. J. "Tips for Choosing Enterprise-wide DSS Software". DS*Star, The On-Line Executive Journal for Data-Intensive Decision Support,
Power, D. J. Decision Support Systems: Concepts and Resources for Managers. Quorum Books, 2002.
The above response is from Power, D., What criteria should be used for evaluating DSS software packages? DSS News, Vol. 3, No. 8,
Last update: 2005-08-07 11:41
Author: Daniel Power
You cannot comment on this entry