What is the difference between formative and summative DSS evaluation?
by Dan Power
More evaluation is needed for decision support, analytics and business intelligence projects. Developers should evaluate and get feedback from intended users as the project progresses and evaluation should continue after the capability is deployed and in use. More evaluation leads to greater use of the systems and capabilities and should lead to better systems.
In Fall 2002, Babajide Adefarati asked about conducting formative and summative evaluations of model-driven DSS. In general, I had associated the terms with course development and program/project reviews more than with building decision support systems (DSS), but the terms do provide some useful distinctions that should be considered in planning the evaluation of decision support projects.
A quick Google search in 2003 indicated formative and summative evaluations were used in some papers related to software usability and testing. Most of the hits were related to uses in education and social services. In 2016, a similar search shows a broader use of the terms. My conclusion is that a formative evaluation would occur during DSS design and development and that a summative evaluation would occur once the development project is completed and the decision support system is in use. Some authors associate formative evaluation with evaluations by users and summative evaluations with expert and managerial evaluations.
There are a number of different approaches to software evaluation that differ based on when the evaluation occurs, either during the development process or when the project is complete, the intentions of the evaluator(s), providing a formative constructive evaluation versus obtaining a judgmental summative evaluation, and who does the evaluation, internal or external evaluators. Managers and developers should confirm how formative and summative are used when you are in a discussion about evaluating a system. I'll suggest the following definitions.
A formative evaluation involves judging the current status or worth of a program/project, activity or software system while development activities are occurring. Formative evaluation focuses on intermediate or preliminary outcomes assessed by intended users during the development process. The evaluation is intended to be constructive, rather than judgmental.
A summative evaluation involves judging the ultimate or final worth of a program/project, activity or software system at the end of the development process and following implementation. The focus is on assessing intermediate and longer term outcomes and results. Summative evaluation may lead to improvements in the application.
In my opinion, potential users should provide the primary feedback for the formative evaluation of a decision support capability and the evaluation criteria should primarily focus on the user interface and usability issues. As part of a formative evaluation of a model-driven DSS, the model needs to reviewed and validated by an "expert". Formative evaluation of a knowledge-driven DSS needs to verify the rules and the content of the knowledge-base. Examining data and document quality are legitimate issues in the formative evaluation of data or document-driven systems.
For a large-scale model-driven DSS project, a summative evaluation should include assessments by both users and expert evaluators. Criteria should be broader and the impact of the decision support capability on decision making and the organization should be assessed.
For both formative and summative evaluations one can collect four main types of data using a variety of data collection methods:
1. Impressionistic or subjective data from developers, users or potential users of the DSS.
2. Objective data from an unbiased observer. In most situations the observer will use an explicit, structured assessment protocol.
3. Qualitative data in text, audio or video format. The data may include answers from potential users to open-ended questions, or anecdotal or impressionistic comments from an observer or a developer. Based upon my own experiences in formative evaluation situations, videotapes of user interactions with a DSS prototype can be especially helpful.
4. Quantitative data is used, but some DSS developers seem to favor anecdotal evidence. Quantitative data should be collected about the use of a DSS. The data may be collected by the decision support software, in a user questionnaire, or from numerical scores given by observers.
As the above discussion suggests, a comprehensive evaluation of a DSS may include collecting all four types of data. We generally expect that qualitative data is more likely to be subjective or impressionistic. Also, we can collect and interpret both quantitative and qualitative objective data. We can collect data using questionnaires and expert reviewers, by videotaping one-on-one interaction between a user and an evaluator, and by using a small group of observers. In both formative and summative evaluation, data from users and potential users whether perceived as objective or subjective is likely to have a major impact on the evaluation conclusions. It seems that the key is to create a positive, constructive feedback loop in formative evaluation. If the evaluation suggests the DSS can not be built, then managers need to act quickly to end the project. A positive approach to evaluation can result in ending a DSS project or in discontinuing use of a DSS.
A number of Web pages credit Robert Stakes with the following quote "When the cook tastes the soup, that's formative; when the guests taste the soup, that's summative." I haven't found a citation for Stakes' quote, but it's interesting and worth repeating. As always your comments, feedback and questions are most welcome.
Northwest Regional Educational Laboratory, Program Evaluation, http://www.nwrel.org.
Phillips, B., Social Research: Strategy and Tactics (3rd Edition),
The above response is from Power, D., What is the difference between formative and summative DSS evaluation? DSS News, Vol. 4, No. 2,
Last update: 2016-01-27 01:50
Author: Daniel Power
You cannot comment on this entry