DSS News
                     D. J. Power, Editor
            January 19, 2003 -- Vol. 4, No. 2
         A Bi-Weekly Publication of DSSResources.COM


 Check the article by Brobst and Rarey at DSSResources.COM



 * Ask Dan! - What is the difference between formative and 
              summative DSS evaluation?
 * AMCIS 2003 Call for Papers, Tampa, FL Aug. 4-6, 2003
 * What's New at DSSResources.COM
 * DSS News Releases


Subscribe to DSSResources.COM. One month $10, six months $25. 


Ask Dan!
by Daniel J. Power

What is the difference between formative and summative DSS evaluation? 

In Fall 2002, Babajide Adefarati asked a question on the Subscriber Zone 
Bulletin Board about conducting formative and summative evaluations of 
model-driven DSS. This Ask Dan! expands on and summarizes my responses 
to his question. In general, I have associated the terms formative and 
summative evaluation with course development and program/project reviews 
more than with building DSS, but the terms do provide some useful 
distinctions that should be considered in planning the evaluation of 
decision support projects. My perception is that more evaluation is 
needed of DSS projects. 

A quick Google search indicates the terms are used in some papers 
related to software usability and testing. Most of the hits were related 
to education and social services. My conclusion is that a formative 
evaluation would occur during DSS design and development and that a 
summative evaluation would occur once the development project is 
completed and the decision support system is in use. Some authors 
associate formative evaluation with evaluations by users and summative 
evaluations with expert and managerial evaluations. There are a number 
of different approaches to evaluation that differ based on when the 
evaluation occurs, either during the development process or when the 
project is complete, the intentions of the evaluator, providing a 
formative constructive evaluation versus obtaining a judgmental 
summative evaluation, and who does the evaluation, internal or external 
evaluators. You'll want to confirm how formative and summative are used 
when you are in a discussion about evaluating a DSS. I'll suggest the 
following definitions based upon what I have read.

A formative evaluation involves judging the worth of a program/project, 
activity or software system while development activities are occurring. 
Formative evaluation focuses on intermediate or preliminary outcomes and 
results during the development process.

A summative evaluation involves judging the worth of a program/project, 
activity or software system at the end of the development process and 
following implementation. The focus is on assessing immediate and longer 
term outcomes and results.

In my opinion, potential users should provide the primary feedback for 
the formative evaluation of a Decision Support System and the evaluation 
criteria should primarily focus on user interface and usability issues. 
As part of a formative evaluation of a model-driven DSS the model needs 
to reviewed and validated by an "expert". Formative evaluation of a 
knowledge-driven DSS needs to verify the rules and knowledge-base. 
Examining data and document quality are legitimate issues in the 
formative evaluation of data or document-driven DSS. 

For a large-scale model-driven DSS project, summative evaluation should 
include assessments by users and expert evaluators. Criteria should be 
broader and the impact of the model-driven DSS on decision making and 
the organization should be assessed.

For both formative and summative evaluations one can collect four main 
types of data using a variety of data collection methods:

1. Impressionistic or subjective data from developers, users or 
potential users of the DSS.

2. Objective data from an unbiased observer. In most situations the 
observer will use an explicit, structured assessment protocol.

3. Qualitative data in text, audio or video format. The data may include 
answers from potential users to open-ended questions, or anecdotal or 
impresionistic comments from an observer or a developer. Based upon my 
own experiences in formative evaluation situations, videotapes of user 
interactions with a DSS prototype can be especially helpful.

4. Quantitative data is used, but some DSS developers seem to favor 
anecdotal evidence. Quantitative data should be collected about the use 
of a DSS.  The data may be collected by the decision support software, in a 
user questionnaire, or from numerical scores given by observers.

As the above discussion suggests, a comprehensive evaluation of a DSS 
may include collecting all four types of data. We generally expect that 
qualitative data is more likely to be subjective or impressionistic. 
Also, we can collect and interpret both quantitative and qualitative 
objective data. We can collect data using questionnaires and expert 
reviewers, by videotaping one-on-one interaction between a user and an 
evaluator, and by using a small group of observers. In both formative 
and summative evaluation, data from users and potential users whether 
perceived as objective or subjective is likely to have a major impact on 
the evaluation conclusions. It seems that the key is to create a 
positive, constructive feedback loop in formative evaluation.  If the 
evaluation suggests the DSS can not be built, then managers need to act 
quickly to end the project. A positive approach to evaluation can result 
in ending a DSS project or in discontinuing use of a DSS.

A number of Web pages credit Robert Stakes with the following quote 
"When the cook tastes the soup, that's formative; when the guests taste 
the soup, that's summative." I haven't found a citation for Stakes' 
quote, but it's interesting and worth repeating. As always your 
comments, feedback and questions are most welcome.


LinguaLinksĀ®, SIL International,

Northwest Regional Educational Laboratory, Program Evaluation,

Phillips, B., Social Research: Strategy and Tactics (3rd Edition), New 
York, NY: Macmillan, 1976.

 Tell your friends! Get DSS NEWS free -- send a blank email 

AMCIS 2003 Call for Papers, Tampa, FL Aug. 4-6, 2003

The 2003 Americas Conference on Information Systems will be 
held in Tampa, Florida from August 4-6, 2003.  The list of
mini-tracks for the conference is now available on the Web.  
The AIS Special Interest Group on Decision Support, Knowledge 
and Data Management Systems (SIG DSS) is sponsoring the 
meta-track on Decision Support and Knowledge Management. Karen 
Dowling, an IS faculty member at Arizona State, is the SIG DSS 
Program Chair and she is coordinating the meta-track. Her 
email address is Contact her or a 
mini-track chair with questions.

The Decision Support and Knowledge Management meta-track includes
five mini-tracks: Decision Support Systems, Databases for the 21st 
Century, Data Warehousing, Geographic and Location-Based Technologies 
for Decision Support, and Knowledge Management. Information on all of 
the AMCIS 2003 tracks can be found at .

Please try to submit abstracts for screening and feedback to
an appropriate mini-track chair by February 17. Final paper 
submissions are due March 17, 2003. Please refer to the AMCIS 
2003 web site for detailed submission 


What's New at DSSResources.COM

01/06/2003 Posted Brobst, S. and J. Rarey, "Five Stages of Data 
Warehouse Decision Support Evolution". Check the articles page.

   Send your Ask Dan! questions to

DSS News Releases - January 6 to January 17, 2003

01/16/2003 Blue Cross Blue Shield of North Carolina selects Teradata; 
enterprise data warehouse to improve member service, drive revenue 

01/16/2003 Kennametal goes live with Ketera Spend Management; 'On 
Demand' model enables rapid deployment at low cost.

01/16/2003 Cognos commands OLAP industry mindshare, tops planned 
extranet deployments, according to new OLAP survey.

01/16/2003 SAP announces industry's leading integration and application 
platform, SAP(R) NetWeaver(TM).

01/15/2003 Intergraph sponsors Spatial Information Cooperative Research 
Centre for Australia.

01/15/2003 A new solution for activity and cost management in the Quebec 
healthcare system.

01/15/2003 KnowNow brings new package to synchronize information among 
300 million users of Excel(R) worldwide.

01/15/2003 Israel's largest bank strengthens customer relationships with 
Cognos BI.

01/14/2003 Open Text to offer team collaboration suite.

01/14/2003 Crystal Decisions recognized for outstanding scalability and 

01/13/2003 Wacker deploys Documentum Enterprise Content Management 

01/13/2003 Answerthink introduces breakthrough business process 
intelligence approach and tools.

01/13/2003 SAP announces web-based workforce management capability for 
retail industry at National Retail Federation Convention.

01/13/2003 iManage WorkSite MP awarded value leader in enterprise 
content and collaboration technologies.

01/13/2003 Applix announces first Applix Interactive Planning customers, 
multiple global customer orders for Applix Integra, Applix iTM1.

01/13/2003 EasyAsk's search and navigation capabilities expands 
information access throughout the enterprise.

01/13/2003 LexisNexis selects SAS Software to increase efficiency of 
strategic marketing campaigns.

01/13/2003 Harte-Hanks delivers Allink Retail; analytical CRM solution 
helps retailers capture, manage and optimize customer data.

01/10/2003 Mapping software for PDA saves EMT time, saves lives.

01/09/2003 Boeing demonstrates new network-centric warfare capability.

01/09/2003 Navman introduces the GPS 4400 navigation solution for 
Bluetooth(TM) enabled Pocket PCs.

01/08/2003 U.S. Department of Defense EMALL saves time streamlining data 
analysis with Business Objects.

01/07/2003 Apple unveils world's first 17-inch notebook PC.

01/07/2003 SGI seizes lead in Linux for technical computing with new 
software environment.

01/07/2003 Hungarian Ministry of Agriculture and Regional Development 
selects GeoMedia Technology for Land Information Management.

01/07/2003 Experian's Z-24(R) Catalog database surpasses 100 million 

01/06/2003 Teradata announces on-schedule shipment of data warehousing 
suite; new release provides most significant data warehousing solution 
in the market.

01/06/2003 Comshare(R) survey reinforces that preparedness is most 
important to responding to changing business environment.

01/06/2003 Chartsmith v1.1 now shipping; all new for 2003, popular Mac 
OS X graphing application has something for everyone.

01/06/2003 QRS Diagnostic introduces instant data synchronization to 
enhance its patient information management software.

        Send your NEWS to

Subscribe to DSSResources.COM. One month $10, six months $25. 

DSS News is copyrighted (c) 2003 by D. J. Power. Please send 
your questions to 

DSS Home |  About Us |  Contact Us |  Site Index |  Subscribe | What's New
Please Tell 
Your Friends about DSSResources.COM Copyright © 1995-2021 by D. J. Power (see his home page). DSSResources.COMsm was maintained by Daniel J. Power. See disclaimer and privacy statement.