************************************************************
DSS News
D. J. Power, Editor
January 19, 2003 -- Vol. 4, No. 2
A Bi-Weekly Publication of DSSResources.COM
************************************************************
Check the article by Brobst and Rarey at DSSResources.COM
************************************************************
Featured:
* Ask Dan! - What is the difference between formative and
summative DSS evaluation?
* AMCIS 2003 Call for Papers, Tampa, FL Aug. 4-6, 2003
* What's New at DSSResources.COM
* DSS News Releases
************************************************************
Subscribe to DSSResources.COM. One month $10, six months $25.
Visit http://dssresources.com/subscriber/subscriber.html
************************************************************
Ask Dan!
by Daniel J. Power
What is the difference between formative and summative DSS evaluation?
In Fall 2002, Babajide Adefarati asked a question on the Subscriber Zone
Bulletin Board about conducting formative and summative evaluations of
model-driven DSS. This Ask Dan! expands on and summarizes my responses
to his question. In general, I have associated the terms formative and
summative evaluation with course development and program/project reviews
more than with building DSS, but the terms do provide some useful
distinctions that should be considered in planning the evaluation of
decision support projects. My perception is that more evaluation is
needed of DSS projects.
A quick Google search indicates the terms are used in some papers
related to software usability and testing. Most of the hits were related
to education and social services. My conclusion is that a formative
evaluation would occur during DSS design and development and that a
summative evaluation would occur once the development project is
completed and the decision support system is in use. Some authors
associate formative evaluation with evaluations by users and summative
evaluations with expert and managerial evaluations. There are a number
of different approaches to evaluation that differ based on when the
evaluation occurs, either during the development process or when the
project is complete, the intentions of the evaluator, providing a
formative constructive evaluation versus obtaining a judgmental
summative evaluation, and who does the evaluation, internal or external
evaluators. You'll want to confirm how formative and summative are used
when you are in a discussion about evaluating a DSS. I'll suggest the
following definitions based upon what I have read.
A formative evaluation involves judging the worth of a program/project,
activity or software system while development activities are occurring.
Formative evaluation focuses on intermediate or preliminary outcomes and
results during the development process.
A summative evaluation involves judging the worth of a program/project,
activity or software system at the end of the development process and
following implementation. The focus is on assessing immediate and longer
term outcomes and results.
In my opinion, potential users should provide the primary feedback for
the formative evaluation of a Decision Support System and the evaluation
criteria should primarily focus on user interface and usability issues.
As part of a formative evaluation of a model-driven DSS the model needs
to reviewed and validated by an "expert". Formative evaluation of a
knowledge-driven DSS needs to verify the rules and knowledge-base.
Examining data and document quality are legitimate issues in the
formative evaluation of data or document-driven DSS.
For a large-scale model-driven DSS project, summative evaluation should
include assessments by users and expert evaluators. Criteria should be
broader and the impact of the model-driven DSS on decision making and
the organization should be assessed.
For both formative and summative evaluations one can collect four main
types of data using a variety of data collection methods:
1. Impressionistic or subjective data from developers, users or
potential users of the DSS.
2. Objective data from an unbiased observer. In most situations the
observer will use an explicit, structured assessment protocol.
3. Qualitative data in text, audio or video format. The data may include
answers from potential users to open-ended questions, or anecdotal or
impresionistic comments from an observer or a developer. Based upon my
own experiences in formative evaluation situations, videotapes of user
interactions with a DSS prototype can be especially helpful.
4. Quantitative data is used, but some DSS developers seem to favor
anecdotal evidence. Quantitative data should be collected about the use
of a DSS. The data may be collected by the decision support software, in a
user questionnaire, or from numerical scores given by observers.
As the above discussion suggests, a comprehensive evaluation of a DSS
may include collecting all four types of data. We generally expect that
qualitative data is more likely to be subjective or impressionistic.
Also, we can collect and interpret both quantitative and qualitative
objective data. We can collect data using questionnaires and expert
reviewers, by videotaping one-on-one interaction between a user and an
evaluator, and by using a small group of observers. In both formative
and summative evaluation, data from users and potential users whether
perceived as objective or subjective is likely to have a major impact on
the evaluation conclusions. It seems that the key is to create a
positive, constructive feedback loop in formative evaluation. If the
evaluation suggests the DSS can not be built, then managers need to act
quickly to end the project. A positive approach to evaluation can result
in ending a DSS project or in discontinuing use of a DSS.
A number of Web pages credit Robert Stakes with the following quote
"When the cook tastes the soup, that's formative; when the guests taste
the soup, that's summative." I haven't found a citation for Stakes'
quote, but it's interesting and worth repeating. As always your
comments, feedback and questions are most welcome.
References
LinguaLinks®, SIL International, http://www.sil.org.
Northwest Regional Educational Laboratory, Program Evaluation,
http://www.nwrel.org/evaluation/formative.shtml.
Phillips, B., Social Research: Strategy and Tactics (3rd Edition), New
York, NY: Macmillan, 1976.
************************************************************
Tell your friends! Get DSS NEWS free -- send a blank email
to dssresources-subscribe@topica.com.
************************************************************
AMCIS 2003 Call for Papers, Tampa, FL Aug. 4-6, 2003
The 2003 Americas Conference on Information Systems will be
held in Tampa, Florida from August 4-6, 2003. The list of
mini-tracks for the conference is now available on the Web.
The AIS Special Interest Group on Decision Support, Knowledge
and Data Management Systems (SIG DSS) is sponsoring the
meta-track on Decision Support and Knowledge Management. Karen
Dowling, an IS faculty member at Arizona State, is the SIG DSS
Program Chair and she is coordinating the meta-track. Her
email address is Karen.Dowling@asu.edu. Contact her or a
mini-track chair with questions.
The Decision Support and Knowledge Management meta-track includes
five mini-tracks: Decision Support Systems, Databases for the 21st
Century, Data Warehousing, Geographic and Location-Based Technologies
for Decision Support, and Knowledge Management. Information on all of
the AMCIS 2003 tracks can be found at
http://galletta.business.pitt.edu/amcis2003/index.htm .
Please try to submit abstracts for screening and feedback to
an appropriate mini-track chair by February 17. Final paper
submissions are due March 17, 2003. Please refer to the AMCIS
2003 web site http://www.amcis2003.org for detailed submission
instructions.
************************************************************
What's New at DSSResources.COM
01/06/2003 Posted Brobst, S. and J. Rarey, "Five Stages of Data
Warehouse Decision Support Evolution". Check the articles page.
************************************************************
Send your Ask Dan! questions to askdan@dssresources.com
************************************************************
DSS News Releases - January 6 to January 17, 2003
01/16/2003 Blue Cross Blue Shield of North Carolina selects Teradata;
enterprise data warehouse to improve member service, drive revenue
growth.
01/16/2003 Kennametal goes live with Ketera Spend Management; 'On
Demand' model enables rapid deployment at low cost.
01/16/2003 Cognos commands OLAP industry mindshare, tops planned
extranet deployments, according to new OLAP survey.
01/16/2003 SAP announces industry's leading integration and application
platform, SAP(R) NetWeaver(TM).
01/15/2003 Intergraph sponsors Spatial Information Cooperative Research
Centre for Australia.
01/15/2003 A new solution for activity and cost management in the Quebec
healthcare system.
01/15/2003 KnowNow brings new package to synchronize information among
300 million users of Excel(R) worldwide.
01/15/2003 Israel's largest bank strengthens customer relationships with
Cognos BI.
01/14/2003 Open Text to offer team collaboration suite.
01/14/2003 Crystal Decisions recognized for outstanding scalability and
performance.
01/13/2003 Wacker deploys Documentum Enterprise Content Management
platform.
01/13/2003 Answerthink introduces breakthrough business process
intelligence approach and tools.
01/13/2003 SAP announces web-based workforce management capability for
retail industry at National Retail Federation Convention.
01/13/2003 iManage WorkSite MP awarded value leader in enterprise
content and collaboration technologies.
01/13/2003 Applix announces first Applix Interactive Planning customers,
multiple global customer orders for Applix Integra, Applix iTM1.
01/13/2003 EasyAsk's search and navigation capabilities expands
information access throughout the enterprise.
01/13/2003 LexisNexis selects SAS Software to increase efficiency of
strategic marketing campaigns.
01/13/2003 Harte-Hanks delivers Allink Retail; analytical CRM solution
helps retailers capture, manage and optimize customer data.
01/10/2003 Mapping software for PDA saves EMT time, saves lives.
01/09/2003 Boeing demonstrates new network-centric warfare capability.
01/09/2003 Navman introduces the GPS 4400 navigation solution for
Bluetooth(TM) enabled Pocket PCs.
01/08/2003 U.S. Department of Defense EMALL saves time streamlining data
analysis with Business Objects.
01/07/2003 Apple unveils world's first 17-inch notebook PC.
01/07/2003 SGI seizes lead in Linux for technical computing with new
software environment.
01/07/2003 Hungarian Ministry of Agriculture and Regional Development
selects GeoMedia Technology for Land Information Management.
01/07/2003 Experian's Z-24(R) Catalog database surpasses 100 million
households.
01/06/2003 Teradata announces on-schedule shipment of data warehousing
suite; new release provides most significant data warehousing solution
in the market.
01/06/2003 Comshare(R) survey reinforces that preparedness is most
important to responding to changing business environment.
01/06/2003 Chartsmith v1.1 now shipping; all new for 2003, popular Mac
OS X graphing application has something for everyone.
01/06/2003 QRS Diagnostic introduces instant data synchronization to
enhance its patient information management software.
************************************************************
Send your NEWS to news@dssresources.com.
************************************************************
Subscribe to DSSResources.COM. One month $10, six months $25.
Check http://dssresources.com/subscriber/subscriber.html
************************************************************
DSS News is copyrighted (c) 2003 by D. J. Power. Please send
your questions to daniel.power@dssresources.com.
|