How does sensitivity analysis differ from "What if?" analysis?
by Dan Power
Editor, DSSResources.COM
In the early days of decision support deployment, one of the major
"selling points" of vendors and academics was the
ability to do "What If?" analysis. In the 1970s, model-driven
decision support for sales and production planning helped a manager
change a decision variable like the number of units to produce
and then immediately get a new result for an outcome variable
like profit. As decision support has gotten more sophisticated and become
more diverse in its use, "What If?" as a concept has broadened.
The decision support community has also introduced more precise terminology from the
mathematical modeling literature.
In most decision support and analytic applications, sensitivity and "What If?" analysis refer to quantitative analyses. In some of the decision making and planning literature, "What if analysis" is also discussed as a qualitative, brainstorming scenario approach that "uses broad, loosely structured questioning to investigate
contingencies." In the context of business intelligence and
data-driven decision support, "What If?" is often used as a descriptor for ad hoc queries of a decision support data base.
According to a vendor website, Applix.com, planners use models to address "What If"
questions such as: 1) What profits can we anticipate next
year if inflation is 7 percent and we continue current pricing
policies? 2) If we open a new plant, what profits can we expect?
3) What if we were to hire 55 people in Sales, 10 in Marketing
and 35 in R&D? 4) What is the impact on manufacturing and shipping
if the price of oil increases 15% during Q2? and 5) What would
be needed for raw material and inventory if the demand of a
product went up 20%?
In the decision support literature and in common discourse, we don't have
agreement about the difference between "What If?" analysis and
sensitivity analysis. Microsoft Excel documentation
defines "What-if analysis" as a "process of changing the values
in cells to see how those changes affect the outcome of formulas
on the worksheet. For example, varying the interest rate that is
used in an amortization table to determine the amount of the
payments." Four tools in Excel are commonly categorized as "What If?"
or sensitivity analysis (Winston, 2004) tools: Data Tables, Goal
Seek, Scenarios, and Solver. The simplest type of "What If?"
analysis is manually changing a value in a cell that is used in
a formula to see the result. Excel experts seem to use the terms
sensitivity and "What If?" analysis interchangeably.
To get a better understanding of what is possible, let's briefly
examine how one would implement "What If?" or sensitivity analysis
using MS Excel tools. First, a data table is a range of cells that
summarizes the results of changing certain values in formulas in
a model. There are two types of data tables: one input variable
tables and two input variable tables. "Two-variable data tables
use only one formula with two lists of input values. The formula
must refer to two different input cells." In Microsoft's Mortgage
Loan Analysis example, a two-variable data table would show how
different interest rates and loan terms would affect the mortgage
payment amount. The table shows the decision maker how sensitive
the payment amount is to the interest rate. The Goal Seek tool
is helpful when you know the desired result from a model and want
to find the appropriate input or decision variable levels. "What
If?" involves incrementally changing an input until the goal is
reached. Goal Seek automates this trial and error process. Scenarios
let an Excel user construct strategies where multiple decision
variables are changed in each scenario. For example, a decision
maker may have best case, most likely, and worst case scenarios.
Finally, Solver is an optimization tool that includes a sensitivity
analysis capability. Monte Carlo simulation in Excel can also be
used to assist in "What If?" or sensitivity analysis. Spreadsheets
models with probability distributions for inputs can simulate
outcomes for a range of input parameters.
According to Parnell (1997), "a decision variable is a variable
over which the decision maker has control and wishes to
select a level, whereas a strategy refers to a set of values for
all the decision variables of a model. An optimal strategy
is the strategy which maximises the value of the decision maker's
objective function (e.g. profit, social welfare, expected
utility)." In general, mathematical methods assess the sensitivity
of a model's output to the range of variation of one or
more inputs. Sensitivity analysis is used to determine what inputs,
parameters or decision variables contribute more to the
variance in the output of a model and hence are the most important
and most "sensitive".
Let's check some definitions from Web sites. Wikipedia (wikipedia.org)
notes "What-if analysis of a model considers the question: 'What
happens to the result if we make a particular change to a parameter?'.
If the change of a parameter is small this is also called sensitivity
analysis: 'How sensitive is the result to a small change of a parameter?'
Wikipedia also defines sensitivity analysis as "the study of how the
variation in the output of a model (numerical or otherwise) can be
apportioned, qualitatively or quantitatively, to different sources of
variation." Wikipedia shows the differing usage of these terms across
various disciplines. The sensitivity analysis entry notes that in a
business context "sensitivity analysis can provide information to
managers about which elements of the business require more concentration.
For example if sales, variable costs, fixed costs or output were to
increase or decrease by 10% which would have the most effect on profit?"
The Web Dictionary of Cybernetics and Systems defines sensitivity
analysis as "a procedure to determine the sensitivity of
the outcomes of an alternative to changes in its parameters (as
opposed to changes in the environment; see contingency
analysis, a fortiori analysis). If a small change in a parameter
results in relatively large changes in the outcomes, the
outcomes are said to be sensitive to that parameter. This may mean
that the parameter has to be determined very accurately or
that the alternative has to be redesigned for low sensitivity. (IIASA)"
Finally, the Michigan Department of Environmental Quality
(www.michigan.gov) defines a sensitivity analysis as "the process
of varying model input parameters over a reasonable range (range of
uncertainty in values of model parameters) and observing
the relative change in model response."
Parnell (1997) identifies uses of sensitivity analysis in decision
making, communication, understanding systems and in model development.
Based on his discussion, a model-driven DSS with appropriate sensitivity
analysis should help in 1) testing the robustness of an optimal
solution, 2) identifying critical values, thresholds or break-even
values where the optimal strategy changes, 3) identifying sensitive
or important variables, 4) investigating sub-optimal solutions,
5) developing flexible recommendations which depend on circumstances,
6) comparing the values of simple and complex decision strategies,
and 7) assessing the "riskiness" of a strategy or scenario.
The most common "What If?" analysis in model-driven DSS is changing
an input value in an ad hoc way and seeing the result. This type of
analysis has severe limitations. The analysis is likely to be more
complete if an input object like a spinner or a slider is used to
change values. Such an approach is much faster and easier than typing
in individually new input values. A range sensitivity analysis evaluates
the effect on outputs by systematically varying one of the model inputs
across its entire range of plausible values. According to Frey and
Patil "results of nominal range sensitivity are most valid when
applied to a linear model."
What are the limitations of "What If?" analysis? If the analysis is
ad hoc rather than systematic, the analysis is likely to miss potential
problems and solutions. Managers may not understand the assumptions of
the sensitivity analysis, e.g. assuming a linear relationship. Also,
in general it is impossible to audit the thoroughness of sensitivity
and "What If?" analyses and their impact on decision making. My general
sense is that systematic sensitivity analysis using a one or two-variable
data table should be required in all model-driven DSS based upon algebraic
models. Relying on an ad hoc manipulation of single variables in a
quantitative model is always problematic and limited.
So "What If?" analysis is used broadly for techniques that help
decision makers assess the consequences of changes in models and
situations. Sensitivity analysis is a more specific and technical
term generally used for assessing the systematic results from changing
input variables across a reasonable range in a model. The current
frontier is animated sensitivity analysis where a visual display like
a chart or graph is sytematically varied showing results of changing
model parameters. Check the Planners Lab review (Power, 2006).
As always your comments and questions are welcome.
References
Alexander, E.R. (1989). Sensitivity analysis in complex decision
models, Journal of the American Planning Association 55: 323-333.
Frey, H. C. and S. R. Patil, "Identification and Review of Sensitivity
Analysis Method,"NCSU/USDA Workshop on Sensitivity Analysis Methods,
http://www.ce.ncsu.edu/risk/abstracts/frey.html
Isukapalli, S.S., "Uncertainty Analysis of Transport-Transformation
Models, "A dissertation submitted to the Graduate School--New
Brunswick, Rutgers, The State University of New Jersey, URL
http://www.ccl.rutgers.edu/~ssi/thesis/thesis-node14.html
Microsoft help, http://office.microsoft.com/en-us/assistance/
Pannell, D.J. (1997). Sensitivity analysis of normative economic
models: Theoretical framework and practical strategies. Agricultural
Economics 16: 139-152, at URL
http://cyllene.uwa.edu.au/~dpannell/dpap971f.htm
Power, D., "What is Planners Lab?" DSS News, Vol. 7, No. 11, May 21,
2006.
Sensitivity analysis, from Wikipedia, the free encyclopedia, URL
http://en.wikipedia.org/wiki/Sensitivity_analysis
Web Dictionary of Cybernetics and Systems at URL
http://pespmc1.vub.ac.be/ASC/SENSIT_ANALY.html
What-if analysis, from Wikipedia, the free encyclopedia, URL
http://en.wikipedia.org/wiki/What-if_analysis
Winston, W. L., Microsoft Excel Data Analysis and Business
Modeling, Microsoft Press, 2004.
**************************
Citation:
Power, D., "How does sensitivity analysis differ from 'What if?' analysis?" DSS News, Vol. 7, No. 16, July 30, 2006; revised for Decision Support News August 31, 2014.
Last update: 2014-08-30 07:31
Author: Daniel Power
Print this record
Show this as PDF file
You cannot comment on this entry
