Chapter 6

Chapter 6
Statistical Analysis of Output from
Terminating Simulations
Statistical Analysis of Output from
Terminating Simulations
• Random input leads to random output (RIRO)
• Run a simulation (once) — what does it mean?
– Was this run “typical” or not?
– Variability from run to run (of the same model)?
• Need statistical analysis of output data
– From a single model configuration
– Compare two or more different configurations
– Search for an optimal configuration
• Statistical analysis of output is often ignored
– This is a big mistake – no idea of precision of results
– Not hard or time-consuming to do this – it just takes a little planning
and thought, then some (cheap) computer time
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
Output Analysis
Output analysis is concerned with
• Designing replications
Obtain most reliable info with minimum number of replications and
minimum run length.
• Computing statistics
Point and confidence interval estimation
Size and independency issues
• Presenting them in a textual and graphical
Aim is to understand the system behavior and
generate predictions for it!
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
Time Frame of Simulations
• Terminating: Specific starting, stopping conditions
– Run length will be well-defined (and finite)
• Steady-state: Long-run (technically forever)
– Theoretically, initial conditions don’t matter (but
practically they usually do)
– Not clear how to terminate a simulation run
• This is really a question of intent of the study
• Has major impact on how output analysis is done
• Sometimes it’s not clear which is appropriate
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
Model 6.1
• Same as Model 5.3
Number of trunk lines=26
No additional staff during 5-8 hrs.
• 10 runs are made
For terminating case, make IID replications
Run>Setup>Replication Parameters:
Number of Replications =10
Check both boxes for Initialize Between Replications
• Outputs are saved to .dat files
Statistics Module, Type=output,
Data file name= Filename.dat
Asli Sencer
Outputs of Model 6.1
• Category Overview report will have some statisticalanalysis results of the output across the replications
Asli Sencer
Output Precision in Model 6.1
This information (except standard deviation) is in Category
Overview report
If > 1 replication specified, Arena uses cross-replication data as above
For other confidence levels or graphics – Output Analyzer
Asli Sencer
Confidence Interval Estimation
 =number of replications
 =sample mean
 =sample standard deviation
−1,1−∝/2 =critical value for ∝ significance level
(1-α)x100% Confidence Interval (CI):
for n < 30,  ± −1,1−∝/2
for n ≥ 30,  ± ∝/2

Interpretation of a Confidence Interval
• A CI is an interval with random (data-dependent) endpoints
that’s supposed to have stated probability (1-α) of
containing, or covering, the expected values
– “Target” expected value is a fixed, but unknown, number
– Expected value = average of infinite number of replications
• A CI is not an interval that contains, say, 95% of the data
generated by simulation outputs.
– That’s a prediction interval … useful too, but different
• 95% CI means:
If we make infinitely many batch runs of size  and
construct C.I. for α=5%, 95% of the time this interval will
contain the true (but unknown) population parameter.
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
Required Number of Replications to
Achieve a Certain Precision
• Half-width =


Want this to be “small,” say
< h where h is prespecified
• We prefer confidence intervals with smaller half width — for
higher precision
• Question: How many replications are needed to get the
required (better) precision?
We can’t control t or s. We may just increase n — how much?
– Trial and error (now)
– Approximate number for acceptable precision (below)
– Sequential sampling (Chapter 11)
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
Half Width and Number of

• Let ℎ =required half-width, solve for  =  2 −1,1−∝/2 2
• Not really solved for n (t, s depend on n)
• Approximation:
• Replace t by z, corresponding normal critical value
– Pretend that current s will hold for larger samples
– Get  ≅  1−∝/2 2
s = sample standard
deviation from “initial”
number n0 of replications
• Easier but different approximation:
ℎ0 2
 ≅ 0 2
Simulation with Arena
h0 = half width from
“initial” number n0 of
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
n grows
as h
Number of Replications Needed
• If we require h=$250 rather than $812 for total cost,
ℎ0 2 10 812.76
 ≅ 0 2 =
250 2
= 105.7
replications are needed.
Asli Sencer
Model 6.2
• 110 Runs
• 95% CI for the total cost=
22,175.19 +- 369.54, half width=369.54>250!
Required accuracy is still not met, why?
As more replications are made, in ℎ =


∝/2 does not change,  increases; but , the sample
standard deviation might increase. So ℎ might decrease.
• 95% CI for the %rejected calls=
11.73.96 +- 0.51, accuracy is quite satisfactory!
Asli Sencer
Model 6.3
• 1000 Runs-as a trial
• Save the output to Total cost.dat
• Open Output analyzer as a separate application
File>Data File>Export
Export binary data in .dat file to a plain ASCII text file and
• Open Arena Input Analyzer
Plot the histogram of the Total Costs
Asli Sencer
Histogram of 1000 Total Cost Values
• Since Total Cost values is a sum, law of large numbers apply. We see
that the distribution approaches normal as the number of
replications increase!
• Same is true for average statistics due to central limit theorem.
• It is not true for extreme value statistics like maximum or minimum.
Asli Sencer
Confidence Intervals (cont’d)
• Usual formulas assume normally-distributed
Never true in simulation
Might be approximately true if output is an
average, rather than an extreme
Central limit theorem
Issues of robustness, coverage, precision –
details in book
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
Comparison of Alternatives
Statistical Hypothesis Test
H 0  1   2  0
H 1  1   2  0
Reject Ho if
X1 X
 i is the mean performance of system i
is significantly large or small, i.e.,
performance of system 1 is significantly different than system 2!
Here: x 1 : Total Cost of Base Model (110 observations)
x 2 : Total Cost of Alternative Model (110 observations)
Simulation with Arena
Chapter 5 – Detailed Modeling and
Terminating Statistical Analysis
Comparing Two Scenarios
• Base Scenario: Model 6.4 (Same as in Model 5.3)
-110 runs
-26 Trunk Lines, No New Staff between 12:00-16:00
• Alternative scenario: Model 6.4 (More-resources scenario)
-110 runs
-29 Trunk Lines, (Change the capacity from 26 to 29)
-Hire three for each of Larry, Moe, Curly, Hermann and
Sales Resources. (Change these variables from 0 to 3)
• Tradeoff is between increased salary cost but decreased excess
waiting costs. Will the total costs decrease?
• Percent Rejected calls will decrease, but how much?
Asli Sencer
Comparison of Scenarios
• Runs both models for 110 times.
• Statistics Data Module
Save output files –BaseCase.dat or -MoreResources.dat.
• 95% CI for total costs are
Base model: 22,175.19 +- 369.54=[21,805, 22,544]
Increased resources: 24,542.82 +- 329.11=[24,213, 24,871]
Intervals do NOT overlap, hence
Total Costs are significantly different at 5% significance level.
• 95% CI for percent rejected are
Base model: 11.74 +- 0.51=[11.23, 12.25]
Increased resources: 1.73 +- 0.31=[1.42, 2.04]
Intervals do NOT overlap, hence
Percent Rejected are significantly different at 5% significance level.
Asli Sencer
Arena Output Analyzer
• Separate application in Arena
• Operates in output files (.dat) generated by Arena
through the Statistics data module
• Data in .dat file is in binary format to be opened
by Arena Output Analyzer only!
• Provides confidence intervals on expected output
statistics as also appear in Arena output reports.
• Provides statistical comparison of two scenarios,
and others.
Asli Sencer
Comparison of Scenarios with Arena
Output Analyzer
• Open Output Analyzer
• Select File>New to open a data group, i.e., list of .dat files
• Add
• Can save this data group as .dgr file to refer easily
• Analyze>Compare Means
Add each pair of comparisons by choosing ‘lumped’ so
that all 110 values are considered in the analysis
Asli Sencer
Hypothesis Tests
• Ho: Mean TC of base case = Mean TC of more resources
Ha: Mean TC of base case ≠ Mean TC of more resources
• Ho: Mean % rejected of base case = Mean % rejected of
more resources case
Ha: Mean % rejected of base case ≠ Mean % rejected of
more resources case
Asli Sencer
Output Report-Compare Means
Confidence interval on difference misses 0, so conclude that there is a (statistically)
significant difference between the base model and the alternative at α=5%
Asli Sencer
Evaluating Many Scenarios with
Process Analyzer
• Separate application in Arena
• Allows making multiple pairwise scenario comparisons
at a time.
• PAN operates on Arena program files with .p extension,
generated when .doe model is run.
• A PAN scenario includes a program file, a set of values
for the input controls (decision variables in the form of
variables and resources), a set of output responses.
• A PAN project is a collection of such scenarios that can
be saved by .pan extension for future reference.
Asli Sencer
Development of a PAN Project
Use Model 6.5
110 runs
Output data files are deleted since they will be
useless in PAN.
Open a PAN project
File > New, File > Open
Add a new scenario.
Double click on the raw
Name=Base Case, Program File=Model 6.5.p Replications=110
Add contols
Right click in this line OR Insert > Control
Under Resources: The capacity of trunk line
Under User Specified: New Tech 1, New Tech 2, New Tech 3, New Tech All,
New Sales
Add responses
Right click on this line OR Insert > Responses
Under user specified: Total Cost, Percent Rejected
Asli Sencer
New Scenarios
• Suppose you have $1360/week to spend on all additional
resources. To which of the six expandable resources should
you allocate the new money?
• Then following 6 alternative scenarios apply in addition to
Base Case.
13 more trunk lines ($98 each)
4 more tech 1,2,3 people ($320 each)
3 more tech all people ($360 each)
4 more sales people ( 340 each)
• Run the scenarios
Check the scenarios to run
Run > Go OR play button OR F5 function key
Asli Sencer
PAN screen
Asli Sencer
Generating Reports for Multiple
Comparison in PAN
• Insert > Chart OR right click on a response column.
Chart type=Box whiskers
Check Identify Best Scenarios box
Select ‘smaller is the better’
• Red boxes are significantly better than blue ones at 5%
significance level.
• To decrease the half width of a scenario, increase the number
of replications of that specific one.
• Error tolerance is a positive value that represents an amount
small enough that you don’t care if the selected scenarios are
actually inferior to the true best one by at most this amount.
A positive error reduce the number of selected scenarios at
the risk of being off by a little bit.
Asli Sencer
A PAN Report
Asli Sencer

similar documents