The second issue to do with simulation accuracy is model validation. Different from verification, validation (Law and Kelton, 2000) is the process of determining whether a simulation model (as opposed to the computer program) is an accurate representation of the system, for the particular objective of the study. In the VIVACE work, three types of validations are carried out in SCMB, namely
Conceptual model validation. Conceptual Model Validation is about determining that the input, content, assumptions and simplifications of the proposed model are sufficiently accurate for the purpose at hand. The question that needs to be asked is thus: does the conceptual model contain all the necessary details to meet the objectives of the simulation study? Robinson (2004) outlines the key components of the conceptual model as:
All these components have been discussed and agreed between the partners during VIVACE task meetings, work program meetings and the deliverable review process.
Data validation. Robinson (2004) again defines data validation as determining that the contextual data and the data required for model realization and validation are sufficiently accurate for the purpose at hand. For this purpose, the PW2000 TEC use case from Volvo Aero and a simplified Trent 900 engine use case from Rolls Royce based on publicly available data are used. An iterative approach was followed until it was agreed by all parties that the data were accurate.
Result validation. Result validation is about validating the simulation model through comparison of model outputs with real system outputs or outputs from other models. If there is an existing system, then performance measures from a simulation model of the existing system can be compared with those collected from the actual existing system. This is the most important model validation technique that is available. If result validation is successful, then it lends significant credibility to the simulation model. The model outputs can also be compared with outputs from other models, especially in the absence of real system performance measures.
To validate the simulation model, its outputs were compared with planned system outputs in two ways –visual and statistical. First, a confidence interval was calculated for the difference between model outputs and real system outputs, based on the defined performance measures. Since real system outputs vary significantly from the planned numbers at VAC and the model is run using the planned number from the master production schedule as input (this is how VAC operates), it was felt more appropriate to compare model output with the planned output on a like-for-like basis. The 95% confidence interval thus obtained between the simulated and planned throughputs is (-1.5, 2.16) with a mean value of 0.333. Since this interval contains the value zero, the throughputs from the simulation model were not statistically different from planned output, at this significance level.
To further explore the validity of the simulation model and the dynamic resemblance between model and planned outputs, a visual observation approach (Kleijnen, 1995) was also used, to compare time series of simulation model outputs with the historical time series of planned outputs. The diagram below compares the monthly planned throughputs and those from five replications of the model. Given the observed correspondence in terms of general level and behaviour between the model output and planned values, it can again be claimed that the model is valid for the types of supply chain investigation for which it was designed.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |