Verification, Validation, and Accreditation in the Life
Cycle of Models and Simulations
Jennifer Chew (HQ, U.S. Army Developmental Test Command) and Cindy Sullivan (U.S. Army Yuma Proving Ground)
Verification, validation, and accreditation (VV&A) activities should be an on-going process throughout the life cycle of models and simulations (M&S). It is important to note that there is no single set of VV&A tasks, events, or methods that would apply every time to every situation. The VV&A emphasis and methods used vary depending on the particular life cycle phase it is in, previous VV&A and use, the risks and uncertainty, its size and complexity, and of course, the resources available. For simplification, this paper discusses the activities and tasks during the early stages of model development and addresses each of the VV&A efforts separately, along with its associated activities. It outlines the specific VV&A activities and products that are appropriate to each phase of model development.
Capability Maturity Models Support of Modeling and
Simulation Verification, Validation, and Accreditation
Candace L. Conwell (Space and Naval Warfare Systems Command) and Rosemary Enright and Marcia A. Stutzman (Logicon Information Systems and Services)
Both government and industry are involved in the acquisition and development of modeling and simulation (M&S) products. The effectiveness and maturity of an organization's acquisition process directly affect the cost, schedule, and quality of the M&S products that are delivered to the user. When M&S program sponsors implement best practices throughout acquisition, critical verification, and validation (V&V) tasks can be conducted without inordinate cost. Department of Defense (DoD) Instruction 5000.61 on M&S Verification, Validation, and Accreditation (VV&A) directs that M&S systems used for acquisition decisions will be verified, validated, and accredited (DoDI 1996). However, many M&S users are attempting to implement VV&A processes for legacy M&S systems that lack documentation and are finding the costs unsupportable. The Carnegie Mellon University Software Engineering Institute (SEI) has described processes involved in implementing and measuring capability in software acquisition and development. These Capability Maturity Models ((CMM) and Capability Maturity Model are trademarks registered by Carnegie Mellon University) when applied together ensure that the acquisition process is in place before the software development process is implemented. This paper discusses how the use of these two CMMs can improve DoD's ability to develop M&S with the customer's need for VV&A in mind.
Planning for Verification, Validation, and
Accreditation of Modeling and Simulation Applications
Osman Balci (Virginia Tech), William F. Ormsby (Naval Surface Warfare Center ) and John T. Carr, III and Said D. Saadi (Naval Surface Warfare Center)
A comprehensive and detailed verification, validation, and accreditation (VV&A) plan and its proper execution are crucially important for the successful accreditation of a modeling and simulation (M&S) application. We provide guidance in developing and executing such a plan throughout the entire M&S application development life cycle.
Enhancing Modeling and Simulation Accreditation by
Structuring Verification and Validation Results
Dirk Brade (Universität der Bundeswehr München )
Model Verification, Validation and Accreditation (VV&A) is as complex as developing a Modeling and Simulation (M&S) application itself. For the purpose of structuring both Verification and Validation (V&V) activities and V&V results, we introduce a refined V&V process. After identification of the major influence factors on applicable V&V, a conceptual approach for subphase-wise organization of V&V activities is presented. Finally a hierarchical presentation of V&V results is shown which addresses different people involved in use or in accreditation of simulation models.
Well-Defined Intended Uses: An Explicit Requirement
for Accreditation of Modeling and Simulation Applications
Osman Balci (Virginia Tech) and William F. Ormsby (Naval Surface Warfare Center)
A modeling and simulation (M&S) application is built for a specific purpose and its acceptability assessment is carried out with respect to that purpose. The accreditation decision for an M&S application is also made with respect to that purpose. The purpose is commonly expressed in terms of “intended uses.” The quality of expressing the intended uses significantly affects the quality of the acceptability assessment as well as the quality of making the accreditation decision. The purpose of this paper is to provide guidance in proper definition of the intended uses.
Joint Warfare System (JWARS) Verification and
Validation Lessons Learned
Michael L. Metz (Innovative Management Concepts, Inc.)
JWARS V&V (a joint venture of Innovative Management Concepts in Sterling VA and BMH Associates in Norfolk VA) has been responsible for conducting the verification and validation (V&V) of the JWARS simulation since September of 1997. This paper describes the lessons learned during the conduct of the effort including: the JWARS V&V process, the JWARS V&V Plan, reports delivered, and results to date. Special emphasis is on the use of the DoD VV&A Recommended Practices Guide as a basis of JWARS V&V planning and procedures and the evolution of the JWARS V&V Integrated Product Team
Verification and Validation without Independence: A
Recipe for Failure
James D. Arthur and Richard E. Nance (Virginia Polytechnic Institute and State University)
Verification and validation (V&V) is a prominent technical area within simulation, attested to by the 12 tutorial sessions (including five advanced) included in the past ten Winter Simulation Conferences (WSCs). In recent WSCs the issue of Independent V&V (IV&V) has drawn increased attention, with sessions examining the perceived lack of use and little concern for the technique evinced within the simulation community. The objectives of this paper are four-fold: (1) to examine the current picture in software systems development, (2) to review the rationale, role and expressed need for IV&V, (3) to identify the benefits attendant in the insistence on the "independent" status of the activity, and (4) to respond to the usual criticisms of negative impacts on cost and schedule. While the treatment is couched in the more general software systems context, we contend that simulations represent prime candidates for IV&V application.
Verification and Validation of Object-Oriented
Artifacts Throughout the Simulation Model Development Life
John T. Carr, III (Naval Surface Warfare Center ) and Osman Balci (Virginia Tech)
The purpose of this paper is to present a series of questions (or indicators) for assessing the verity and validity of the artifacts produced during the entire object-oriented simulation model development life cycle. Using modern object-oriented development processes, artifacts developed in one phase flow seamlessly from those of the previous phase. This provides forward and backward traceability between artifacts. This inherent backward traceability has been exploited by tracing defects in artifacts back to their defective ancestral artifacts. Questions are then phrased such that when answered in the negative indicate the presence of defects. Use of the Evaluation Environment software tool facilitates the integration of the answers to the assessment questions and enables an overall evaluation. The collection of questions can be useful for the verification and validation of artifacts in any object-oriented simulation model development.
An Integrated Approach to Verification, Validation,
and Accreditation of Models and Simulations
Don Caughlin (University of Colorado at Colorado Springs)
In an M&S-Based Systems Acquisition, computer simulation is used throughout the development process not just as an analysis tool but also as a development tool. In general, development of a system capability using M&S-Based Systems Development will result in multiple models or simulations to meet specific needs. The Verification, Validation and Accreditation (VV&A) of each these tools is integral to M&S development. Integrating V&V activities with M&S development and then integrating the VV&A activities for all of the M&S resources that support a program provides a cost effective approach to ensure the necessary confidence in M&S results within the time and resources available. This paper presents such an integrated approach to VV&A from a system perspective and identifies the relationships between the M&S resources in an integrated V&V program.
Validation of Trace-Driven Simulation Models: More
on Bootstrap Tests
Jack P.C. Kleijnen (Tilburg University (KUB)), Russell C. H. Cheng (University of Southampton) and Bert Bettonvil (Tilburg University)
‘Trace-driven' or ‘correlated inspection' simulation means that the simulated and the real systems have some common inputs (say, arrival times) so the two systems' outputs are cross-correlated. To validate such simulation models, this paper formulates six validation statistics, which are inspired by practice and statistical analysis; for example, the simplest statistic is the difference between the average simulated and real responses. To evaluate these validation statistics, the paper develops novel types of bootstrapping based on subruns. Three basic bootstrap procedures are devised, depending on the number of simulation replicates: one, two, or more replicates. Moreover, for the case of more than two replicates the paper considers conditional versus unconditional resampling. These six validation statistics and four bootstrap procedures are evaluated in extensive Monte Carlo experiments with single-server queueing systems. The main conclusion is that bootstrapping of the simplest validation statistic gives the correct type I error probability, and has relatively high power.
Verification of Controller Software
Harald Schludermann, Thomas Kirchmair, and Markus Vorderwinkler (PROFACTOR Produktionsforschungs GmbH)
The basic idea of Soft-Commissioning (SoftCom) is to test industrial control software by connecting a controller, e. g. a PLC (Programmable Logic Controller) to a commercial discrete event simulator (DES), which provides system reactions and sensor signals similar to the behavior of real hardware, e. g. an industrial manufacturing line. In order to establish a connection between simulator and PLC, a modular architecture was developed. The basis of this modular system is a communication protocol common to all members. The two basic modules are the I/O Devices Driver (IODD), which is used to interface between the I/O hardware and the SoftCom protocol, and the Simulator to real World Interface (SWI). The SWI is used to link the simulator to the SoftCom system.
The Design of a Solid-State Physical Model of an
Automated System to be Used as a Test Bed for Control
Fernando G. Gonzalez, Alicia Helton, Douglas Helton, Jeffrey Smith, Eileen Thompson, and Gerry Walterscheild (University of Central Florida)
In order to develop, test, and validate control software for managing automated systems, laboratories have traditionally constructed experimental test beds using actual physical equipment (small scale). These experimental systems typically occupy a large amount of lab space, cost thousands of dollars to construct, and require considerable human expertise to operate. Using dedicated micro-controllers (programmable logic controllers), we have proposed the use of a solid-state physical model of an automated system which faithfully replicates the operating characteristics of an ensemble of physical equipment that would typically comprise an automated system. In this paper we present the design of a solid-state physical model of a Flexible Manufacturing System (FMS). Solid-state models have several unique advantages over the traditional models. First, they are inexpensive and can easily be replicated at other laboratories. Second, they can be easily reconfigured to consider alternative scenarios. Third, they can consider an emulated environment that is far more complex than those that are typically addressed by models using actual equipment. Finally, they are totally reliable and safe, and require minimal expertise to operate. This paper discusses the design and operational characteristics of the solid-state model along with its anticipated uses and current limitations.
Strategic Directions in Verification, Validation,
and Accreditation Research
Robert G. Sargent (Syracuse University), Priscilla A. Glasow (The MITRE Corporation), Jack P.C. Kleijnen (Tilburg University (KUB)), Averill M. Law (Averill M. Law & Associates, Inc.), Ian McGregor (Brooks Automation, AutoSimulations Division) and Simone Youngblood (DMSO)
Six simulation professionals present their views on the directions that they believe that verification, validation, and accreditation research should take. Two of the six are active verification, validation, and accreditation researchers from academia, two develop industry simulation models, and two work in verification, validation, and accreditation of military simulation models. A number of areas and topics for research in verification, validation, and accreditation are identified. It appears that application domains of simulation models affect what topics need verification, validation, and accreditation research.