 |
WSC'00 |
|
WSC'00 |
Verification, Validation, and Accreditation in the Life
Cycle of Models and Simulations
Jennifer Chew (HQ, U.S. Army
Developmental Test Command) and Cindy Sullivan (U.S. Army Yuma Proving Ground)
Abstract:
Verification, validation, and accreditation (VV&A)
activities should be an on-going process throughout the life cycle of models
and simulations (M&S). It is important to note that there is no single set
of VV&A tasks, events, or methods that would apply every time to every
situation. The VV&A emphasis and methods used vary depending on the
particular life cycle phase it is in, previous VV&A and use, the risks and
uncertainty, its size and complexity, and of course, the resources available.
For simplification, this paper discusses the activities and tasks during the
early stages of model development and addresses each of the VV&A efforts
separately, along with its associated activities. It outlines the specific
VV&A activities and products that are appropriate to each phase of model
development.
Capability Maturity Models Support of Modeling and
Simulation Verification, Validation, and Accreditation
Candace L.
Conwell (Space and Naval Warfare Systems Command) and Rosemary Enright and
Marcia A. Stutzman (Logicon Information Systems and Services)
Abstract:
Both government and industry are involved in the
acquisition and development of modeling and simulation (M&S) products. The
effectiveness and maturity of an organization's acquisition process directly
affect the cost, schedule, and quality of the M&S products that are
delivered to the user. When M&S program sponsors implement best practices
throughout acquisition, critical verification, and validation (V&V) tasks
can be conducted without inordinate cost. Department of Defense (DoD)
Instruction 5000.61 on M&S Verification, Validation, and Accreditation
(VV&A) directs that M&S systems used for acquisition decisions will be
verified, validated, and accredited (DoDI 1996). However, many M&S users
are attempting to implement VV&A processes for legacy M&S systems that
lack documentation and are finding the costs unsupportable. The Carnegie
Mellon University Software Engineering Institute (SEI) has described processes
involved in implementing and measuring capability in software acquisition and
development. These Capability Maturity Models ((CMM) and Capability Maturity
Model are trademarks registered by Carnegie Mellon University) when applied
together ensure that the acquisition process is in place before the software
development process is implemented. This paper discusses how the use of these
two CMMs can improve DoD's ability to develop M&S with the customer's need
for VV&A in mind.
Planning for Verification, Validation, and
Accreditation of Modeling and Simulation Applications
Osman Balci
(Virginia Tech), William F. Ormsby (Naval Surface Warfare Center ) and John T.
Carr, III and Said D. Saadi (Naval Surface Warfare Center)
Abstract:
A comprehensive and detailed verification, validation,
and accreditation (VV&A) plan and its proper execution are crucially
important for the successful accreditation of a modeling and simulation
(M&S) application. We provide guidance in developing and executing such a
plan throughout the entire M&S application development life cycle.
Enhancing Modeling and Simulation Accreditation by
Structuring Verification and Validation Results
Dirk Brade
(Universität der Bundeswehr München )
Abstract:
Model Verification, Validation and Accreditation
(VV&A) is as complex as developing a Modeling and Simulation (M&S)
application itself. For the purpose of structuring both Verification and
Validation (V&V) activities and V&V results, we introduce a refined
V&V process. After identification of the major influence factors on
applicable V&V, a conceptual approach for subphase-wise organization of
V&V activities is presented. Finally a hierarchical presentation of
V&V results is shown which addresses different people involved in use or
in accreditation of simulation models.
Well-Defined Intended Uses: An Explicit Requirement
for Accreditation of Modeling and Simulation Applications
Osman
Balci (Virginia Tech) and William F. Ormsby (Naval Surface Warfare Center)
Abstract:
A modeling and simulation (M&S) application is
built for a specific purpose and its acceptability assessment is carried out
with respect to that purpose. The accreditation decision for an M&S
application is also made with respect to that purpose. The purpose is commonly
expressed in terms of “intended uses.” The quality of expressing the intended
uses significantly affects the quality of the acceptability assessment as well
as the quality of making the accreditation decision. The purpose of this paper
is to provide guidance in proper definition of the intended uses.
Joint Warfare System (JWARS) Verification and
Validation Lessons Learned
Michael L. Metz (Innovative Management
Concepts, Inc.)
Abstract:
JWARS V&V (a joint venture of Innovative Management
Concepts in Sterling VA and BMH Associates in Norfolk VA) has been responsible
for conducting the verification and validation (V&V) of the JWARS
simulation since September of 1997. This paper describes the lessons learned
during the conduct of the effort including: the JWARS V&V process, the
JWARS V&V Plan, reports delivered, and results to date. Special emphasis
is on the use of the DoD VV&A Recommended Practices Guide as a basis of
JWARS V&V planning and procedures and the evolution of the JWARS V&V
Integrated Product Team
Verification and Validation without Independence: A
Recipe for Failure
James D. Arthur and Richard E. Nance (Virginia
Polytechnic Institute and State University)
Abstract:
Verification and validation (V&V) is a prominent
technical area within simulation, attested to by the 12 tutorial sessions
(including five advanced) included in the past ten Winter Simulation
Conferences (WSCs). In recent WSCs the issue of Independent V&V (IV&V)
has drawn increased attention, with sessions examining the perceived lack of
use and little concern for the technique evinced within the simulation
community. The objectives of this paper are four-fold: (1) to examine the
current picture in software systems development, (2) to review the rationale,
role and expressed need for IV&V, (3) to identify the benefits attendant
in the insistence on the "independent" status of the activity, and (4) to
respond to the usual criticisms of negative impacts on cost and schedule.
While the treatment is couched in the more general software systems context,
we contend that simulations represent prime candidates for IV&V
application.
Verification and Validation of Object-Oriented
Artifacts Throughout the Simulation Model Development Life
Cycle
John T. Carr, III (Naval Surface Warfare Center ) and Osman
Balci (Virginia Tech)
Abstract:
The purpose of this paper is to present a series of
questions (or indicators) for assessing the verity and validity of the
artifacts produced during the entire object-oriented simulation model
development life cycle. Using modern object-oriented development processes,
artifacts developed in one phase flow seamlessly from those of the previous
phase. This provides forward and backward traceability between artifacts. This
inherent backward traceability has been exploited by tracing defects in
artifacts back to their defective ancestral artifacts. Questions are then
phrased such that when answered in the negative indicate the presence of
defects. Use of the Evaluation Environment software tool facilitates the
integration of the answers to the assessment questions and enables an overall
evaluation. The collection of questions can be useful for the verification and
validation of artifacts in any object-oriented simulation model development.
An Integrated Approach to Verification, Validation,
and Accreditation of Models and Simulations
Don Caughlin
(University of Colorado at Colorado Springs)
Abstract:
In an M&S-Based Systems Acquisition, computer
simulation is used throughout the development process not just as an analysis
tool but also as a development tool. In general, development of a system
capability using M&S-Based Systems Development will result in multiple
models or simulations to meet specific needs. The Verification, Validation and
Accreditation (VV&A) of each these tools is integral to M&S
development. Integrating V&V activities with M&S development and then
integrating the VV&A activities for all of the M&S resources that
support a program provides a cost effective approach to ensure the necessary
confidence in M&S results within the time and resources available. This
paper presents such an integrated approach to VV&A from a system
perspective and identifies the relationships between the M&S resources in
an integrated V&V program.
Validation of Trace-Driven Simulation Models: More
on Bootstrap Tests
Jack P.C. Kleijnen (Tilburg University (KUB)),
Russell C. H. Cheng (University of Southampton) and Bert Bettonvil (Tilburg
University)
Abstract:
‘Trace-driven' or ‘correlated inspection' simulation
means that the simulated and the real systems have some common inputs (say,
arrival times) so the two systems' outputs are cross-correlated. To validate
such simulation models, this paper formulates six validation statistics, which
are inspired by practice and statistical analysis; for example, the simplest
statistic is the difference between the average simulated and real responses.
To evaluate these validation statistics, the paper develops novel types of
bootstrapping based on subruns. Three basic bootstrap procedures are devised,
depending on the number of simulation replicates: one, two, or more
replicates. Moreover, for the case of more than two replicates the paper
considers conditional versus unconditional resampling. These six validation
statistics and four bootstrap procedures are evaluated in extensive Monte
Carlo experiments with single-server queueing systems. The main conclusion is
that bootstrapping of the simplest validation statistic gives the correct type
I error probability, and has relatively high power.
Soft-Commissioning: Hardware-in-the-Loop-Based
Verification of Controller Software
Harald Schludermann, Thomas
Kirchmair, and Markus Vorderwinkler (PROFACTOR Produktionsforschungs GmbH)
Abstract:
The basic idea of Soft-Commissioning (SoftCom) is to
test industrial control software by connecting a controller, e. g. a PLC
(Programmable Logic Controller) to a commercial discrete event simulator
(DES), which provides system reactions and sensor signals similar to the
behavior of real hardware, e. g. an industrial manufacturing line. In order to
establish a connection between simulator and PLC, a modular architecture was
developed. The basis of this modular system is a communication protocol common
to all members. The two basic modules are the I/O Devices Driver (IODD), which
is used to interface between the I/O hardware and the SoftCom protocol, and
the Simulator to real World Interface (SWI). The SWI is used to link the
simulator to the SoftCom system.
The Design of a Solid-State Physical Model of an
Automated System to be Used as a Test Bed for Control
Applications
Fernando G. Gonzalez, Alicia Helton, Douglas Helton,
Jeffrey Smith, Eileen Thompson, and Gerry Walterscheild (University of Central
Florida)
Abstract:
In order to develop, test, and validate control
software for managing automated systems, laboratories have traditionally
constructed experimental test beds using actual physical equipment (small
scale). These experimental systems typically occupy a large amount of lab
space, cost thousands of dollars to construct, and require considerable human
expertise to operate. Using dedicated micro-controllers (programmable logic
controllers), we have proposed the use of a solid-state physical model of an
automated system which faithfully replicates the operating characteristics of
an ensemble of physical equipment that would typically comprise an automated
system. In this paper we present the design of a solid-state physical model of
a Flexible Manufacturing System (FMS). Solid-state models have several unique
advantages over the traditional models. First, they are inexpensive and can
easily be replicated at other laboratories. Second, they can be easily
reconfigured to consider alternative scenarios. Third, they can consider an
emulated environment that is far more complex than those that are typically
addressed by models using actual equipment. Finally, they are totally reliable
and safe, and require minimal expertise to operate. This paper discusses the
design and operational characteristics of the solid-state model along with its
anticipated uses and current limitations.
Strategic Directions in Verification, Validation,
and Accreditation Research
Robert G. Sargent (Syracuse University),
Priscilla A. Glasow (The MITRE Corporation), Jack P.C. Kleijnen (Tilburg
University (KUB)), Averill M. Law (Averill M. Law & Associates, Inc.), Ian
McGregor (Brooks Automation, AutoSimulations Division) and Simone Youngblood
(DMSO)
Abstract:
Six simulation professionals present their views on the
directions that they believe that verification, validation, and accreditation
research should take. Two of the six are active verification, validation, and
accreditation researchers from academia, two develop industry simulation
models, and two work in verification, validation, and accreditation of
military simulation models. A number of areas and topics for research in
verification, validation, and accreditation are identified. It appears that
application domains of simulation models affect what topics need verification,
validation, and accreditation research.