Andy Moyer Cleanroom Software Engineering What is it? Goals Properties of Cleanroom Cleanroom Technologies Case Studies Critiques Cleanroom Software Engineering: What is it? Set of principles and practices for the specification, development, and certification of software-intensive systems. Based on the work of Harlan D. Mills who was an employee of IBM in early 1980s. Influenced by Dijkstras structured programming, Nicholas Wirth on stepwise refinement, and David Parnas on modular design Methods based on two principles: Programs are rules for mathematical functions Software testing is sampling Where the Idea Came From The cleanroom model follows the ideas of manufacturing semiconductors in a “cleanroom” environment. Goals of the Cleanroom Process Main goal: Achieve or approach zero defects Other goals: Prevent defects Create correct, verifiable software Incremental development Properties of Cleanroom Spend a lot of time and money "up-front" preventing defects Use statistical methods to ensure quality Formally state and “prove” requirement needs Specification, development, and certification may be done by separate teams depending on the size of the project Cleanroom Technologies 1. 2. 3. 4. Incremental development under Statistical Process Control Function-Based Specification, Design, and Verification Correctness Verification Statistical Testing and Software Verification Incremental Development Under Statistical Process Control Used to reduce the risk associated with managing large systems by focusing on smaller, manageable subsystems of increments. Each increment is developed and tested separate from the whole system before integrating into the whole system. As increments are added, the whole system is tested. Some amount of end-user function used to obtain customer confirmation or clarification Incremental Development Under Statistical Process Control Increment should have these properties: Externally executable Contain all functionality of prior increments Benefits: Specification, design, and certification activities may be performed often Timely feedback can be obtained from customer Intellectual control over a system Cleanroom Technologies 1. 2. 3. 4. Incremental development under Statistical Process Control Function-Based Specification, Design, and Verification Correctness verification Statistical Testing and Software Verification Function-Based Specification, Design, and Verification Objectives: Expand the specification into implementation via small steps Maintain intellectual control by designing data and control structures at appropriate levels of abstraction The Three Boxes: Black Box State Box Clear Box Function-Based Specification, Design, and Verification The Box Structures Begins with an external view – Black Box Transformed into a state machine – State Box Fully developed into a procedure – Clear Box The Black Box A view of an object that hides data implementation and process implementation. It describes how a system responds to stimuli, usually in a formal specification language. No internal state is looked at Our good friend – Z (Zed) The State Box A view of an object that shows data implementation but hides process implementation. It describes how "state" information is transformed. Derived from the black box – first step to implementation This is represented as a finite state machine The Clear Box A view of an object that shows both data implementation and process implementation. The goal is to stepwise refine procedures and to prove them correct. Derived from the state box May introduce new black boxes defining major operations Creates the required responses (outputs) to stimuli (inputs) Stepwise Refinement of Specifications to Code Steps: 1. Expand clear box into lower level designs 2. Verify the correctness of the expanded designs 3. Expand low level designs into code Cleanroom Technologies 1. 2. 3. 4. Incremental development under Statistical Process Control Function-Based Specification, Design, and Verification Correctness verification Statistical Testing and Software Verification Correctness Verification The primary debugging process for Cleanroom Each box structure subject to correctness verification Objectives: Determine that the box structures correctly implements design Remove any errors that were introduced during development Completely review the code for completeness, consistency, and correctness Correctness Verification So what is verified? If the box structure behaves as defined Correctness of each refinement Response mapping defined in one step is preserved Cleanroom Technologies 1. 2. 3. 4. Incremental development process model Stepwise refinement of specifications to code Correctness verification of developed code Statistical Testing and Software Verification Statistical Testing and Software Verification Each increment of compilable functionality is statistically tested to: Ensure that it meets the quality standards defined by the development organization Certify a level of reliability that the product will deliver in the field Provide feedback for quality control and process improvement Statistical Testing and Software Verification Steps taken: 1. Plan the test ○ ○ ○ Stratification plan which describes each layer to be developed The sampling plan A description of which physical testing resources are to be devoted to each increment and each stratum Develop usage models required by the test plan 3. Develop and validate the usage distribution 2. Example Usage Model NASA Case Study In March of 1990 NASA preformed a case study of SEL Cleanroom versus their typical process Each group applied the process to the same project – a Coarse/Fine Attitude Determination Subsystem of the Upper Atmosphere Research Satellite. Completed system contained about 34,000 Source Lines of Code (SLOC) of primarily FORTRAN NASA Case Study-SEL Cleanroom Lifecycle NASA Case Study-Experience Comparison NASA Case Study Testers and developers are on completely separate teams Developers have no access to the mainframe computer for compilation and testing Developers rely on code reading instead of unit testing to verify correctness Testers use a statistical testing approach NASA Case Study - Results NASA Case Study - Results Failure Rate: Standard SEL process – 6 errors/KSLOC Cleanroom SEL procees – 3.3 errors/KSLOC The inexperience of the cleanroom team had no effect on the outcome Impact of unstable requirements lessened by concentrated effort in team reviews DoD Case Study The STARS program is a US Department of Defense (DoD) reasearch and development program Emphasizes: Process driven Re-use based Integrated software engineering environment DoD Case Study Typical Process Cleanroom Process Productivity Productivity measured at 121 loc/person month Failure rate: 23~27 failures/KLOC measured at 559 loc/person month Failure rate: 1 failure/KLOC DoD Case Study Important benefits: Improved Productivity Improved quality High staff morale Critiques In 1997, Beizer argued the elimination of unit testing is contradicting “known testing theory and common sense” Also, is it possible to find bugs without compiling/running code? References 1. Cleanroom Software Engineering. Retrieved from University of Texas at Arlington website: http://www.uta.edu/cse/levine/fall99/cse5324/c r/clean/page1.html 2. DACS, The Data and Analysis Center for Software. Retrieved January 5, 2009 Found at: www.dacs.dtic.mil/databases/url/key.php? keycode=64 References 3. Green, Scott et al. (1990). "The Cleanroom Case Study in the Software Engineering Laboratory: Project Description and Early Analysis". NASA. Retrieved from http://ntrs.nasa.gov/archive/nasa/casi.ntrs. nasa.gov/19910008271_1991008271.pdf 4. Prowell, Stacy et al. (1993). "Cleanroom Software Engineering: Technology and Process". Reading, MA: Addison Wesley Longman, Inc.