System and Software Safety Nancy G Leveson MIT Aero/Astro Dept Safeware Engineering Corp The Problem The first step in solving any problem is to understand it We often propose solutions to problems that we do not understand and then are surprised when the solutions fail to have the anticipated effect
. c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟✌☞ System and Software Safety Nancy G. Leveson MIT Aero/Astro Dept. Safeware Engineering Corp. Copyright by the author, June 2001. All rights reserved. Copying without fee is permitted provided that the copies are not made or distributed for direct commercial advantage and provided that credit to the source is given. Abstracting with credit is permitted. c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟☛✡ The Problem The first step in solving any problem is to understand it. We often propose solutions to problems that we do not understand and then are surprised when the solutions fail to have the anticipated effect. c
Accident with No Component Failures ( LAHGEARBOX LC ONDENSER CATALYST VAPOR COOLING REFLUX REACTOR COMPUTER eveson-4 The Problem Ty pes of Accidents Component Failure Accidents Single or multiple component failures Usually assume random failure · System Accidents Arise in interactions among components No components may have"failed Caused by interactive complexity and tight coupling Exacerbated by the introduction of computers
Accident with No Component Failures c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟☛✘ ✍✏✎✂✁✠✑✓✒✝✂✔✂✕✁✂✖ LC COMPUTER WATER COOLING CONDENSER VENT REFLUX REACTOR VAPOR LA CATALYST GEARBOX Types of Accidents Component Failure Accidents Single or multiple component failures Usually assume random failure System Accidents Arise in interactions among components No components may have "failed" c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟☛✗ ✍✏✎✂✁✠✑✓✒✝✂✔✂✕✁✂✖ Caused by interactive complexity and tight coupling Exacerbated by the introduction of computers. .
Interactive Complexity Complexity is a moving target The underlying factor is intellectual manageability 1. A"simple"system has a small number of unknowns in its nteractions within the system and with its environment 2. a system is intellectually unmanageable when the level of interactions reaches the point where they cannot be thoroughly planned understood anticipated guarded against 3. Introducing new technology introduces unknowns and even"unk-unks Computers and risk We seem not to trust one another as much as would be desirable. In lieu of trusting each other, are we putting too much trust in our technology ?.. Perhaps we are not educating our children sufficient/y we// to understand the reasonable uses and limits of technology. Thomas b. sheridan
c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟☛✩ ✍✏✎✂✁✠✑✓✒✝✂✔✂✕✁✂✖ Interactive Complexity Complexity is a moving target The underlying factor is intellectual manageability 1. A "simple" system has a small number of unknowns in its interactions within the system and with its environment. 2. A system is intellectually unmanageable when the level of interactions reaches the point where they cannot be thoroughly planned understood anticipated guarded against 3. Introducing new technology introduces unknowns and even "unk−unks." c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟☛✪ ✙✝✂✖✠✚✂✛✂✜✁✂✒✆✣✢✂✞✂✤✠✥✧✦✆☎★ Computers and Risk We seem not to trust one another as much as would be desirable. In lieu of trusting each other, are we putting too much trust in our technology? . . . Perhaps we are not educating our children sufficiently well to understand the reasonable uses and limits of technology. Thomas B. Sheridan
The Computer Revolution General Special Purpose Software = Purpose Machine Machine Software is simply the design of a machine abstracted from its physical realization Machines that were physically impossible or impractical to build become feasible Design can be changed without retooling or manufacturing Can concentrate on steps to be achieved without worrying about how steps will be realized physically Advantages disadvantages Computer so powerful and so useful because it has eliminated many of physical constraints of previous machines Both its blessing and its curse no longer have to worry about physical realization of our designs No longer have physical laws that limit the complexity of our designs
c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟☛✮ ✙✝✂✖✠✚✂✛✂✜✁✂✒✆✣✢✂✞✂✤✠✥✫✦✆☎★ The Computer Revolution General Special Purpose + Software = Purpose Machine Machine Software is simply the design of a machine abstracted from its physical realization. Machines that were physically impossible or impractical to build become feasible. Design can be changed without retooling or manufacturing. Can concentrate on steps to be achieved without worrying about how steps will be realized physically. Advantages = Disadvantages Computer so powerful and so useful because it has eliminated many of physical constraints of previous machines. Both its blessing and its curse: + No longer have to worry about physical realization of our designs. c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟✭✬ ✙✝✂✖✠✚✂✛✂✜✁✂✒✆✣✢✂✞✂✤✠✥✫✦✆☎★ − No longer have physical laws that limit the complexity of our designs
The Curse of Flexibility Software is the resting place of afterthoughts No physical constraints To enforce discipline on design, construction and modification To control complexity So flexible that start working with it before fully understanding what need to . And they looked upon the software and saw that it was good, but they just had to add one other feature Software Myths Good software engineering is the same for all types of software 2. Software is easy to change 3. Software errors are simply"teething" problems 4. Reusing software will increase safety 5. Testing or"proving" software correct will remove all the errors
c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟☛✯ ✙✝✂✖✠✚✂✛✂✜✁✂✒✆✣✢✂✞✂✤✠✥✧✦✆☎★ The Curse of Flexibility Software is the resting place of afterthoughts No physical constraints To enforce discipline on design, construction and modification To control complexity So flexible that start working with it before fully understanding what need to do ‘‘And they looked upon the software and saw that it was good, but they just had to add one other feature ...’’ c ✂✁✂✄☎✁✂✆☎✝✂✞✠✟✌☞✱✰ ✙✝✂✖✠✚✂✛✂✜✁✂✒✆✣✢✂✞✂✤✠✥✧✦✆☎★ Software Myths 1. Good software engineering is the same for all types of software. 2. Software is easy to change. 3. Software errors are simply ‘‘teething’’ problems. 4. Reusing software will increase safety. 5. Testing or ‘‘proving’’ software correct will remove all the errors