Software System Safety Copyright Nancy G Leveson, July 2002
. Software System Safety Copyright Nancy G. Leveson, July 2002. c
Accident with No Component Failures VENT LAH GEARBOX LC CATALYST VAPOR COOLING WATER REFLUX REACTOR COMPUTER ypes of Accidents Component Failure Accidents Single or multiple component failures Usually assume random failure System Accidents Arise in interactions among components No components may have"failed Caused by interactive complexity and tight coupling Exacerbated by the introduction of computers
Accident with No Component Failures c ✌☎✄☎✍✎✄☎✏✎✠☎✑✝✒✔✓ ✂✁☎✄✝✆✟✞✠☎✡☎☛✄☎☞ LC COMPUTER WATER COOLING CONDENSER VENT REFLUX REACTOR VAPOR LA CATALYST GEARBOX Types of Accidents Component Failure Accidents Single or multiple component failures Usually assume random failure System Accidents Arise in interactions among components No components may have "failed" c ✌☎✄☎✍✎✄☎✏✎✠☎✑✝✒✖✕ ✂✁☎✄✝✆✟✞✠☎✡☎☛✄☎☞ Caused by interactive complexity and tight coupling Exacerbated by the introduction of computers. .
Interactive Complexity Complexity is a moving target The underlying factor is intellectual manageability 1. A simple"system has a small number of unknowns in its interactions within the system and with its environment 2. A system is intellectually unmanageable when the level of interactions reaches the point where they cannot be thoroughly planned understood anticipated guarded agains 3. Introducing new technology introduces unknowns and even "unk-unks Computers and Risk Computers and risk We seem not to trust one another as much as would be desirable. In lieu of trusting each other, are we putting too much trust in our technology?.. Perhaps we are not educating our children sufficiently well to understand he reasonable uses and limits of technology. Thomas B. sheridan
c ✌☎✄☎✍✎✄☎✏✎✠☎✑✝✒✖✧ Interactive Complexity ✂✁☎✄✝✆✟✞✠☎✡☎☛✄☎☞ Complexity is a moving target The underlying factor is intellectual manageability 1. A "simple" system has a small number of unknowns in its interactions within the system and with its environment. 2. A system is intellectually unmanageable when the level of interactions reaches the point where they cannot be thoroughly planned understood anticipated guarded against 3. Introducing new technology introduces unknowns and even "unk−unks." c ✌☎✄☎✍✎✄☎✏✎✠☎✑✝✒✖★ ✗✠☎☞✝✘☎✙☎✚✄☎✞✏✜✛☎✑☎✢✝✣✥✤✏✎✦ Computers and Risk We seem not to trust one another as much as would be desirable. In lieu of trusting each other, are we putting too much trust in our technology? . . . Perhaps we are not educating our children sufficiently well to understand the reasonable uses and limits of technology. Thomas B. Sheridan
A Possible solution Com Enforce discipline and control complexity Limits have changed from structural integrity and physical constraints of materials to intellectual limits Improve communication among engineers Build safety in by enforcing constraints on behavior Example(batch reactor) System safety constraint: Water must be flowing into reflux condenser whenever catalyst is added to reactor Software safety constraint: Software must always open water valve before catalyst valve Computers and Risk Stages in Process Control System Evolution 1. Mechanical systems Direct sensory perception of process Displays are directly connected to process and thus are physical extensions of it Design decisions highly constrained by Available space Physics of underlying process Limited possibility of action at a distance
c ✌☎✄☎✍✎✄☎✏✎✠☎✑✝✒✖✪ ✗✠☎☞✝✘☎✙☎✚✄☎✞✏✜✛☎✑☎✢✝✣✥✤✏✎✦ A Possible Solution Enforce discipline and control complexity Limits have changed from structural integrity and physical constraints of materials to intellectual limits Improve communication among engineers Build safety in by enforcing constraints on behavior Example (batch reactor) System safety constraint: Water must be flowing into reflux condenser whenever catalyst is added to reactor. Software safety constraint: Software must always open water valve before catalyst valve ✌☎✄☎✍✎✄☎✏✎✠☎✑✝✒✖✩ ✗✠☎☞✝✘☎✙☎✚✄☎✞✏✜✛☎✑☎✢✝✣✥✤✏✎✦ Stages in Process Control System Evolution 1. Mechanical systems Direct sensory perception of process Displays are directly connected to process and thus are physical extensions of it. Design decisions highly constrained by: Available space c Physics of underlying process Limited possibility of action at a distance
Stages in Process Control System Evolution(2) 2. Electromechanical systems Capability for action at a distance Need to provide an image of process to operators Need to provide feedback on actions taken Relaxed constraints on designers but created new possibilities for designer and operator error Computers and Risk Stages in Process Control System Evolution 3) 3. Computer-based systems Allow multiplexing of controls and displays Relaxes even more constraints and introduces more possibility for error But constraints shaped environment in ways that efficiently transmitted valuable process information and supported cognitive processes of operators Finding it hard to capture and present these qualities in new systems
c ✌☎✄☎✍✎✄☎✏✎✠☎✑✝✒✖✫ ✗✠☎☞✝✘☎✙☎✚✄☎✞✏✜✛☎✑☎✢✝✣✥✤✏✎✦ Stages in Process Control System Evolution (2) 2. Electromechanical systems Capability for action at a distance Need to provide an image of process to operators Need to provide feedback on actions taken. Relaxed constraints on designers but created new possibilities for designer and operator error. c ✌☎✄☎✍✎✄☎✏✎✠☎✑✝✒✔✓✭✬ ✗✠☎☞✝✘☎✙☎✚✄☎✞✏✜✛☎✑☎✢✝✣✥✤✏✎✦ Stages in Process Control System Evolution (3) 3. Computer−based systems Allow multiplexing of controls and displays. Relaxes even more constraints and introduces more possibility for error. But constraints shaped environment in ways that efficiently transmitted valuable process information and supported cognitive processes of operators. Finding it hard to capture and present these qualities in new systems