by Ludwig Benner, Jr
System Safety Society 10th International System Safety Conference
Dallas Texas July 18-22, 1991
© 1991 by Ludwig Benner, Jr. All rights reserved

Abstract One major function of safety management is the investigation of accidents and the issuance of safety recommendations for corrective safety actions, based on investigation findings. This paper evaluates the effectiveness of a typical recommendation process, at the National Transportation Safety Board, using predicted performance and system safety concepts as the bases for evaluation. The evaluation disclosed four serious deficiencies, including documentation of problems and associated risk levels, selection of recommendations and designation of priorities, misdirected follow-up efforts, and lack of accountability for the effectiveness of the recommendations. These deficiencies deprive decision makers of a basis for judging the urgency for action, the validity of a recommended action, or risk acceptance thresholds; delay or misdirect action; create trade-off analysis problems; force duplication of analyses; and contribute to unnecessary controversy, among other effects. Application of system safety concepts and principles could remedy the deficiencies identified. Introduction Investigating accidents and developing corrective actions to prevent similar accidents is a widespread practice, and a tradition of long standing in our society. Traditionally, society has relied on this process as its primary source of safety knowledge and initiatives for safety actions. Typically, the process starts with an accident or mishap investigation, followed by the determination of cause and the development of safety recommendations[1]. Once made, a safety recommendation takes on a life of its own during its development, its assessment by others, and its implementation. The basic concept underlying this approach is to learn from experience so we can prevent or control similar experiences. Unfortunately, the safety-related experience may involve a wide range of losses before we learn anything. This "have accident/investigate/recommend" process is wholly retrospective. It requires that you have an accident, investigate it, draw conclusions about cause, and then come up with safety recommendations. The assumption that we will make good recommendations if we find the right cause seems to be a given in most organizations I have observed. In this process, investigators are usually tasked with developing the recommendations that somehow flow from the investigation findings. This may be unfair. Investigations to determine what happened are retrospective; the investigator must determine what happened from "historical" data. Safety recommendations, on the other hand, deal with prospective matters, and must therefore be predictive in nature. By common agreement, recommendations are designed to influence future occurrences. Thus, the development of safety recommendations involves entirely different data, predictive analysis, effectiveness assessment and quality control processes than a backward-looking investigation effort.[2] Complaints and differences about recommendations are fairly commonplace, but complaints about the process are rare. Does that mean the process is performing satisfactorily? The answer depends on the criteria you chose for your evaluation, as will be seen. The Safety Recommendation Process Little has been written about the process by which safety recommendations evolve during or after an investigation. For example, statisticians analyzing safety statistics have no documented process for identifying and proposing recommended actions, the recommendations being left to the professional judgment of the investigator after a "problem" is identified from data collected during investigations of accidents. Wood has written on how investigators develop recommendations[3] for the International Society of Air Safety Investigators, (ISASI) whose members' explicit goal is accident prevention through investigation, but does not offer a process description. Bruggink and Fritsch make the point to ISASI members that the safety recommendation process is far from standardized, and for that reason is not as effective as it should be, but do not describe the processes.[4] In the system safety field, Hammer offers a set of actions to "achieve safety" but does not describe the recommendation process[5]. MILSTD 882B does not address this process explicitly either, although it requires recommendations and certain priorities for types of actions to control hazards.[6] It is important to differentiate between the complete recommendation process discussed by Bruggink and Fritsch, and that part of the process which produces the recommendation, as discussed by Wood, which should be termed the recommendation development process. This paper considers the entire process, beginning with the establishment of the objectives of a recommendation, and ending when a recommendation is in place in the organization which implemented it. A case study of one of the preeminent organizations established for the purpose of investigating accidents and making safety recommendations, the National Transportation Safety Board (NTSB), is used to illustrate the typical recommendation process. Most attributes of the NTSB investigation and recommendation development process are incorporated into or adapted by similar investigating organizations, and many attributes of the process can be observed in other organizations' practices. Thus, the NTSB provides an informative case study for evaluation purposes. However, the findings should not be considered unique to the NTSB. The NTSB Mandate The NTSB is an agency of the US Government established by Congress and operating under Public Law 93-633 Part II, Independent Safety Board Act of 1974, and subsequent authorization bills. Congress formed the NTSB in that legislation ". . . to promote transportation safety by conducting independent accident investigations and by formulating safety improvement recommendations." In the execution of its mandate, the NTSB investigates around 2700 accidents annually, in all modes of transportation. From its investigation findings, it determines cause or causes, accumulates accident data, performs safety studies and, under Section 304(a)(3) of the law, issues safety recommendations. The recommendations it issues have no binding effect on the recipients. The NTSB has no enforcement power, so each of its recommendations must stand on its merits. How do the investigators develop recommendations? Overview of NTSB The NTSB does not investigate all transportation accidents. It selects major accidents and incidents, usually of national interest, to investigate with Central Office teams, and publishes soft-cover reports of those investigations. Accident investigation reports contain the facts of the accident, its analysis and determination of cause(s), and recommended actions to prevent their recurrence, typically based on the cause(s). The NTSB also investigates many less publicized accidents with field staff, acquiring a large quantity of data to support its accident data base and its determination of cause of each accident, and to identify potential safety recommendations. Staff may perform studies using NTSB's or other databases to identify safety problems, and propose recommendations. Staff members can propose any safety recommendation they elect to develop. Proposed recommendations pass through an internal review process that is somewhat similar to a peer review process. As is typical of the many processes observed by the author elsewhere, specific technical analysis procedures for recommendation development and assessment have not been established for proposed recommendations. Reliance is placed on individual technical judgments and internal reviews before proposals become NTSB recommendations. The internal NTSB staff structure, technical capabilities and modal coverage have expanded over the years, particularly in the 1970s. However, the general approach, scheme and objectives of the accident investigation and recommendation processes have changed little since the 1930s when NTSB's predecessor organization was established as a Bureau in the Civil Aeronautics Board. One area of major change has been in the NTSB's recommendation process, instituted about the time Public Law 93-633 was passed making the NTSB an independent Federal agency. This change was precipitated primarily by an aviation accident near Paris with a large number of casualties that resulted in part because NTSB recommendations following a similar incident had not been acted on promptly. That accident resulted in the adoption of a new recommendation follow-up and tracking process by the NTSB. The nature of events at the time heavily influenced the content and form of that process, and its essential elements remain in effect today. NTSB Recommendation Policy The NTSB's process for the development, issuance, follow-up and closing of its recommendations is described in its Order NTSB 82, of June 11, 1987. The Policy section of the Order has this to say about the NTSB's recommendation development process. ". . . Safety recommendations are fully justified and adequately supported by the findings of accident investigations or conclusions reached as a result of safety studies". Although the Board is not staffed to conduct extensive cost-benefit analyses, a "reasonableness test" is applied during the development of every safety recommendation. The estimated resources needed to carry out the recommended action are considered and the impact of this potential resource commitment on the addressee is discussed, both in-house and with appropriate outside parties whenever deemed useful." Full consideration of the expected results of the recommendation are given during the development process. What gain is expected? How is progress and completion of a recommendation action to be measured? ". . . Each Safety Recommendation is tracked from the date of issue until the closing of the file. Safety Recommendations receive some form of follow-up activity at least once every 12 months."[7] Basis for Recommendations. Note that a recommendation is expected to be supported by the investigation or special study findings. One obvious justification to support a recommendation is the fact that an accident or accidents happened, so there must be need for action. The basis for a recommendation is the prevention of similar accidents in the future. The logical basis for a recommendation is contained in reports of accident investigations and special studies, in the sections of reports describing the facts, analysis and conclusions supporting the recommendation. That narrative material makes the argument for the recommendation. As a recommendation evolves, an NTSB Proposed Safety Recommendation Analysis and Justification Form requires the recommendation developer to document various aspects of the analysis and justification process. For example, it requires that the problem addressed by the proposed recommendation be documented, along with other alternative approaches considered. Analysis questions posed in Appendix B from NTSB Order 82 are summarized below. Proposed Safety Recommendation Analysis and Justification Accident location and date: (or special study title) Text of proposed recommendation: Proposed addressee: Problem addressed by the proposed recommendation: (state page of report where facts, analysis and conclusions are found in support of this recommendation) Describe any alternate approaches considered: If regulatory action recommended, discuss reasons for this approach over voluntary industry action or less formal "guideline" approach: What organizations have capability to implement this recommendation? Why was proposed addressee chosen to receive this recommendation? What constituency will benefit from proposed action? (E.G. Public, users, operators, management, etc.) What are previous related safety recommendations issued related to this problem? Should old recommendations be closed in any way? (Attach list) What is the estimated first response from the addressee? How is completion of action to be measured? What is the final result of the recommended action to be? Other comments to justify this proposed recommendation. The extent of their use is not apparent. Most of the documented information does not appear in NTSB reports, or the sample write-up contained in Appendix A of the Order. Answers to question 4, 9 and 12 may be addressed obliquely in reports, but are not crisply stated if indeed they are present. For example, a recent release of the NTSB's "Most Wanted" safety items is devoid of this kind of persuasive information.[8] More typically, the NTSB discusses safety problems in terms of "issues" in its outputs. For example, in its 1988 Annual Report it describes issues in each of the modal discussion sections. Procedures and justifications for selecting the issues from among all the safety problems it viewed since the last similar list are not described in the sense suggested by the questions in the Analysis and Justification form above. The report gives no indication of the risk levels associated with each issue. The issues may be based on the predicted possibility of recurrence of a similar accident described in the NTSB's recommendation follow-up requirements, but the basis for their selection (and predicted safety value if implemented) is not provided. Recommendations Produced This process and predecessor recommendation processes have produced over 8000 NTSB safety recommendations. A tabulation of the NTSB's recommendations, by transportation mode, is shown in Table 1. Table 1. Summary of NTSB Recommendations through July 25, 1990 Mode Total Recommendations Aviation 2714 Highway 1297 Marine 1591 Railroad 1412 Pipeline 921 Intermodal 170 Totals 8115 Source: NTSB Office of Safety Recommendations During the investigation process, any staff member can propose any safety recommendation, if is "justified and adequately supported by the findings of accident investigations or conclusions or reached as a result of safety studies." Each proposal must be assigned a "follow-up priority" before it is launched into the development or Review Board steps on its way to the NTSB Member-level docket item approval process. When the NTSB Members adopt a recommendation, one of three follow-up classification is assigned to the recommendation, as shown in Table 2. Table 2 NTSB SAFETY RECOMMENDATION FOLLOWUP CLASSES Safety Recommendation Class Summary of criteria for assignment in NTSB Order 82 Class I- (Urgent Action) . . . completion of action is necessary to avoid imminent loss due to similar accidents that could occur under the same circumstances if a deficiency is not corrected. (Follow-up contact is initiated within 10 days, and letter couched in urgent terms goes to addressee[9].) Class II (Priority Action) . . . a high priority should be set to avoid a probable loss due to similar accident that could occur under the same circumstances if a deficiency is not corrected. (Follow-up is initiated within 90 days.) Class II (Longer Term Action) . . . an immediate or near future similar accident is not likely if an identified deficiency is not corrected, or where the desired action cannot reasonably be expected to be accomplished within 2 years. (Goal is to close out in 5 years.) While not explicitly stated in NTSB documents, the Order indicates that the main purpose of the follow-up system is to indicate follow-up urgency by the NTSB for each recommendation, based on criteria for assigning a follow-up class. Table 3 on the following page shows the follow-up classes assigned to recommendations approved by the NTSB since 1975. Table 3 Distribution of NTSB Recommendation Follow-up Priority Classifications Class I Urgent Class II Priority Class III Long term Total See note 1 Transport Mode Mode Number % of mode Mode Number % of mode Mode Number % of Mode Mode Number % of Total Aviation 242 12% 1593 81% 143 7% 1978 32% Highway 63 7% 824 87% 65 7% 952 15% Intermodal 9 6% 132 83% 18 11% 159 3% Marine 19 1% 1286 95% 44 3% 1349 22% Pipeline 128 19% 490 72% 64 9% 682 11% Railroad 135 12% 991 87% 16 1% 1142 18% Total 596 9.5% 5316 84.9% 350 5.6% 6262 Note: Excludes 1853 recommendations issued prior to mid 1975 to which no follow-up class was assigned. Source: NTSB Office of Safety Recommendations. Evaluating the Recommendation Process Is this kind of process effective? The answer depends on criteria used to judge the process effectiveness. The criteria "fully justified" and "adequately supported," used by the NTSB for this recommendation process, are too abstract to be considered useful criteria for this purpose. However, such ambiguous or abstract kinds of criteria are typical for accident investigation programs observed by this author[10]. How then can the process be evaluated? Results-based Effectiveness Evaluation A common-sense criterion or test is to ask whether the recommendation process is achieving what is expected of it. The mandate for the organization is to promote safety. What safety has been promoted? What have been the predicted and the actual safety results (gains and losses) achieved by these 8115 NTSB recommendations? More specifically, given the mission of the NTSB, what have been the transportation safety results or gains and losses, or the number of problems "solved?" How many solutions were attributable to the recommendations? Nobody knows. First the problems are not defined in a manner that lends itself to measurement of the correction. Secondly, with few exceptions,[11] the safety results of the recommended actions - predicted over the remaining life cycle - are not documented in observable terms, and thus remain unknown. Determining the recommendation effectiveness record would take an extraordinary amount of research because the generation of relevant safety performance data has not been a part of the NTSB or related processes.[12] The uncertainties shown by these questions can be observed in most investigation programs that produce safety recommendations. In the case of the NTSB, careful examination of the NTSB's recommendation follow-up classification process suggests the reasons. That process is geared to the possibility of recurrence of similar accidents.[13] While question 12 calls completion of action to be measured, the measure of effectiveness is suggested to be recurrence of similar accidents over time. Effectiveness is not predicted in terms of the exposures, and the combination of the probability of recurrence and the probable consequences, or the risk. The NTSB recommendation process does not require or report, in terms managers or the public can understand, the observed risk[14] associated with a safety problem it finds. Further, the predicted reduction in risk that will be achieved if the recipient of a recommendation does what the NTSB proposes is not reported either. By not specifying the risks attached to either the safety problem or proposed action, the safety value of the recommendations can not be judged, and priorities for its implementation must be inferred. What then is the basis for determining if recommendations should be implemented, and if so, in what order? This raises questions about a national policy perspective. If the relative priority among 8115 recommendations can not be determined, how is the competition for resources to implement those actions supposed to be resolved except on an ad hoc basis? Pity the recipients! This deficiency escalates the probability of misdirecting energies and resources to "the squeaking wheel of the day," which may be a low-risk problem, rather than toward serious, properly defined, long term high-risk problems. Similarly, for an industry or regulatory agency perspective  where competition for or concern about resources existed, the relative priorities among the 2714 aviation safety recommendations, for example, can not be determined. Are marine risks really as low compared to pipeline risks as suggested by the percentage (1% vs 19%) of Class I Urgent priority recommendations in these modes? What actions should be funded - marine or pipeline, and when, if the basic desire is to be responsive to the most important safety needs with available resources? The absence of safety risk data ultimately leads to poorly informed or misdirected judgment calls, because judgments are based on differing experiences and unclear safety improvement values. Alternatively, it leads to avoidance efforts based on such judgments.[15] Without an indication of the changes in risks, what technical basis remains to resolve uncertainties that arise in such circumstances, in any organization? System Safety-based Effectiveness Evaluation System safety concepts and principles can provide another set of criteria for evaluating recommendation effectiveness. Published system safety materials are used for these criteria.[16] System Safety Evaluation Criteria Several key system safety concepts and principles are directly relevant to the development of safety recommendations for actions to eliminate or control risks, and the assessment of the safety recommendation process. For example the following have direct application to the recommendation development process: 1. Life cycle risks. Base your actions on a risk-based view of safety performance, taking into consideration management of the system performance over the full life cycle of the system, including disposal at the end of the system life. Recommendation effectiveness should be considered over the same period. 2. Working Group or Team effort. Organize your system safety efforts to involve the key players who will be involved in acceptance of or acting on the risks found, to get the benefit of their knowledge and skills, and to preview your findings as they evolve. Recommendation development should similarly be an open process. 3. Predictive analytical technology. Predict the effect of your actions before you act. Apply the best predictive safety analysis technology to systematically (1) define your system over its life cycle, (2) discover and define the safety problems in the system and the relative risks they pose, (3) identify applicable strategies and all feasible action options for the risks that require reduction, and (4) evaluate relative effectiveness of each option before you select the "best" option. Recommendations should be developed using the best available safety technology, rather than an ad hoc process dependent on experience or good judgment. 4. Tailored, timely analyses. Tailor your analyses to the system and the demands indicated by your analyses, doing your analyses when you can make changes on paper during design, a much more efficient and expeditious approach than retrofits based on accidents. Recommendations should be tailored to the problem as it is defined and articulated from the accident description. 5. Package for decision making. Document the risks you found, and the safety improvement predicted for the action you propose. Then package your findings so managers with finite resources can balance safety risk reductions for proposed actions with other risks and resource demands confronting them. Recommendations should offer decision makers choices, and the safety benefits to be weighed and weighted against other resource and action demands. 6. Use proven control strategies. Take advantage of proven safety and risk control strategies to eliminate or reduce risks to acceptable levels that can be maintained over the life cycle of the system, and document the rationale. Strategies help identify and rank order potential recommendation effectiveness. 7. Monitor your hazard predictions. Document your predictions and findings so you or anyone else can monitor system performance over its entire life cycle, and then set up a plan to accomplish the monitoring. Recommendations should be stated in a way that permits the monitoring of their implementation, and the assessment of their actual vs predicted effectiveness. 8. Track problems until resolved. Document your "lessons learned" so you and others engaged in similar activities do not have to repeat the learning process, and losses that would accompany the "re-learning" experience. Recommendations should be based on clearly defined and readily accessible problem statements, such as those found in a Preliminary Hazard Analysis list, so they can be widely disseminated for training, monitoring, supervisory or analytic activities, among other uses. System safety technology, applied predictively, offers a structured, disciplined management and technical process for defining safety problems and developing predictive safety action recommendations. Safety problem discovery and definition are one of the goals of the predictive analytical technology used in the system safety process. In system safety documentation, such as a Preliminary Hazard Analysis, each safety hazard or problem is listed. Each problem on the list is typically followed by an estimate of the risk level it poses, using a Risk Assessment Coding scheme, and by one or more action options that would eliminate or control the risk (e.g., a safety recommendation) along with the new RAC if the option is implemented. That process can be used as a yardstick for assessing the recommendation development process. NTSB Recommendation Process Evaluation A review of the NTSB recommendation process as described in its Order 82 using these criteria suggests the following observations: Table 4 . Application of System Safety Criteria for Evaluation of NTSB Recommendation Process Criteria Observations 1. Life cycle risks No requirement to consider life cycle or life cycle effects or risks. 2. Team effort (SSWG). Developers encouraged to discuss resource needs with addressee, but Order does not require or encourage a team recommendation development effort among investigation or study parties for its recommendations. NTSB provides Safety Proposal Review Board for field-initiated safety recommendations. 3. Proper analytical technology. Does not specify the analytical methodology used to define problems, generate recommendations, or predict safety effects of its recommendations. The expected system operation on which the recommendation is based is sometimes reported in narrative form. The safety strategies are not required to be disclosed. Options considered and unknowns are rarely disclosed or acknowledged. 4. Tailored, timely analyses. Does not address tailoring issue; timeliness of analysis partially addressed. 5. Package for decision making. Does not require succinct statement of problem or need, risks if no action is taken, options considered to resolve problem, predicted risks if each option is considered, or safety strategy being employed for each. 6. Use proven control strategies. Does not require disclosure of safety strategies on which the recommendation was based. 7. Monitor your predictions. Does not require monitoring of effectiveness; requires monitoring of action taken rather than effectiveness of action taken. 8. Track problems until resolved. Does not require succinct statement of problem to accompany recommendation, nor close recommendations based on observed safety results achieved. Overall, Table 4 suggests that the process is not very effective, by most system safety criteria. Each criterion raises an area of concern about the process. However, basic deficiencies, from which others flow, seem to be: failure to succinctly summarize and document safety problems and the level of risk they pose. failure to use a risk-based approach to recommendation selection and assignment of priorities in the recommendation development and follow-up process. focusing of the follow-up effort on action taken rather than safety results achieved. lack of accountability for safety results achieved by the recommending organization. Problem and Risk Level Documentation Failure to document and publish safety problems or hazards in a concise form, with indicated risk levels as is done in system safety tasks, creates several problems. First, the present form impedes dissemination of knowledge of the problem in a timely manner reflecting the level of risk. By leaving the gravity of a problem to the judgment of report readers, the process suggests that implementation of the recommendation solves the safety problem. This deficiency also deprives implementing personnel of a sense of the urgency for maintaining an implemented action over the remaining life cycle of the system. The risk level before implementation would exist every time the control is not functioning, any time during the remaining life of the system. In the absence of indicated risk levels, operating personnel are deprived of the rationale and motivation to initiate and maintain a long term control monitoring effort for actions they implement. This is not to say that risks are ignored in the recommendation process. For example, an analysis of the contents of NTSB's Order 82 and its investigation reports suggests that the NTSB considers risk involved in accidents it investigates in several ways. It could be argued that the risk posed by the problem the recommendation addressed had an influence on the follow-up class NTSB assigned to its recommendations. Based on this premise, the risks addressed by the 6262 recommendations assigned a follow-up class under this process are summarized in Table 5. However, in the absence of specific instructions relating the risks to the classification, the implicit risk levels in Table 5 should be viewed as speculative. Inconsistencies among modes and Table 5 suggest that no risk-based criteria are intended for recommendations. Table 5. Possible Risk Levels Inferred by NTSB Recommendation Classification. Follow-up priority Implicit risk level % of recommendations Class I Urgent Action High risk 9.5 Class II Priority Action Moderate risk 84.9 Class II Longer Term Action Low risk 5.6 The NTSB also considers risks when it makes interim recommendations to address any urgent problem it finds in an investigation, by the assignment of urgent processing priorities, as described above. It could also be argued that when the NTSB issues an interim recommendation, it considers the risk involved so unacceptable that Members want to act quickly to control it, although the Order uses different terms. The time limits for preparation of an NTSB decision package for its consideration of an "urgent" safety recommendation proposal is 15 days after assignment of a proposal to one of its internal Bureaus, 75 days for a "priority" proposal, and 120 working days for a "longer term" proposal. However, the follow-up period is substantially longer, as shown in Table 2. These time limits seem to be predicated on the organization's desire to get action from the recipient rather than getting the problem corrected, perhaps on the assumption that its recommendation will solve the problem. The 90 day response period allowed under the law further diminishes the urgency of a response even to an urgent problem. Because the risk level posed by the problem is not made explicit the impetus for action is driven by the focus on closing the recommendation file, rather than the solved problem. An "Urgent" recommendation follow-up classification conveys a different message to recipients than a risk level rating denoting a catastrophic risk level for a hazard or safety problem. Focusing on action taken on recommendations rather than the safety problems or hazards and the reduction in risk needed would introduce a strong tendency to gloss over problems if you have no recommendation to offer. Many field accident investigations produce no recommendations[17]. Recommendation Selection and Priorities During investigations, alternatives or options to better control risks are usually identified. Some investigation methodologies produce more options than others. When accidents are reported, however, typically only one recommendation per problem survives. This deprives managers of choices, and the ability to weigh and weight trade-offs among safety risks, resource allocation risks, schedule risks, technical risks, social considerations, and other factors that should be considered in arriving at a corrective action decision. The NTSB Order provides no formalized process for reporting corrective action choices, their derivation, and the relative effectiveness of each option it considered, or the other factors by which it arrives at its recommendations. Thus, the NTSB circumvents the trade-offs and hard choices among different ways to fix a problem faced by recipients of its recommendations who must act on them. The large number of recommendations it makes suggests that rather than dealing with options and making its choices on the relative merits, it recommends everything it can think of, including recommendations that might be characterized with a "nice to do" label[18]. Additionally, the conventional approach of recommending a single action implies that the recommender of an action, like the NTSB, is capable of determining the correct solutions to the problems it defines in its investigations. If the options considered and new risks are not reported, and the tradeoffs in the selection of the action recommended are not presented, operators wanting to solve the problem may have little assurance that the proposed action is truly the "best" remedy. Such uncertainties breed hesitation, duplication of analyses, delay and ultimately controversy.[19] Again, pity the recipients! When given a choice, the "best" corrective actions to reach a specified risk level are often best identified, selected and implemented by line personnel closest to the operations or resources - the regulatory agencies or operators who must identify and then balance all the factors that must be weighted and weighed in arriving at the "best" action, and live with the results. But the NTSB, like many other recommending organizations, does not consider economics, for example, in selecting the "best" action to recommend. By shifting its focus from its recommendation to the problem definition and risk levels involved, the NTSB could trigger the creativity of a lot more people who have a direct interest in fixing the problem in the "best" way. From a corporate or national policy perspective, focusing on the problem and presenting safety effectiveness-ranked options with indicated risk reductions would seem far superior to the present system of focusing on a recommendation. Misdirection of Follow-up Efforts The gravest consequence of the emphasis on implementation of recommendations, rather than assessment of their effectiveness during follow-up efforts, is the diversion of resources from substance to process. Overall, this kind of approach indicates that a recommending organization judges its own performance based on how often and how fast its recommendations are implemented, rather than the degree of improved safety (or reduced risk) the recommendations achieve[20]. And that is what will be recorded and measured. It could be argued that the NTSB's recommendations will prevent all similar accidents if implemented, e.g., achieve 100% safety for that kind of accident. Continuing accidents suggest otherwise. Unfortunately, verification of that argument reflects a retrospective approach to safety, and with the advances in safety analysis technology over the past 20 years, has little merit since accidents have to happen before the NTSB gets involved again. Further, they can continue to happen after it closes its recommendations, if either the recommendation was wrong, or if the monitoring action taken over the remaining life of the system is misdirected or ineffective. This deficiency can be exacerbated by the way a safety problem is stated, as when a judgment of cause is stated by the investigator. If a recipient's action is not readily seen to correct the "cause" the manager is likely to encounter trouble - or possibly even bad press if "the public interest" is involved- for not implementing a recommended action. The recipient's choice of the "best" overall action must weigh ALL risks, including financial, social, schedule and political risks. The result may be to select the investigator's second or third ranked preference. However, if the recipient is offered no options, the investigator places the manager in position where a third-rated action may have to be taken because of the cause problem. Accountability for Recommendations Made That possibility reveals another deficiency in the present processes. If it focuses on the recommendation rather than the results achieved, the recommending entity has no incentive to act responsibly or in any way to be held accountable for its decisions and actions because it does not have to predict the expected safety improvement when its recommendation is adopted. The NTSB, for example, claims to apply a "reasonableness test" to its recommendations, but since the predicted safety effects of that reasonableness test are not documented, others must guess at that portion of the "test." Unfortunately, the recipient of a recommendation does not have that luxury. Again, pity the recipient! In fairness, the investigating entity should describe options it considered, and the pros and cons or trade-offs considered in selecting the proposed action. A better alternative would be to just define the problem adequately and suggest options to demonstrate that it is possible to control, and leave the actual action to the recipients. Most investigators will argue the position that the trade-offs are for the recipient to analyze, especially the economics of the action. This paper does not argue that point. What is unacceptable for the recommending entity - whether at the local plant level or at a body that views itself as the national overseer of transportation safety - is for it to shirk the routine prediction and publication of at least the safety effects of its recommendations.[21] The potential for arbitrary and capricious making of recommendations without such assessments demands that the choices and trade-offs considered be documented, preferably including the safety level before and the predicted safety level after a recommendation is implemented. If the risk levels and options are being properly considered, documentation should be no problem. Alternatively if they are not being considered, they certainly should be before a particular option to recommend is chosen. Any other basis for judging performance effectiveness is suspect, at best. A Practical Alternative Correction of these problems requires two steps. The first is a change in an organization's approach to its recommendation process. The second is implementation of the change with a practical system of assigning relative risk estimates to the problems it identifies and recommended action it proposes. Both steps are readily achievable by the application of system safety concepts, principles and tools. Changing the Approach A viable alternative to the present investigation approach is to succinctly state each problem or hazard discovered, its urgency as shown by estimated risk levels, and potential action options to control the risks, as in a PHA or other system safety analysis documents. Then assemble the information in a decision package. The content of that decision package would support proper consideration of the risks and options for corrective action. It would be directed to and used by the persons who are responsible for the risk levels achieved by their system, or by a regulatory agency if regulation is needed to insure action and fairness. To illustrate the approach, any organization -including the NTSB- could define the operation and document the specific safety problems it finds during an accident investigation. It would define the problem in terms of the current estimated risk level if no action is taken (which it considers unacceptable as evidenced by its recommendation). It would then report the options it has identified to demonstrate that the risk it has defined is controllable, showing how much each option is expected to reduce the safety risk. Then it could simply recommend that the operator or regulator act to eliminate or reduce the risk to an acceptable level. The estimated current risk level would indicate the priority for action (and, concurrently, follow-up action by NTSB and the operator or regulator.) However the selection of the "best" option, would be left to others who must consider more than the safety consequences of the actions.[22] Then investigating organizations such as the NTSB would be free to establish a specific follow-up process and monitoring plan for each safety problem, rather than its recommendation, to see that the corrections introduced really fixed the problem as it was defined. Occasionally, the recipient might argue about the estimated risk level. The dialogue flowing from these differences could be used to refine either the estimating process or the levels that are acceptable. At other times, the recipient organization might argue that it is willing to accept the risk. There isn't enough money in the world to fix ALL safety problems by tomorrow morning! Resolution of this kind of difference could be aided by another system safety concept - the system safety working group. With this approach, representatives from major interest groups participating in the recommendation development process, and airing differences early in a process, would help the recommending individual reach better decisions. Implementing Changed Approach Actual implementation of these changes involves several changes in most organization's operations. The first change is a need to refine the problem definition process to incorporate problem statements or "judgments of need[23]" into the investigation and recommendation procedures. The judgment of needs approach seems especially well suited for organizations which may have to deal publicly with the accident and recommendation response issues. It also seems well suited to the societal decision making demands of Federal agencies, such as decisions by the NTSB and regulatory agencies. A second change is to institute a requirement for investigators and others who propose recommendations to analyze and assign risk levels to problems or hazards and effects of corrective actions. A practical method for the investigators and recommendation developers is immediately available. It is a Risk Assessment Code assignment process widely used in the Department of Defense and other agencies and their contractors. It is described in greatest detail in MILSTD 882B, the Department of Defense's standard for System Safety analysis. The risk assessment code (RAC) assignment process has been used widely enough that it has proven to be a viable way to distinguish relative risk levels, to aid in the determination of the acceptability of risks, and to assess the relative safety value of safety actions to control unacceptable risks. By adopting the RAC system, organizations can establish problem risk level ratings for the problems they discover and the corrective action options they define in their investigations. Use of RACs could be expected to have several effects. The first effect would be to provide an indication of the relative significance of each problem the investigation has defined as a result of its investigations. For example, the NTSB could state a need and assign a RAC number to the risk level represented by that need. This would enable everyone with an interest in the problem to see and to use the risk level for determining the urgency of their own response and long term monitoring actions. A second effect would be that the "before and after" RACs would provide a basis for comparing the relative safety effectiveness of actions proposed by the recommendation developer or other options offered by other parties. This would overcome the stifling effect of single options as now practiced, or the proliferation of recommendations that sound nice, but would have little discernible effect on the risk levels. A third effect would be to provide a reasoned, relatively consistent basis for recommending staff follow-up actions to spur action on problems or solutions, if required. The urgency of follow-up action, and possibly the design of the follow-up plan itself[24], are far different for a RAC 1 (very high) risk than for a very low RAC 5 risk. This would help shift the emphasis from just answering the recommendation to solving the safety problem. It would also change the nature and approach to data gathering for the purpose of monitoring the achievement of the reduction in risk. A fourth effect would be to provide a basis for evaluating the effectiveness of the recommendation development organization itself. This effectiveness should be judged by the support it provides to line managers in controlling the risks, rather than the count of recommendations with a positive response by the recipient. By comparing predicted with actual results, the effectiveness of both the recommendation and the recommending organization become visible. For example, as an investigating and recommendation development organization, under the new approach the NTSB would not be judged on how many recommendations it made or had accepted. Rather, it would be judged in the context of a knowledge-building organization, on the importance of the safety risks it discovers and defines, and the risk reduction effectiveness of its recommendations. The sting of such accountability could have a dramatic effects on any recommendation-producing group in any organization. It can be argued, with justification, that the RAC method is less than a perfect solution to estimating risks. However, RAC use in decision making is already so widespread that this argument should not be allowed to interfere with its adoption to overcome the deficiencies described. Implementation of the RAC assignment process may result in frequent negotiations about the RAC that should be applied in particular instances. In the example, since the NTSB is a deliberative body of Congressionally-approved Presidential appointees, the Members' negotiations among themselves or with others could produce valuable criteria for the "right" RACs over time. Conclusion Processes for the investigation of accidents and development and implementation of safety recommendations, as presently practiced, have numerous deficiencies when judged by safety performance and by system safety criteria. The evaluation disclosed four serious deficiencies: documentation of problems and associated risk levels, selection of recommendations and designation of priorities, misdirected follow-up efforts, and lack of accountability for the effectiveness of the recommendations. These deficiencies deprive decision makers of a basis for judging the urgency for action, the validity of a recommended action, or risk acceptance thresholds; delay or misdirect action; create trade-off analysis problems; compel duplication of analyses; deny needed information to support performance monitoring efforts; misdirect assessments of recommendation effectiveness; create problems for managers; and contribute to unnecessary controversy, among other consequences. By applying selected system safety concepts, principles and tools, the accident investigation recommendation process can be significantly improved in any organization, and can help overcome the deficiencies cited. These actions include documenting problems in the manner of system safety analyses outputs to shift the focus onto the problem to be solved, assigning RACs to each safety problem or hazard identified to indicate urgency for action and monitoring, offering options and new RACs for each option to show safety value of action choices, planning future safety performance monitoring to assure that the action has controlled the risk over the system's life cycle, and evaluating the recommending group's performance on the basis of risk reductions achieved, rather than recommendation actions closed or accepted. In summary, to improve the investigation and recommendation process, focus on safety problem definition, recommendation options, risk level documentation, and safety performance monitoring follow-up efforts, and relax the emphasis on "cause" and recommendations accepted. About the author: Ludwig Benner, Jr. PE, Executive VP, Events Analysis, Inc. , 12101 Toreador Ln, Oakton, VA 22124 Ludwig Benner is currently engaged in safety risk analyses and investigations, with experience in managing advanced system safety analyses and accident investigation projects, developing and instructing risk assessment and accident investigation courses for the Army, GSA, U.S.DoT, #000080 and private sector, and investigation and decision research. He is the co-author of INVESTIGATING ACCIDENTS WITH STEP, a system safety-based investigation system, and has over 60 publications in the safety field. He career includes positions with the University of Southern California, the NTSB, and Air Products and Chemicals. End Notes [1] For the purposes of this paper, a safety recommendation is viewed as a proposed action to address a safety problem discovered by the investigation of accidents., either after or before they happen. [2] For a fuller discussion of this difference, see Hendrick, K.M. and Benner, L. INVESTIGATING ACCIDENTS WITH STEP, 1987, Marcel Dekker, NY p 197. [3] Wood, , R.H., HOW DOES THE INVESTIGATOR DEVELOP RECOMMENDATIONS?, Proceedings of the 10th International ISASI Seminar, September 24-27, Quebec, Canada. [4] Bruggink, G.M. and Fritsch, O., THE SAFETY RECOMMENDATION PROCESS, ISASI forum, November 1989, V22:3, p 1. [5] Hammer, W. HANDBOOK OF SYSTEM AND PRODUCT SAFETY, 1972 Prentice Hall, Englewood Cliffs, NJ, p 251-296. [6] This does not suggest that system safety principles can not be applied to the process, but only that the process has not been documented in that standard. [7] From POLICY section of NTSB Order 82, 6/11/87, p 1. [8] See NTSB News Digest, Volume 9, No. 10, October 10, 1990. [9] The law allows a 90 day response period from the U. S. Department of Transportation. [10] See Benner, L., RATING ACCIDENT MODELS AND INVESTIGATION METHODOLOGIES, JSR 16:, pp 105-126, 1985. During that research, terms such as "clear" or "concise" were observed frequently. [11] See for example the prediction of 95% effectiveness in the reduction of accidents in Department of Transportation Docket HM 44; subsequent accident observations and statistical results indicated that the predicted safety effectiveness of the recommendations reflected in this rulemaking were achieved. [12] The NTSB 1988 Annual Report to Congress is cast more in terms of function than safety achievement, and does not contain information to permit an assessment of safety achievement of its recommendations, either individually or in the aggregate. In this respect, the NTSB program is typical of most safety programs. Few recommendations claim safety results - except in the most general way. Aggregate accident counts or rates can rarely be related to specific actions. For example, the NTSB publishes annual statistics purporting to describe transportation safety performance, but does not claim that any of its specific actions produced that record. [13] The same thought is also expressed as "accident prevention" in other investigation programs. [14] Risk is used in the sense of a predicted chance of loss associated with the predicted chance of the gain from a proposed action. In the context of a safety recommendation, a decision to act on a safety improvement recommendation must consider the safety and related gains likely to follow the decision, and balance that gain against the loss or costs likely to follow the decision to act or not to act. This definition should not be interpreted to infer a "cost-benefit" analysis in economic terms, since other social values tradeoffs are usually involved. [15] For a discussion of this phenomenon, see Hendrick, K.M. and Benner, L. INVESTIGATING ACCIDENTS WITH STEP, 1987, Marcel Dekker, NY p 364-365. [16] See, for example, MILSTD 882B, Hammer's book and others available through the System Safety Society for a description of system safety management and engineering concepts, principles and procedures. [17] At the rate of 2700 investigations per year, one recommendation per accident would have resulted in 40,500 recommendations, rather than 8000. [18] For example, in one report, the probable cause was the failure of a train crew to properly secure their train by placing the train brakes in emergency, and applying hand brakes when it was left standing on a mountain grade. Contributing factors included decisions by a Helper locomotive engineer, effects of cold on air brake system and crew and crew members, management assessment of employee qualifications and employee training and hazardous materials. Ten months after the accident, well into the next winter season, the NTSB adopted 25 new recommendations and reiterated 3 previous recommendations. All 28 recommendations carried the NTSB's Class II Priority Action designation. (NTSB /RAR-89/05) [19] The NTSB provides for acceptance of alternate action, but unless relative safety effectiveness is predicted and disclosed, it can not be monitored and managed properly. [20] The FAA, for example, by law has 90 days to respond to an NTSB recommendation, regardless of the urgency of the problem. If the problem, rather than the recommendation were given priority ratings, the response time could be tailored to the problem by whoever is in a position to correct it. [21] The NTSB has done this in a few instances in the past, with significant effect. For example, the NTSB's recommendations and activities which led to DOT docket HM-44 requiring head shelf couplers and head shields predicted the degree of improvement over time. [22] The NTSB has a precedent for this approach when it closes a recommendation file with its "Closed,-Acceptable Alternative Action" designation. [23] For example, the US Department of Energy specifies that a section on "Judgment of needs" be included in its accident reports. This kind of a section could be adapted to NTSB reports. See p 129, U. S. Department of Energy Accident Investigation Manual, DOE/SSDC 27, 2nd Edition. [24] An operational safety performance monitoring plan for predicted hazards is an integral part of system safety practices, but not often addressed in NTSB recommendations.