|
CONTENTS
ACCIDENT INVESTIGATION PROBLEMS NATURE OF THE RESEARCH SUMMARY OF FINDINGS APPLICATION OF FINDINGS CONCLUSIONS REFERENCES
ABSTRACTAt least 5 different perceptions of the accident phenomenon, 44 different reasons for investigating accidents, 7 investigative processes, 6 general methodologies, and 3 differing types of output requirements affect accident investigations. These differences reflect the lack of a unifying conceptual framework for accident investigation and safety; corrupt data search, selection, organization and reporting decisions during investigations; and result in spurious accident data. A tentative unifying perception of accident phenomena, with supporting assumptions, principles and rules of procedure for their investigation was synthesized from the findings. Four games simulating the resultant accident investigation process have been developed. The process has been applied to improve hazardous materials emergency response decision making, and evaluation of safety countermeasures. N.B. 2007. Though it may not be clearly stated, this report is based direct observations of invesitgation processes, interviews with investigators and others, during investigations, and actual participation in investigations. The significance of this use of primary data was not recognized until the results of alternative research tools becaume available subsequent to this study. Awareness of the issues reported here began to permeate the investigation community, attactcing researchers from other disciplines to undertake studies using the tools of their disciplines, usually relying on secondary data . The differences in the results led to my awareness of the critical significance of primary data for studying investigation processes. LB ACCIDENT
INVESTIGATORS HAVE PROBLEMS that affect a lot of people beside the
investigators. I am an investigator. Let me tell you about those problems, and
how they might be overcome.
ACCIDENT INVESTIGATION PROBLEMSAccident investigators investigate accidents. What is this thing called “accident”that investigators investigate? When they begin an investigation what should be the scope of their investigative efforts? What data should they seek during an investigation? How will the “right” data be recognized? When will the investigator have enough data? How should this be determined? How should the data be organized, summarized or reported? How does the investigator determine if the outputs or work products from the investigation are satisfactory, and whom the data and outputs must satisfy? At what point does the investigative task end and the analytical task or use of the outputs begin? What conclusions can be drawn from the outputs, and how should their validity be tested? Equally importantly, where should one turn for the answers to these questions, and how does one assess the quality of the answers? And finally, how do accident investigations contribute to safety in the most effective way? These are not rhetorical questions. Every investigator is faced with these questions each time an accident investigation is begun, because every accident is different from previous accidents in one or more ways. Presently each investigator answers such questions with “common sense” and with “good judgment” seemingly predicated on experience and academic background, rather than on generally accepted decision rules that led to reproducible outputs. In these circumstances, were the investigators’ tasks of such a nature that final work products were theoretically consistent and replicable anyway? Observed differences in the purposes of investigations, in the scope of matters investigated and in the nature of data reported indicated a negative answer. Observations of uses of the work products suggested that it is these differences that create problems for users of investigative work products, and persons involved in the accidents. For the users, unreproducible, inconsistent and incomplete work products can impede discovery of safety problems, misdirect corrective efforts, generate controversy, undermine credibility, and confuse the users’ audience. For the persons involved in the accidents, work products can overly simplify complex relationships and result in unjust blame for the accident, or even worse effects. For all concerned, investigators’ problems get magnified when the outputs are used. For these reasons, investigators’ problems need to be addressed and resolved. This paper is addressed to that need. As the questions were pursued with other investigators, early ill-defined uncertainties accumulated rapidly and in growing numbers. Upon analyses, they have been categorized in terms of accident investigation objectives, scope, methods, outputs and uses for study purposes.
Why bother to investigate a specific accident? Investigators may be told to do an investigation by their employers, or they may elect to conduct an investigation on their own initiative. Of over 200 investigators queries informally, almost all had difficulty articulating the objectives for the investigation they were conducting. Most replied in terms of reasons for investigative programs, rather than objectives for a specific case. General answers most frequently included “prevent accidents,” ”complete the forms,” “save lives,” and “find the cause.” Reasons reported since 1974 have been summarized in Appendix A. 44 reasons have been reported. Note the substantive differences in reasons. Be aware that only once was “understanding the accident phenomenon” stated as a reason. Because of the nature of the reasons expressed, attempts to transform these reasons into objectives that could be used to measure the quality or success of the investigation were fruitless. SCOPE During an accident investigation, every investigator makes an implicit determination of the beginning and end of the accident by deciding what data will be reported. Attempts to find out how this decision was reached by practicing investigators remain unsuccessful. Very few investigators are even willing to discuss the question. Most evade the question or begin to talk about examples which defied generalization. Not one of the investigators queries articulated a rule of procedure for identifying the beginning or end of an accident. In practice, this difficulty resulted in frequent disagreement among investigative team members about matters that should be reported, and how far backward in time to track “the accident.” METHODS Informal inquiries among investigators showed essentially no agreement about the “best” accident investigation methods to be used. The choice of methods was usually intuitive, and was not addressed explicitly by any investigators. Additionally, no pressure from users of the work products was exerted on investigators to address this issue. Investigators became very uncomfortable when the issue was raised, and were unable to discuss the question of criteria for selecting methodologies for their investigations, probably because they had not previously attempted to articulate them. OUTPUTS.
Specifications for work products from accident investigations were, for
practical purposes, non-existent. The closest approaches to specifications were
scattered instructions for the use of forms for reporting accidents. However,
even
the simplest forms required interpretations by investigators, as will be shown
later, and none of the instructions provided a basis for determining the
quality of the data reported with accident-based criteria. The numerous outputs
often generated from a single accident further demonstrated the problem
confronting an investigator seeking criteria by which to judge the
acceptability of the work products--both with respect to one’s purpose
and the realities of the accident. The same dilemma confronts users and
evaluators of the investigators’ outputs.
USES. When one attempts to use investigative outputs for evaluation purposes, or to
reach a better understanding of accidents, problems with the scope, methods and
outputs are accentuated. The almost universal lament about the “accident
data” available for subsequent uses was experienced personally by the
author in the evaluation of safety program efforts in his field. It is
discussed in a report of the National Transportation Safety Board released in
1971, in detail. (1) The problem cited remains unresolved.
To
sum up the problems, accident investigators have been resolving their
investigative dilemmas in every accident--each in his or her own way, using
personally developed decision criteria. The results are not hard to anticipate.
Replicability, verification, prediction, communication, utility, consensus and
credibility suffer. Users never have the right, good or reliable (4) or enough
data. The more one delves into these problems, the greater the enormity of
their consequences seems. Ponder the effects on safety policy, public opinion,
safety programs and objectives, research outputs or on the assignment of
personal fault or culpability, and the need for their resolution can be seen.
NATURE OF THE RESEARCHA word about the research is in order, because it does not meet traditional research design standards. It has, however, been tested. It grew out of personal needs that were informally addressed, into a determined pursuit of rational understanding that would form the basis of investigative decision and evaluation criteria. The research was driven by personal concern, practical needs, process realities, and predictive validation necessities. The personal concerns initially related to friends in the fire service whose lives might depend on the quality of the lessons learned and reported during the author’s accident investigations. Firefighters and other emergency response personnel bore the highest risks from accidents involving hazardous materials. Until the reasons for these high risks could be understood, they could not be reliably controlled. Accidents did not occur frequently enough to develop a traditional statistical base for analysis of the reasons, so an alternative methodology had to be found. The practical needs reflected job performance requirements for the author and fellow investigators of the phenomenon called “accidents.” Principles and rules of procedure for the investigative process that would produce consistent and defensible results would provide great comfort to any investigator who was involved in some of the controversies which arise after investigations are concluded. Process realities involved both investigative and accident processes. The investigative process realities included the diverse interests of parties in large investigations and the diversity of views, skills and ideas brought into an investigation by individual specialists--all of which had to be reconciled in a convincing way by the lead investigators. The accident process realities included the continuing need to reconcile existing concepts and views about the phenomenon with the realities of the accident observed during the investigations. Prediction validations demanded the research. Evaluations require norms. Ways to identify these norms had to be related to ways accident data could be reported, so the evaluator could “close the loop” between the predictions and the accident experience. Traditional concepts and methods demonstrated little promise in actual investigations. Did this mean that the task was impossible? If so, why bother with accident investigations? Until this question could be resolved, the author had to continue to guess at the data which would serve this need. As the need for investigative criteria became apparent through the informal inquiries, study of the accident processes, the frameworks within which they functioned, the process outputs, process participants, interactions within the processes, and process results was initiated. That study, in turn, required methodological choices by the author. The traditional comparative approaches failed to provide working insights, so an attempt was made to formalize the investigative method itself to support the study effort. The success of the method developed led to an attempt to develop a coherent unifying framework and set of assumptions, principles, rules of procedure and predictive methodology. Since the research was primarily part-time outside of working hours, record-keeping was kept to a bare minimum, and no nice neat records of early observations are available. While not formally tested in classical ways, the research findings have been tested during accident investigations informally. As new insights were suspected, they were tested against customary procedures during investigations, and also with simulations in the classroom. These tests clarified investigative task criteria, facilitated discovery of safety problems, improved the efficiency of the investigations, and enhanced credibility of the investigation outputs . Experimental applications of the findings during classroom simulations triggered deeper insights into both the problems and ways to resolve them, including the “mental movies” and “advancing time” concepts discussed below. These interactions with experienced, practicing investigators and students are acknowledged with gratitude. SUMMARY OF FINDINGSBecause of the scope of this research, and its initial informality, •a full report of all the findings is too lengthy for a paper. Further detail is contained in the works referenced. Of special significance are the findings about accident perceptions, investigation processes, methodologies, outputs and relationships among these findings. ACCIDENT PERCEPTIONS. Accident investigators investigate accidents. What is this thing called “accident?” The question is not new. The author’s initial approach was to examine models of an accident developed by previous researchers, and test them in practice. A list of the models tested and the deficiencies found in them is shown in Appendix B. Each model was tested for its potential value in providing investigative criteria during investigations. No single model fulfilled the need. The next step was to try to answer the questions using “definitions” of accidents found in the regulations, literature, safety publications, investigation manuals, and periodicals. Review of over 200 different definitions disclosed that the diversity of opinions about the nature of the accident phenomenon was even greater than suspected. The findings from the accident definitions are tabulated in Appendix C. The third step was to observe the methods used by individual investigators during investigations, and to try to discern from their actions, comments and decisions what their perceptions of the accident phenomenon were. These observations and the work described above led to the identification of five general and differing perceptions of the accident phenomenon. (5) Each perception was found to be accompanied by a set of ‘implicit assumptions, rules for investigative procedures, and “principles.” The five perceptions, and their implicit assumptions, principles and rules of procedure (theories?) include 1) the single event perception and related “cause” theory; 2) the chain-of-events perception and “domino” theory; 3) the determinant variable(s) perception and “factorial” theory; 4) the branched events chain perception and “logic tree” theory; and 5) the multilinear events sequences perception and “process” or “p-theory.” A brief description of each is found in Appendix D. After these perceptions were identified, an attempt was made to link them to commonly held safety concepts and philosophies, in order to attack the “close the loop” problem. Relationships between these perceptions and numerous safety concepts became visible. For example, the single event view is clearly linked to the concept of “cause” of accidents, and the body of investigative procedures related to that determination. The idea of unsafe . acts and unsafe conditions appears to be linked inextricably to the domino view, as is the idea of “causes” and safety actions to “break the chain of events.” The branched chain and multilinear events sequences perceptions, dealing with interactions and probabilities of events sets, compel a risk perception of safety, and the view of a non-zero probability of an accident with any activity. The determinant variable perception relates clearly to the “causal factor” view of safety problems, and data sampling for safety analysis. Each view has the effect of influencing a different philosophy of safety, and the safety programs which ensue. These perceptions also help to interpret the meaning of safety concepts in concrete terms, as will be shown in the next section. ACCIDENT INVESTIGATION PROCESSES . The early dialogues quickly revealed that there is no single accident investigation process. As individual investigators were observed, differences in the processes used by the investigators were observed. The processes ranged from the simple “investigation” of a few minutes duration by an employee’s supervisor in a “minor” accident to a multi-million dollar effort for a major accident. Among the differences observed were the sizes of technical staffs, and their technical capabilities; authorities; funds available; methods employed; work products; and the effectiveness of the investigative efforts. One way to categorize the processes identified is in terms of their staffing. Each category of investigative processes is briefly described in Appendix E. The categories are the 1) one-person process; 2) intra-organizational ad hoc team process; 3) intra-organizational standing team process; 4) multi-organizational ad hoc team process; 5) multi-organizational standing team process; 6) judicial investigation process; and 7) legislative investigation process. When examined for their underlying perceptions, purposes, objectives, scope, methods, outputs and applications, process differences and commonalties were observed. The processes incorporated at least one of the five perceptions cited above, and sometimes more than one. Their purposes ranged from getting forms filled out to restoration of national confidence in a large system accident. Methods ranged from intuitive methods in one-person cases to the use of sophisticated analyses and calculations in others. One person methods included the use of multiple choice codes for completing forms, while other processes attempted to identify sequences of events, determine “cause” and report all the facts and circumstances of the accident. One—person, judicial and legislative processes relied most heavily on witnesses’ statements. The standing team processes relied on a more balanced split between data from people and things. Some scenario modeling was used by standing teams or in teams with trained investigators. Uses varied widely, forming no readily discernable patterns within a process. Common
to almost all process observations were attempts by investigators to determine
“cause(s)” and usually also a “sequence of events.” As
a rough outline of a course of events evolved, most investigators tended to
desegregate events for a more detailed understanding of what happened.
With
in
team processes, common time and spatial events testing procedures were noted.
Investigators
felt impelled to “get all the facts” about an accident so the
“facts” could be analyzed and reported. The distinction between
fact gathering and analysis of the facts was sharply drawn almost universally,
and is encouraged by the format of most narrative reports. This may be a
carryover from legal concepts related to the weighing of evidence under
adversary proceedings. Criteria for identifying “facts” during an
investigation, however, were either absent or were stated in equivocal terms;
the determination that data constituted a “fact” was a judgment
call which sometimes generated heated controversy. The “facts”
sought seemed to be most heavily influenced by the investigators’
personal background and experience, and by the assumed hypotheses formed early
in the investigation. (6) Interestingly, organizations using accident reporting
forms customarily provided at least one additional level of persons to verify
or evaluate the completed forms from the investigator before processing them
for accident “analysis.”
The
observed uses of the data reported on forms suggested that a distinction should
be made between “primary” and “secondary” accident
investigations. The primary investigation incorporated direct observations of
debris or physical evidence and witness’ information collected by the
investigators. The secondary investigations were conducted solely with data
reported by others after primary investigation efforts, to draw conclusions
about the accidents. For example, field investigators usually completed the
report forms used by the secondary investigator .
ACCIDENT
INVESTIGATION METHODOLOGIES.
At least six different basic methodological approaches were observed in
accident investigations. The approaches included events reconstruction,
statistical, modeling, simulation and “hunt and peck.”
Events
reconstruction methods used “events” deduced from physical evidence
remaining after the accident, witness interviews and speculations by the
investigator to “reconstruct” the sequence of events viewed as the
accident. The methodology drew heavily on physical science disciplines and
examination techniques. The degree to which events were decomposed or broken
down into sub-events was left to the investigators’ judgments.
Frequently, one or several events were selected as the “cause” or
“causes” or “probable cause” or “proximate
cause” of the accident. Events were undefined, and therefore the nature
of the events described varied widely. Reported events sequences often stopped
with a crash or collision. Reports often included recommendations designed to
“break the chain” of events .
Events
reconstruction methods were augmented with logic trees, which provided for
treatment of concurrent as well as sequential events. Logic trees culminated in
a single undesired event, selected by the investigator. However, these logic
trees did not show actual time relationships among interacting events connected
with “and” gates.
Statistical
methodologies were used for secondary investigations, usually involving several
accidents. They included manipulation of data using statistical rules of
procedure to identify determinant variables in an accident occurrence. Both
static and dynamic descriptors were used indiscriminately, and the accident
scope issue was not addressed. A fatal defect with these methods from an
investigator's perspective was the failure to treat time relationships among
events, although sequences were often addressed. Observation of conditions
being represented as events was common. A frequent complaint about the outputs
was the inability to identify corrective actions after “problems”
were identified. (7) For example, the conclusion that drinking is involved in
half the highway accidents does not provide a basis for corrective actions.
Adversary
methods were used frequently in the judicial and legislative investigative
processes. The premise that opposing interests would bring out the truth about
the accident forms the basis for the formation of investigative teams with
representatives of several parties involved in the accident. Information was
often gathered in an adversary setting, particularly during witnesses’
“interviews.” The “facts” gathered were tested
informally by discussing their apparent logic and consistency against the
summary hypotheses that incorporated these “facts.” The method was
disciplined by rules of evidence and other procedures drawn from the judicial
processes, but the outputs were usually incomplete, and not subjected to any
rigorous investigative disciplining criteria such as “beginning”
and “end” tests. Issues reported were largely judgment calls by
investigators, and no methodology for the calls could be detected.
Modeling Flow
charting was another kind of modeling observed. Events flow charts describing
the accident mechanism or scenario in its sequential order were developed and
recorded on paper in chart form. The charts, in some cases, implied or
presented the timing and duration of events. (40) Logic trees were a form of
this kind of modeling during the investigations, and sometimes were included in
the outputs. However, charts rarely constituted the
sole outputs.
Scale
modeling was another technique used in accident investigations. Use of scale
models with witnesses during interviews helps them explain their observations
to investigators. The attitude of aircraft, their flight paths and other data
were sometimes conveyed to investigators with such models . .
Two
other observations about models were noteworthy. First, events .f low charts or
models did not have to deal with variables in an accident investigation,
because the events had .a certainty of one, i.e., the accident happened.
Secondly, inadequately disciplined models were often observed. Curious mixtures
of actions and conditions within supposed events sequence models were not
uncommon. Another defect in most models was the violation of the
“advancing time” discipline. Time was represented by the order in
which events were presented and linked with arrows. However, arrows often
pointed in opposite directions, creating the suggestion that time flowed
forward and backward. In accidents, time was never observed to have flowed
backward; participants in accidents never got to pass through a given time
segment a second time! Undisciplined use of arrows without regard to the
implications about time flows, and the mixing of static and dynamic descriptors
in models does not represent the accident phenomena investigated by this author.
Simulations
constituted another methodology. Crash simulations intended to reproduce
automotive accidents were widely used to form the basis for safety actions.
Flight simulations, explosives tests and similar reenactments of hypothesized
accident scenarios were observed too. Simulations were usually thought of as
reenactments by the investigators. The reenactments were used both to develop
data about gaps in their understanding of what happened, and to formulate
hypotheses by seeing if the simulations resulted in the same outcomes,
Simulations also permitted investigators to vary the assumed events
hypothesized during investigations, to assess the effects of the changes in the
accident scenario. Simulations constitute one way of demonstrating that the
investigator fully understands an accident mechanism.
“Hunt
and peck” is the catch-all term used to describe unstructured
investigative approaches observed. Some investigators--especially inexperienced
ones--have no methodology in mind when they begin an investigation. This
methodology is indicated when an investigator must visit the scene of the
accident before anything else can be done, in the hope that some insights will
come from just looking at the accident. No useful results have been observed
with this methodology.
While
not distinguished separately from the above, comparative analytical methods
were employed in each of the investigative methodologies described above. In
investigations, comparisons of the expected versus the actual actions, events
or conditions were frequently addressed, and the determination of the
expectations was an important task for many investigators. Comparisons also
form the basis for the use of checklists and accident report forms; the
comparison in the one case is between
the
expected and actual investigative tasks, and in the second case is between the
form designers’ view of the accident, the investigator’s view and
the realities of the accident.
Supporting
these general methodologies are at least 22 investigative examination
techniques that the author observed. These techniques are listed in Appendix F.
The quality criteria for these examination techniques usually reside within the
disciplines represented, and are not published in useful forms for accident
investigators’ use. Examinations were usually performed without
substantive test plans. They did not address the entire accident phenomenon
when they were used, and they usually focused on a narrow set of events. On a
few occasions, these examinations constituted the principal investigative effort.
ACCIDENT
INVESTIGATION OUTPUTS
.
Three types of investigative work products were observed. They were, in the
estimated order of frequency, 1) completed accident report forms; 2) narrative
accident reports; and 3)models. Sometimes, more than one kind of work product
was observed from a single investigation. Supporting materials for these work
products included documentation such as photos, test reports, maps, sketches,
diagrams, etc.
Comp1eted
forms usually were designed by someone other than the investigator; often they
were designed by insurers, researchers, or regulators, seeking specific
information to serve their needs. Entries were specified in what appeared at
first to be concrete terms, but which upon analysis were found to require
extensive interpretation by the investigators. Typically, forms were not fully
completed by investigators in all detail, or else non—specific entries
were used (“other”) when all the blanks had to have some kind of
entry. Every form required investigators to make judgment calls about the
entries, for reasons that included the inapplicability of the form to their
specific accident, lack of investigative data, misperceptions about the
accident phenomenon that found their way into the forms, and conflicts with
entry specifications. Using the five accident perceptions described above, the
form content and entry specifications were found to be internally consistent
with one of the perceptions, or in some cases two of the perceptions, when they
were analyzed.
Observed
difficulties with the use of forms in accident investigations included
limitations on the opportunities to discover new insights into the accident
phenomenon being investigated; distortion of data about an accident because of
the need to fit actual observations into predetermined specifications that
didn’t fit the actual observations; and the tremendous variations in the
scope of the data about an accident that were reported.
Narrative
reports were observed in two forms: written
and verbal. Report length (reflecting the scope of the data about an accident)
varied from a few minutes duration to over 300 pages plus appendices. No
universal pattern was discerned for narrative reports. However, a general
format used in National Transportation Safety Board reports was observed
frequently in other reports. That format provides for presentation of
facts/analysis! conclusions! recommendations sections in that order in written reports about accidents. (9) Non-governmental reports had no
discernible pattern beyond a narrative description of what happened, followed
by a discussion of investigative, cause or related information. In narrative
reports of accident investigations, the accident was described as an events
sequence description, without exceptions. Analysis sections had no common
pattern.
Most
organizations prescribing narrative reports provided some form of investigative
manual. These supporting manuals contained varying levels of detail about
investigative purposes, procedures, examination methods reporting formats,
and--on occasion safety philosophy. Manuals often prescribed matter to be
reported, but did not provide criteria for determining whether the quality of
the matter covered was adequate. The International Civil Aviation Organization
and the Energy Research And Development Administration investigation manuals
are typical of the better manuals available, but neither treats the quality
criteria question explicitly. (10A2)
Models Recommendations
are often considered an output from an accident investigation. However, the
preparation of recommendations from information developed during an accident
investigation seems more properly to be an end use of the investigative output,
rather than an integral function of the investigation. The view that an
investigation should produce recommendations is apparently tied to the accident
prevention purposes of accident investigations, rather than the risk-based
perception of safety reflected in the events sequences perceptions. Of all
functions related to accident investigations, the development of
recommendations was the least structured, and the most dependent on the
investigators’ common sense and good judgment. Research into
countermeasure development and evaluation has disclosed only one major
published effort to present countermeasure theory and principles for the
investigator. (13)
Supporting
documentation should not be confused with total outputs, but their contribution
to understanding accidents can be significant. An aerial photo of an accident
site, for example, contains data for both the investigators and the users of
outputs. However, specifications for supporting documents and for their role in
the outputs were skimpy. Only the judicial processes provided criteria for
evaluating data, in that rules of evidence were used. (14)
RELATIONSHIPS
AMONG FINDINGS
.
These findings were interrelated. An individual’s perceptions of the
nature of the accident phenomenon influenced the individual’s reasons for
investigating an accident. These reasons determined the process selected, the
scope of the investigation, and the methods used. These in turn influenced the
work products. Both the perceptions of an accident and
the
deliverables from the investigation influenced the eventual applications of the
work products. These relationships are shown in the following model.
|