Go to Guide: 0 1 2 3 4 5 6 7 8 9 10 11

What is investigation "quality assurance?" It is steps take to ensure that the criteria for a satisfactory investigation have been met. What are those criteria? Today, they vary with the type of investigation and investigation method chosen. If we can focus on development of descriptions of what happened during episodes being investigated, some criteria common to all investigations can be identified.

The episode descriptions should be logically valid, tested and timely representations of everything that had to happen to produce the episode outcome. They should be developed from the residual data about the episode remaining in people and objects after the episode, transformed into valid building blocks used to create the description. They should produce data that will meet user's needs for inputs to develop their response actions to the episode. They should be produced efficiently, avoid subjective, comparative and extraneous content, be readily communicable to users, and to the extent possible, visualizable. The content must consistent in form and grammar to facilitate congruent aggregation of episode descriptions.

OVERVIEW OF INVESTIGATION QUALITY ASSURANCE

Traditional investigations lack objective quality assurance procedures for either the investigation process or the investiation work products. Quality of work products is controlled primarily by one of several variations of the peer review procedures, and validity is decided by power of persuasion in what is essentially an adversarial process. A second general approach is the "fly-fix-fly" approach, where conclusions are tested by repeating the experiment or episode or simulating it, and determining from the outcome if the investigation findings "reproduced" the episode. Another approach is to trust highly qualified investigators to do quality work.

Quality assurance of the investigation process is similarly determined. In the absence of objective quality assurance criteria, and procedures to apply those criteria, consistently high quality investigation performance and outputs should not be expected.

The STEP-MES investigation process provides new opportunities for assuring the quality of the investigation and its work products, including the explanatory description of what happened, recommendations flowing from the investigation, investigation reports, and effectiveness of the assurance actions implemented. The primary quality assurance vehicles are the STEP-MES-based Matrix and work products developed from that Matrix. These Investigation Quality Control (IQA) procedures constitute an essential element of the STEP-MES-based investigation process, and provide a way to check reports resulting from other methods.

OBJECTIVE

The objective of this Guide is to present a quality assurance process that can be used to ensure valid, effective STEP-MES investigations and episode descriptions.

APPLICABILITY

This investigation quality assurance procedure is applicable to all kinds of investigations and investigation work products purporting to describe and explain an episode, and to the analyses of users' response needs and actions flowing from that episode description. It can be applied to any investigation and work product to ensure confidence in their outputs.

Note: The procedure is also applicable to hazard and risk investigations where the system operation is described with STEP-MES Matrixes. If not so described, the analyses can be recast into that format, and the reformatted process flow and analysis conducted as described.

DATA AND DATA SOURCES REQUIRED

The investigation quality assurance process utilizes the data from the investigation and resultant work products. For STEP-MES-based investigations, data on Matrixes, interim work products and source references are required. For non-MES-based investigations, only the investigation work products and, if available, source references are used. Data in the check lists presented in this Guide may also be used.

IQA PROCEDURES

Procedures for STEP-MES-based and non-MES-based investigations are similar in that BBs and Matrixes form the basis for both. They differ in that Matrixes must be created for non-MES-based in investigations, whereas Matrixes are developed as part of the STEP-MES-based investigation process. The effect is that the IQA person will perform BB and Matrix development tasks (Guides 1 and 2) for non-MES-based investigations, whereas investigators perform these tasks during STEP-MES-based investigations. This is an important difference, because it means the IQA person must be capable of performing the BB and Matrix tasks as well as the IQA tasks. For this Guide, the BB and Matrix development tasks are not repeated in their entirety, but only where they differ due to the IQA requirements.

IQA procedures are implemented during the investigation process. They begin with the creation of the first BB, and do not end until the final investigation based work product is completed. The IQA tasks include the following, if the guidance in Guides 1-9 has been implemented and a Matrix is available for IQA review. If the Matrix is not available, treat the IQA task the same as a non-MES-based investigation work product, and go to Part B.

MATRIX QUALITY CHECK.

Examine each BB an the Matrix to verify that the format and content are consistent with Guide 1 specifications. If they are not, note the discrepancies on the Matrix.

Examine every event pair on the Matrix for the sequential logic of the BB placement. If BBs are not properly ordered, note the discrepancies on the Matrix.

Examine each event set on the Matrix for the necessary and sufficient logic of the BB links displayed, and try to visualize what is being described from the data presented. Try not to let prior knowledge of the system get into the way. Consult with someone having systems operations knowledge if you can't visualize the process being described. If logic fallacies or gaps are found, note the discrepancies on the Matrix.

Examine each countermeasure tab, needs statement and recommended action to ensure they are tied to the correct BBs describing the episode, and do not conflict with each other or introduce new, unknown relationships.

NEEDS DEFINITION QUALITY CHECK.

Review the Matrix for countermeasure tabs to locate the BBs or relationships that were concluded to be "needs" and determine that this decision was made by the proper person or persons. This should not become an exercise in second guessing, but rather review to ascertain that the needs identified are indeed validly defined and reported.

Review each countermeasure diamond on the Matrix to verify that the need identified MEETS THE SPECIFICATIONS described in Guide 8, and that the rationale is described in terms of the BBs involved, rather than some abstractions or judgment call based on past experience, past episodes or other unsupported rationale. Note any problems with the rationale from those specifications on the Matrix.

1. Review the needs/problem statement to ensure that the EXPECTED PERFORMANCE and the source of those expectations is identified in terms of specific actions by specific actors! Again note any deficiencies in the identification of the expectations on the Matrix. Challenge any ..ly words or allegations of error or failure if they do not include the preexisting expected performance sources in a concrete way.

2. Review the process by which the author determined that the need/problem merits attention, which is implied by stating it, to ensure that the rationale is presented, and is based on an agreed measure of significance or some agreed-upon criteria, as described in Guide 8.

3. Then make sure the selection decision(s) for the needs that will be addressed by recommended action was made by the proper "authority" in the organization.

4. If the IQA check does not support its selection as a need or problem to address, note the rationale on the Matrix, relating it to the diamond number.

ACTION OPTION CHECK.

1. Review the needs selected for action, and the candidate action options suggested. Ideally, the options should be related to some principle to provide the supporting rationale for its candidacy. That principle should be readily apparent or stated.

2. For each candidate action, review the tradeoffs identified in favor of its selection, and opposed to its selection. Tradeoffs should address the predicted relative effectiveness, resources required, technical feasibility, organizational impact or other criteria of significance to the decision maker. At a minimum, the predicted effects of the action on similar future episodes should be described. Review the weights assigned to each trade-off to assess their validity. Since this is a judgment call, variances in weights are to be expected. Negotiations to reach a consensus on the trade-offs and weights is a useful exercise during the IQA process, because it helps develop weighting criteria over the long term. Notations on the trade-off Matrix are recommended.

3. Review the predicted effects to ascertain that they provide a way to track the predicted results. Remember, if the predicted results are not defined so this can be done, the proposed action is suspect, and should be challenged. Again, make notes of the findings on the Matrix.

4. Review the recommendation selection process to verify that the options recommended are the higher rated options among the candidates suggested, using the criteria described in Guide 8.

5. Finally, review the proposed actions against the IQA check list (Figure 10-1)as a final test of the recommendations.

Over time, the IQA procedures conclude with a quality check of the final documented work products resulting from the recommended action(s) implemented. Depending on the scope of the investigation program, the last document may be a report on the success of the action implemented.

ƒ

IQA PROCESS OVERVIEW.

The IQA procedures begin by recasting information from the work products into the STEP-MES-based input data Building Block (BB) format. Then an STEP-MES-based Matrix array is created with those BBs. Then the IQA process proceeds as described above. Before attempting to perform an IQA on a set of documents, the reviewer should become skilled in the procedures for creating and testing Matrixes in Guides 1 and 2.

GENERAL IQA FRAMEWORK.

Kipling's faithful servants: WHO-WHAT-WHEN-WHERE-WHY form the general framework for IQA procedures.

Who did what

As a first step, a quality investigative output will give every person or object one name and use only that name throughout the rest of the episode description or discussion.

Secondly, the output will present the episode data and episode description in the "active voice" ( ball struck floor) where the name of the actor who or which does something comes before any words describing what the actor did (not floor was struck by ball.) An abbreviated way to state this is to ensure that data are transformed into an Actor + Action (ball + struck) format. This is the basic "BB" used to describe what happened.

When

The description will describe when each BB occurred, relative to some other BB or reference point. Review each BB to verify that it is in the proper temporal sequence based on the logic of the BB flow.

Where

In addition to confirming that the BBs are in their proper order, the BBs must be arranged in their spatial sequence to make sense. Don't add actions that require someone to fall up two long flights of stairs, for example.

Why

This is probably the most subtle and the most abused quality assurance criterion. For one BB to affect another BB, an input/output relationship - or linkage - between the BBs must exist. This input/output linkage must be established for all BBs in the description, because the input/output links define why the episode continued to its conclusion - the outcome -the whole after it started. If there is no input/output linkage, the BB may be nice to know, but it is not needed to understand and explain the episode! Input/output links connote an if/then relationship: if A occurs then B follows. If B does not always follow A, then either A is not the input of B, or additional inputs are required for B to happen.. Need to spend more time

In an episode description this kind of critical logic testing can be used to select BBs that must be reported to describe what happened and why it happened. Tests for input/output linkages are well know to critical thinkers. The tests require application of the necessary and sufficient reasoning from logic. They provide a basis for establishing investigation quality assurance to determine and ensure that precede/ follow logic governs the episode description.

These ideas provide the essential criteria to permit a check of the quality of episode descriptions and resultant work products.

IQA TECHNICAL PROCEDURE

The following technical procedure provides a way to do a quality assurance check of an episode description, using STEP-MES-based criteria to ensure that the description is valid and complete, explain why the episode occurred, eliminate unnecessary data from the description, and identify uncertainties. Although relevant, this quality assurance procedure does not address subsequent uses of the episode description, judgments or opinions about the episode, recommendations, or other aspects of the investigation quality assurance issue. It is based on the premise that the quality of other aspects of the investigation depend on the quality of the episode description, because if the quality of the description is unsatisfactory, any subsequent uses of the episode description are tainted. Quality criteria and procedures for those aspects of an investigation require separate consideration[1]

a. The IQA procedure

This procedure is designed to ensure that the who, what, when, where and why are properly reported, using the concepts just described. To apply the procedure, start with any description of an episode and take the following steps.

1. Find the reported BBs.

Highlight or underline each actor and concrete action set reported in documents. Each actor/action set is an BB. These BBs become the basis for assessment of the data quality. If the data are not properly transformed into the actor/action format, you will experience difficulties when you go to use it. You may have to dig hard to find all these actor + action (BB) sets, because the actor and the action may be separated by many words, - or in bad cases, sentences - or be described ambiguously or in the passive voice.[2] For whatever the investigator's reasons, sometimes the name of the actor is not stated when an action is reported. "Passive voice" sentences ("was" or "were") or pronouns or other ambiguous actor names, or abstract action words like failed, erred, etc., obstruct formulation of your mental movie of what happened. DO NOT assume or infer complete BBs from these entries. LET THE REPORT PREPARER DO THAT. Rather, use the names of people or objects in the report, or the action words, by recording the given part of the BB only, leaving the other part of the BB either blank or represented by a question mark.

2. Checking your BBs.

The quality assurance process first addresses the adequacy of the building blocks created during the investigation. If they are flawed, the investigation is likely to be flawed. Each BB used on a Matrix should be checked for content and form, as it is placed on the Matrix, and again when the Matrix has been competed prior to its use for problem definition and assurance purposes.

Before accepting a final STEP-MES-based Matrixes, check each BB to verify that

1. It contains at least the essential content - the source, actor, action and descriptors, or ? if appropriate in lieu of these three required elements, and a many of the other contents as available.

   

2. The name for each actor is unique and used consistently in all BBs for that actor.

3. Each BB is free of the following poison words (From Guide 1):
   

4. Each BB has sufficient information to identify about when it happened, at least relative to other BBs, so the BB can be placed in the right column on a Matrix.

5. Each BB with a ? in the box has been reviewed to be ensure the data is not available, or that provision has been made to acquire the missing data, or that STEP-MES-Trees or other form of logic tree structure will be used to try to postulate what probably occurred, before closing the investigation.

6. Any codes used are identified and recorded in a master list of codes.

7. The sources have all been identified and retained where necessary or when so instructed.

When an BB has passed these checks, it is ready for use in subsequent IQA tasks.

After BBs have been underlined or circled (highlighting also is OK too) all the "BBs" in the document, go to the next step.

3. Organize the highlighted BBs.

Record all the BB building blocks (actor + action +source + any descriptive words needed to describe the action) on a medium like "POST-IT"&tm; sticky note pads. As you record them, put each new BB on a Matrix laid out to provide time and actor coordinates. Use one row for each actor, and a time line across the top with tick marks to indicate approximate times under which the BBs are aligned. Arrange the BBs so their order from left to right represents the relative time when they occurred. As this display progresses, look at each BB relative to each other BB to be sure each is in the right sequence. If two actors were doing something at the same time, the BBs should be placed one above or below the other, to indicate that the actors represented by the rows did something at the same time - in parallel, rather than in series.

4. Apply sequencing tests

When all the highlighted BB building blocks have been placed in their proper rows and in their proper time "columns" along the rows, review the Matrix to see if they are aligned properly according to their spatial location during the episode.

5. Add partial BBs.

Now add BBs for which you have only the actor or the action - not both - and place those BBs on your Matrix. At this time, it is also convenient to review again the source of your BBs, and add BBs that can be inferred reasonably. When all the BBs are entered, review each pair to ascertain whether the order in which they are displayed is logical with respect to time and space. If not rearrange the BBs to array them into their proper order. When both time and spatial relationships have been tested and are satisfactory, proceed to the next step.

6 Determine input/output relationships among BBs

This next step is a critical quality assurance test step. (See Guide 2 for additional information about this procedure.) Add linking arrows to describe the sequences or flow of the changes that produced the outcome you are investigating, from the first BB in the episode process through the last BB (outcome) in the process. An inability to add input/output links shows where you have a quality assurance problem - in the form of an investigation data, data processing or reporting failure - and shows where you have problems with the episode description or the reported data.

The procedure is based on working with BB pairs or sets on your Matrix. You begin with the right-most (earliest in time) BB pair on the Matrix and examine the BB pair for a input/output relationship. If the left BB had to occur to make the right BB occur, you have established an initial "necessary" input/output relationship between the BBs. In other words, the right BB could not have occurred until the left BB occurred IN THIS EPISODE. (Do NOT introduce expected relationships yet; forget about what usually happens and focus on what the report says did happen in this episode.)

7. Enter input/output links.

The next step is to examine the same BBs to establish a "sufficient" logical relationship between the BBs. If the left BB was the only BB that was required for the right BB to occur, you have established an initial "sufficient" input/output relationship between the BBs. In other words, the left BB was both necessary and sufficient to produce the right BB of the pair, and in this episode did so.

At that point, you draw a linking arrow from the left BB to the right BB, signifying a input/output link between the two BBs. Each linking arrow on your work sheet shows you that you have a input/output relationship between two BBs.

Often you will find that more than one BB is must occur before another BB will occur. This is similar to an "AND" logic gate in a fault tree (See Figure 1.) "OR" logic gates are not permitted in a final episode description because they indicate uncertainty about what happened. It is better to use a question mark if the input/output links can not be established, to indicate the uncertainty in the description. If you find that the left BB was not sufficient by itself to produce the right BB, you begin to look for the other BB(s) that HAD TO OCCUR to produce the right BB. For each linked BB pair, draw a linking arrow between them.

Necessary and sufficient tests may produce input/output linked (1) pairs , (2) converging BB sets or (3) diverging BBs sets. Uncertainties are indicated by a question mark between the BBs, indicating an unmet data need. Uncertainties are not objectionable if they are faithfully represented in the text of the report.

After you establish all the necessary and sufficient input/output relationships among the initial BBs pair, repeat the same logical reasoning for each pair of BBs on the Matrix. If any BBs are not linked to the other BBs, those BBs identify problems with the episode description that you should resolve. At the conclusion of this process, you know what the episode description contains, and how well it enables users to visualize and understand the episode process. If the Matrix display has gaps or flawed logic, they are apparent from your IQA Matrix. The output can be returned to the preparer with concrete shortcomings very visible for all to see ( and correct.)

b. Continue IQA process for Needs and Recommended Actions.

Upon completion of the Matrix IQA tasks, the needs statements and recommended actions tasks described in the preceding section are implemented to complete the IQA task.

HANDLING PROBLEMS DISCLOSED BY THE IQA TASK

IQA reviewers quickly discover several kinds of problems after doing the input/output links for the episode. If you are an independent IQA reviewer, try to get the investigator to fix any problems you note.

1. A typical problem is that Matrixes have gaps in the links between the first and last BBs. Work with the IQA sponsor (bill payer) to determine if you should

   ask for more data through further investigation,

   report the gaps in the documentation, or

   request hypotheses to provide potential descriptions to fill the gaps using systematic methods like STEP-MES-based BackSTEP logic trees, etc., or

   reduce the scope of the review or report.

   If you accept the gaps, recognize you are depriving users of potential corrective action choices and the investigation is more likely to promote less effective recommendations, or misdirect action efforts.

2. A second kind of problem is extraneous BBs that are left over after completing the input/output links. What do you do with them? The best course of action is to remove them from the description, because they mislead rather than illuminate users of the reports, and provide "hooks" for others to grasp and raise irrelevant, unnecessary and invalid questions about the episode. At their worst, they divert efforts to assurance future performance from bona fide needs demonstrated by the episode.

3. A third kind of problem arises when BBs are flawed. The report does not transform the episode data into useful building blocks, so you can't establish or validate input/output links or BB displays. This is a clear warning that the report is of questionable"#FAF818" quality.

4. A fourth kind of problem is that some managers demand a "simple" description of a complex episode, like a "root cause" or "causal factors." The Matrix provides a way to present the complexity of the episode that most managers will accept when it is demonstrated. Remember, a properly completed STEP-MES Matrix describes all the necessary and sufficient BBs - and only those BBs - that had to occur to produce the outcome. Presenting the most effective, efficient changes helps refocus managers attentions in desirable channels.

5. A fifth kind of problem is that individuals keeping episode statistics insist that investigators completing their forms even though the episode data do not fit the blanks on forms. This can be addressed by providing the IQA Matrixes with the forms, and letting whoever wants the forms completed fill in the forms on a best efforts basis. Unfortunately, the statistician's solution is to raise the level of abstraction of the data and blur detail until the investigator's data can be fit into the blank. The consequences of this on the statistical outputs are obvious - GIGO.

   Most forms will not fare well with this quality assurance method, but be assured the problem is with the forms, rather than the IQA method. For example, how does one state the time and date of an accident when the accident stretches out over several days, as occurs in some hazardous materials transportation incidents. "Cause of the accident" blanks create similar problems.
   It is uncommon for the narrative section of forms to fare much better when IQA checked this way, but that problem can be ameliorated by providing the Matrix to the investigator, forms preparer and user.

6. Another problem is that individuals who want to make unwarranted judgments or draw unjustified conclusions, or propose actions to solve last year's problems instead of the problems demonstrated by this episode, or insist on a single cause or root cause of the episode will get quite upset with your IQA results. Your IQA outputs with their multiple input/output links and "diamond" tabs expose sloppy logic, loose interpretations of data, subjective characterizations, or unsubstantiated hypotheses, and are very frustrating to hip shooters used to having their flawed outputs and abstractions accepted uncritically. You will just have to endure their frustrations and hope they will begin to understand your way over time.

7. The problem of hidden assumptions is one of the more subtle and difficult problems to detect during the quality assurance checks. For example, a report will propose more training, which assumes the additional training will be valid. Or it assumes prior training was valid, but just not frequent enough. Investigators must guard against these logic failures by challenging each statement in a report that is not clearly supported by BBs or BB set or links showing relevant interactions.

[1]  See Hendrick & Benner "Investigating Accidents with STEP" Marcel Dekker, NY 1987 for discussions of these aspects of investigations and assurance of their quality.