Safety Data: Signal and Noise Continued from page 49 Qualitative data, like its quantita- tive cousin, also exist across a range of forward- and backward-looking measures. A few qualitative data sources we might regularly encoun- ter in aviation ground service are: ■■ Responses to training feedback questions ■■ Safety or hazard report responses ■■ Categorization of employee- reported competence ■■ ■■ ■■ Interview notes from inter- nal or external audits Observational data Statements from investigations As with so many things in the aviation business, the choice between qualitative and quantitative data is not an either/or scenario. In fact, scientists often use both in what they refer to as mixed-method research designs. In our case, where data is sparse and rapidly changing, look- ing at problems from both perspec- tives is especially important. Aggregating Data Collecting data is great, but his- tory does not show a lot of problems being solved by simply gathering information. For statistical tools to work correctly, and to generate enough information upon which conclusions can be based, we need a requisite amount of data. How much often depends on the kind of tools we use. Data mining, for instance, might be more appropriate for large, mixed-data sets, whereas a simple data summary might only require a minimal collection of data. The catch is that for most statistical operations, more is better. Having a larger set of data allows for scrubbing (to remove Aviation Business Journal | 3rd Quarter 2017 outliers or faulty observations) and increases the strength of observa- tions and statistical power that can be drawn. Data collection is also criti- cal to effective safety management, and is one of the core elements of safety assurance. Data aggregation, and then analysis, to support safety performance indicators and targets is something a lot of operators still struggle with; but, it is increasingly required to demonstrate effectiveness of safety efforts, whether internally or to satisfy audit standards such as the International Standard for Business Aircraft Handling (IS-BAH). Most aviation operators find that existing software platforms have plenty of capability for securely storing data for analysis; and, in many cases, off-the-shelf spreadsheet or database software can do an admirable job of performing statistical operations. Statistics Once we’ve gathered, sorted and cleaned our data, the goal is to extract some intelligence from it. Intelligence is used purposefully here as a way of distinguishing it from information or knowledge. In a general sense, intelligence is actionable. It puts data in context that has practical use. To turn information into intelli- gence, there are a number of statisti- cal tools at our disposal. Broadly, statistics are either descriptive or inferential, with descriptive statis- tics reporting the news, so to speak, and inferential methods allowing us to identify correlated elements and make projections about some future state. Descriptive statistics allow us to describe the data through mea- sures of shape and spread across the Hawthorne Effect: By virtue of knowing that they are part of an effort to gather data, the behavior or response from individuals may be influenced Confirmation Bias: Confirmation bias stems from our natural desire to support what we already believe Survivorship Bias: Focusing only on those ideas or responses that gain traction introduce bias by ignoring lessons from losing concepts The following are just a few types of bias we must work to control in data collection: Selection Bias: Volunteers, or even purposeful selection of a group, might result in a sample that does not accu- rately reflect the population Detection Bias: Detection bias results from seek- ing greater depth of data in one group than another (maintenance vs. pilots, as an example) Response Bias: Similar to selection bias, and resulting from skewed information because of self-selection Continued on page 53 51
