Consistent Practices for Characterizing the Detection Limits of Inspection Systems (POD)

NASA-STD-5009 requires that successful flaw detection by NDE methods be statistically qualified for use on fracture critical metallic components using Probability of Detection (POD) studies, but does not standardize practices. This task works towards standardizing calculations and record retention with a web-based tool, the NASA POD Standards Library, or NPSL. Best practices for test methods and specimen storage will also be provided with appendices to NASA-STD-5009, which is currently unded revision. Additionally, this appendix will describe how POD specimens used to qualify NDE systems will be cataloged, stored and protected from corrosion, damage, or loss.

Background

Flaw detection capability is established for inspection systems on the basis of Probability of Detection (POD).  The commonly accepted metric for an adequate inspection system is as follows: for a minimum flaw size which is smaller than the critical defect being sought, there is 90% probability of detection with 95% confidence (90/95 POD).  Inspection systems that are incapable of meeting the 90/95 POD requirement at or below the critical defect level for fracture critical components are deemed unsuitable for that inspection.  

To provide an efficient and accurate methodology that yields estimates of POD and confidence bounds for Hit-Miss or signal amplitude testing the directed design of experiments for probability of detection (DOEPOD) method has been developed.  In DOEPOD, signal amplitudes are reduced to Hit-Miss data by defining a signal threshold.  The directed DOEPOD method uses a nonparametric approach for the analysis of inspection data which, unlike other methods, does not rely on simlifying assumptions regarding the general form of a POD function.  This differs with other methods that define a POD curve based on a curve fit and does not assume increasing detection with increasing flaw sizes that can often be proven untrue.  For a given sample set, the DOEPOD procedure identifies whether the minimum requirement of 90% probability of detection with 95% confidence is demonstrated for a minimum flaw size and for all greater flaw sizes (90/95 POD).  These procedures are sequentially executed to minimize the number of samples needed to demonstrate that there is a 90/95 POD lower confidence bound at a given flaw size and that the POD is monotonic for flaw sizes exceeding that 90/95 POD flaw size.

This work provides strong experimental and simulation evidence that a 90/95 POD flaw size will be identified by DOEPOD 95% of the time if it exists, and the procedures will yield a determination that the POD is non-monotonic 97% of the time when it is non-monotonic.  Based on this evidence, the DOEPOD methodology may be used to reduce mission risk by quantifiably meeting the requirements of NASA-STD-5009, “Nondestructive Evaluation Requirements for Fracture Critical Metallic Components.”

Approach

A total of 860 metal specimens have been produced with a selection of fatigue cracks, fastener hole cracks, lack of weld fusion, and electrical discharge machined (EDM) flaws.  The metals used in this study are common throughout aerospace, and include aluminum, titanium, nickel-chromium alloy, and stainless steel.  Flat plates and tubes with programmed defects are being examined.  These specimens are being examined with x-ray radiography with differing film densities, digital radiography, ultrasound (including phased array), eddy current (including automated methods), florescent penetrant testing (L3 & L4), magnetic particle testing, and visual testing. 

Customers

All NASA centers and missions utilizing failure critical components.

Products

The data set generated by this study is vast.  Specialized web-based software, termed the NNWG NDE Standards Library (NPSL), is being developed to archive and analyze all data and results.  While this web application is being developed and validated to provide analytical support for this study, it will also be expanded upon to provide a centralized and living NDE capability database.  Further, it will also provide NDE experts with a standard POD analysis tool for future studies.  

Lessons learned during this study will be published in guidance documents for designing statistically adequate POD tests, as new standards, or as appendices to NASA-STD-5009. A comparison to other POD estimation methods will be made to identify validation gaps in methods used for failure critical inspections increasing overall mission success.  (NASA/TM–2014-218183). 

Project Manager

Charles Nichols
Johnson Space Center/White Sands Test Facility
(575) 524-5389
charles.nichols@nasa.gov

Technical Point of Contact

Edward R. Generazio
Langley Research Center
(575) 864-4968
edward.r.generazio@nasa.gov

Accomplishments

• 100 percent of the flawed samples have been manufactured and delivered for testing.

• Publication of DOEPOD methodology. 70% of the specimens have been tested.

• DOEPOD v.1.2 and manual available.

• Familiarization training on LaRC/GSFC file structure and DOEPOD complete.

• Electronic/hard copy data consolidated

• NPSL software requirements agreed to including cataloging and searching capabilities.

• POD literature review and draft ‘Best Practices’ completed.

• Review outsourced to POD experts (Brown, Aldrin, Volovoi) and NESC NDE TDT statused.