NASA-STD-5009 requires that successful flaw detection by NDE methods be statistically qualified for use on fracture critical metallic components using Probability of Detection (POD) studies, but does not standardize practices. This task works towards standardizing calculations and record retention with a web-based tool, the NASA POD Standards Library, or NPSL. Best practices for test methods and specimen storage will also be provided with appendices to NASA-STD-5009, which is currently unded revision. Additionally, this appendix will describe how POD specimens used to qualify NDE systems will be cataloged, stored and protected from corrosion, damage, or loss.
Flaw detection capability is established for inspection systems on the basis of Probability of Detection (POD). The commonly accepted metric for an adequate inspection system is as follows: for a minimum flaw size which is smaller than the critical defect being sought, there is 90% probability of detection with 95% confidence (90/95 POD). Inspection systems that are incapable of meeting the 90/95 POD requirement at or below the critical defect level for fracture critical components are deemed unsuitable for that inspection.
To provide an efficient and accurate methodology that yields estimates of POD and confidence bounds for Hit-Miss or signal amplitude testing the directed design of experiments for probability of detection (DOEPOD) method has been developed. In DOEPOD, signal amplitudes are reduced to Hit-Miss data by defining a signal threshold. The directed DOEPOD method uses a nonparametric approach for the analysis of inspection data which, unlike other methods, does not rely on simlifying assumptions regarding the general form of a POD function. This differs with other methods that define a POD curve based on a curve fit and does not assume increasing detection with increasing flaw sizes that can often be proven untrue. For a given sample set, the DOEPOD procedure identifies whether the minimum requirement of 90% probability of detection with 95% confidence is demonstrated for a minimum flaw size and for all greater flaw sizes (90/95 POD). These procedures are sequentially executed to minimize the number of samples needed to demonstrate that there is a 90/95 POD lower confidence bound at a given flaw size and that the POD is monotonic for flaw sizes exceeding that 90/95 POD flaw size.
This work provides strong experimental and simulation evidence that a 90/95 POD flaw size will be identified by DOEPOD 95% of the time if it exists, and the procedures will yield a determination that the POD is non-monotonic 97% of the time when it is non-monotonic. Based on this evidence, the DOEPOD methodology may be used to reduce mission risk by quantifiably meeting the requirements of NASA-STD-5009, “Nondestructive Evaluation Requirements for Fracture Critical Metallic Components.”
A total of 860 metal specimens have been produced with a selection of fatigue cracks, fastener hole cracks, lack of weld fusion, and electrical discharge machined (EDM) flaws. The metals used in this study are common throughout aerospace, and include aluminum, titanium, nickel-chromium alloy, and stainless steel. Flat plates and tubes with programmed defects are being examined. These specimens are being examined with x-ray radiography with differing film densities, digital radiography, ultrasound (including phased array), eddy current (including automated methods), florescent penetrant testing (L3 & L4), magnetic particle testing, and visual testing.
All NASA centers and missions utilizing failure critical components.
The data set generated by this study is vast. Specialized web-based software, termed the NNWG NDE Standards Library (NPSL), is being developed to archive and analyze all data and results. While this web application is being developed and validated to provide analytical support for this study, it will also be expanded upon to provide a centralized and living NDE capability database. Further, it will also provide NDE experts with a standard POD analysis tool for future studies.
Lessons learned during this study will be published in guidance documents for designing statistically adequate POD tests, as new standards, or as appendices to NASA-STD-5009. A comparison to other POD estimation methods will be made to identify validation gaps in methods used for failure critical inspections increasing overall mission success. (NASA/TM–2014-218183).
Johnson Space Center/White Sands Test Facility
Technical Point of Contact
Edward R. Generazio
Langley Research Center
100 percent of the flawed samples have been manufactured and delivered for testing.
Publication of DOEPOD methodology. 70% of the specimens have been tested.
DOEPOD v.1.2 and manual available.
Familiarization training on LaRC/GSFC file structure and DOEPOD complete.
Electronic/hard copy data consolidated
NPSL software requirements agreed to including cataloging and searching capabilities.
POD literature review and draft ‘Best Practices’ completed.
Although required for fracture critical metallic components per NASA-STD-5009 and used throught the Agency to qualify inspection capabilities, the conduct of and tools for Probability of Detection (POD) analyses are not standardized. Software used for POD analyses are not validated meeting Agency standards spelled out in NPR 7150.2B. Because of this and lack of configuration control, the results of analyses completed today on the same data set do not arrive at the same results. Further, no system exists for cateloging POD data nor specimens used across the agency to provide tracability and reduce needless rework. Specimens used in POD studies are costly to produce and many have been damaged through mishandling and improper storage. This project applies NASA's common 90/95 POD methods and (1) drafts an appendix to the NASA-STD-5009, focusing on best practices for performing POD analyses and storing NDT specimens. (2) An online database, meeting Agency backup and software requirements, will be produced to catelog POD reports, log data, and track specimen locations. (3) The online tool will later be enhanced, perhaps in collaboration with AFRL, with common analysis routines used in POD studies. This software will be validated meeting the stringent standards set forth in NPR 7150.2B and will be configuration controlled.More »
|Organizations Performing Work||Role||Type||Location|
|White Sands Test Facility (WSTF)||Lead Organization||NASA Facility||Las Cruces, New Mexico|
|Goddard Space Flight Center (GSFC)||Supporting Organization||NASA Center||Greenbelt, Maryland|
|Johnson Space Center (JSC)||Supporting Organization||NASA Center||Houston, Texas|
|Langley Research Center (LaRC)||Supporting Organization||NASA Center||Hampton, Virginia|
|Marshall Space Flight Center (MSFC)||Supporting Organization||NASA Center||Huntsville, Alabama|
Draft appendices to NASA-STD-5009 was completed to identify the overall POD process for NASA partner organizations. Specimen storage best practices were also provided, addressing a longstanding shortcoming with costly NASA physical reference standards. An online database was generated (TRL-2 to TRL-8), providing a single location for obtaining a listing of all existing physical reference standards - potentially saving the agency over one million dollars in the first 5 years. Center tribalism must be overcome, and agency-wide funding provided, to populate and sustain the database bringing it to full adoption and TRL-9. Further, additional support and push are required at the top level to fully implement all NASA-STD-5009 appendices.More »