Ask-the-Expert: Minimizing human review for big data analytics through active learning

In this work we propose to semi-automate the process of knowledge discovery from anomaly detection algorithms through the use of active learning. Active learning is an area of research within machine learning that uses an expert in the loop to learn from large data sets that have very few annotations or labels available. In this case, the task can be defined as the identification of safety events from flight operational data. Since traditional detection algorithms cannot differentiate between operationally relevant and irrelevant statistical anomalies, SMEs have a huge burden of investigating each and every example identified by the detection algorithm, classifying and labeling them as relevant or irrelevant. Active learning requires very few labels to start learning the classifier. A positive label indicates an operationally significant safety event whereas a negative label indicates otherwise. Based on these few labels we propose to build an active learning system that utilizes the SME's time in the most effective manner by iteratively asking for labels for few informative instances until a desired accuracy is reached. If the algorithmic work of this CIF is successful, then we expect the implementation and deployment of the system with user interface to be pursued by the Aviation Operations and Safety Program given its interest in safety monitoring and discovery of safety incidents. More »