Automation Interface Design Development

Our research makes its contributions at two levels. At one level, we addressed the problems of interaction between humans and computers/automation in a particular application domain. The domain of application was the planning work done by Attitude Determination and Control Officers (ADCO), part of Mission Control for the International Space Station (ISS). At a second, higher level, we abstracted from this case to suggest a more general method for needs analysis and this was the primary motivation for our research. We briefly summarize work on the ADCO domain, before describing our more general contribution, the methods and tools that emerged from working on this case.

A core aspect of our research was a detailed study of the planning work done by ADCO and identification of the work needs that should be supported by software. As we carried out the analysis of the ADCO planning domain, we changed our characterization of the problem. We realized that rather than focusing on the current tasks and practices, as in Task Analysis, we should try to directly identify their needs, which future software should meet. One result of our work was support for ADCO. We provided an analysis of ADCO needs. We guided development of an illustrative prototype designed to better fit these needs. We conducted an experimental study of this prototype, comparing performance to that with the legacy system. These products are of value to ADCO operators seeking the design of software that is more effective than their current legacy systems. The second, more general result was development of methods and tools for carrying out such analyses. We used the ADCO domain to develop Structure Identification, our approach to needs analysis. We developed Structure Identification to be particularly appropriate for rapid identification of needs for safety critical, technical, information work. Needs analysis based on Structure Identification finds the high-level structure in the work domain and uses this to design the structure of the interaction between the human and computer or automation. We rely on a combination of eliciting function information from expert users, identifying candidate structure from documents and functional descriptions, and vetting the developing characterization with experts.

Structure Identification contrasts with conducting needs analysis based on Task Analysis; task analysis identifies current tasks, yet a change in the work applications naturally brings with it change in the tasks so that matching the old tasks is not a reliable design guide. Task Analysis can be a helpful approach to identifying structure, but we prioritize identifying the domain structure not the activities. Our approach is related to both Work Domain Analysis (WDA) and Contextual Inquiry (CI), in that these also seek to identify stable aspects of work in order to guide design. WDA focuses on identifying constraints, particularly constraints in how a physical system, such as a chemical plant, operates; it is directly applicable to control tasks, but much less applicable to work consisting of finding, transforming, building, and distributing information products. CI methods focus on observing users, typically carrying out office work; this approach is less adequate in highly technical domains where critical aspects of work cannot be understood from watching users.

Our goal is to make needs analysis more efficient and effective. To this end the methods that we developed focused on gathering important information quickly. We consolidated what we learned to make the methods easier to reuse and to apply to another case, by building simple tools as we carried out the needs analysis for the ADCO planning domain and developed the SI approach. These include templates for gathering high-level function information from experts, templates for presenting the identified structure to experts for verification, and templates for comparing the contents of multiple product documents.

An additional contribution of the research was a preliminary assessment of Structure Identification. Broadly, we investigated whether Structure Identification, followed by Structure Matching from the domain to an application structure, contributes to better design of the application. We conducted an illustrative study using the ADCO planning domain. We used the domain structure we had identified to guide design of an experimental prototype for ADCO planning. We conducted an experiment comparing the experimental prototype, which closely matched the domain structure, versus the legacy system, which matched much more poorly. We included a variety of measures, from speed of performance to conceptual understanding and retention of periods of disuse. We predicted differences in performance on a variety of planning tasks that are detailed analogs of simple ADCO planning tasks: overall faster performance by users of the new, well-matched system compared to that by users of the legacy, poorly-matching system; and particular performance advantage for the new system at points where the legacy system most mismatched domain structure. We found that performance times were cut in half for the new prototype vs legacy system on some tasks, accompanied by much lower error rates as well. Further, we also found the predicted pattern of poor performance at legacy points of mismatch.

We ran through the whole design cycle, from needs analysis through evaluation, in the ADCO domain. This process provided an illustrative case showing the feasibility of our approach. The results from our experiment suggest that capturing and matching domain structure may be an efficient, productive way to guide design of interaction between humans and computers/automation for technical information work.