Towards Committee-Based Anomaly Detection
An important problem of complex autonomous systems is that they cannot provide insight into their behaviors and thought processes. This problem becomes even more pertinent when we consider that autonomous vehicles are taking control of actions previously entrusted to humans, without any explanation of their decisions. We are working on methodologies and technologies to support building robust, articulate systems that use explanations to dynamically detect, and mitigate, anomalous behaviors.
Our goal is to extend the ideas behind articulate machines to full system design, to create complex multipartite systems that use explanation and reasonableness monitoring to dynamically detect and resolve subsystem anomalies. We are inspired by the structure of successful human organizations: tasks are accomplished by (possibly overlapping) committees of multiple people. Committees are able to survive bad work by any single member, because members of the committee observe each other’s work and can jointly decide on actions to correct bad work or remove misbehaving members. Our approach applies this philosophy of committees to subsystems of machines, with the aim of producing more robust and secure systems for safety-critical or mission-critical tasks. Only when systems can explain their actions and decisions will we be confident of their competence and benign intent.
Authors: Leilani Gilpin