Workshop on Metacognitive Prediction of AI Behavior

This event was sponsored by the Army Research Office held on Nov. 13-15, 2023 in Scottsdale, AZ.

As artificial intelligence become more prevalent in military systems, improved characterization of such systems will, in-turn, become important to ensure that such systems are safe and reliable in supporting the warfighter.  However, while AI systems, often using supervised machine learning or reinforcement learning, have provided excellent results for a variety of applications, the reasons behind their failure modes or anomalous behavior are generally not well understood.  The idea of metacognition, reasoning about an AI system itself, is a key avenue to understanding the behavior and performance of machine learning systems.  Recently, a variety of methodologies have been explored in the literature, which including stress testing of robotic systems [1], model introspection [2], model certification [3], and performance prediction [4].  Moreover, researchers across multiple disciplines including computer science, control theory, mechanical engineering, human factors, and business schools have explored these problems from different angles.  The objectives of the workshop are as follows:

• Create a taxonomy of various approaches to metacognition of AI systems
• Understand the requirements for various metacognitive approaches
• Summarize recent results obtained in the study of AI metacognition
• Enumerate current applications for which AI metacognitive techniques have been applied
• Understand the relationship between AI metacognition and human operators

Specific topics to be covered include, but are not limited to:

• Explainable performance prediction of black-box AI systems
• Stress testing of reinforcement learning systems
• How can metacognition be used to increase trust in AI systems by the operator
• Applications of AI metacognition to robotic and vision systems