METACOG-25 (May, 2025)
Second Workshop on Metacognitive Prediction of AI Behavior
Held at SDM-2025: https://www.siam.org/conferences-events/siam-conferences/sdm25/
Video release form: https://neurosymbolic.asu.edu/wp-content/uploads/sites/28/2025/03/Video-Consent-and-Release-Form.pdf
Date: May 1, 2025 10:00am-3:30pm
Location: Alexandria, VA (The Westin Alexandria Old Town Hotel, room Edison C)
Part of SIAM Data Mining 2025 (SDM-25) (SDM occurs May 1-3, 2025 – full schedule here)
Thanks to our sponsor, SSCI for their support of this event.
Main Keynote: Andrea Stocco
What is “meta” in metacognition? Insights from brain’s cognitive architecture
Andrea Stocco is a computational cognitive neuroscientist from Friuli, Italy. Dr. Stocco earned his Ph.D. from the University of Trieste and completed postdoctoral work at Carnegie Mellon University. He is now an associate professor of Psychology and an adjunct associate professor of Computer Science at the University of Washington, Seattle.
Most of my Dr. Stocco’s work focuses on characterizing the architecture of the human brain from a computational point of view, investigating which fundamental algorithms are used by the brain and how they can be identified from neural and behavioral data. His research uses multiple methods, including neuroimaging (EEG and fMRI) and neurostimulation (Transcranial Magnetic Stimulation) techniques, with the goal of developing predictive models of brain-behavior relationships in normal populations and in patients–a field that is also known as computational psychiatry.
In 2013, Dr. Stocco co-developed the first non-invasive human brain-to-brain interface, and, in 2019, the first non-invasive multi-directional, multi-person brain network.
Talks will be 13 minutes each (10 minutes plus 3 minutes Q&A)
These posted papers are non-archival, but authors will be invited to submit to a special issue of IEEE Intelligent Systems.
Tentative Schedule
| Time | Session | Talks |
| 10:00-10:05 | Workshop Introduction Paulo Shakarian, ASU and Nathaniel Bastian, USMA | |
| 10:05-10:20 | Morning Keynote: Bonnie Johnson, NPS | Synthetic Metacognition for Managing Tactical Complexity |
| 10:20-10:55 | Session 1: Metacognitive Conditions Chair: Sergei Nirenburg, RPI | Relevance Scoring as a Feature of Metacognitive Artificial Intelligence Alexander Michael Berenbeim, Nathaniel D. Bastian Exploring Cognitive Attributes in Financial Decision-Making Mallika Mainali, Rosina Weber |
| 10:55-11:55 | Session 2: Explainable Metacognition and T&E Chair: Nathaniel Bastian, USMA | Explaining model robustness: combining saliency maps and natural robustness testing Alexander Lynch, Bharadwaj Ravichandran, Brandon RichardWebster, Emily Veenhuis, Stephen Crowell, Roderic Collins, Austin Whitesell, Anthony Hoogs, Brian H Hu Meta-Cognitive Empowerment Jaehoon Choe, Jinhong K. Guo, Valerie Champagne, Aysha Khan, Andrzej Banaszuk, Tsai-Ching Lu Why the Agent Made that Decision: Contrastive Explanation Learning for Reinforcement Learning Rui Zuo, Zifan Wang, Simon Khan, Garrett Ethan Katz, Qinru Qiu Combinatorial Testing Applications for Artificial Intelligence Metacognition Erin Lanus, Laura J. Freeman |
| 11:55-12:25 | Main Keynote Andrea Stocco, University of Washington | What is “meta” in metacognition? Insights from brain’s cognitive architecture |
| 12:25-1:30 | Lunch break | |
| 1:30-1:35 | Sponsor commentary Todd Hughes, SSCI | |
| 1:35-2:30 | Session 3: Metacognition via HDC Chair: Paulo Shakarian, ASU | Perspectives on Hyperdimensional Computing for Metacognitive Artificial Intelligence Zhenzhu Nelson, Mohsen Imani, Nathaniel D. Bastian Metacognitive Modeling with Hyperdimensional Computing Zhuowen Zou, Nathaniel D. Bastian, Mohsen Imani Hyperdimensional Computing for Metacognition Peter Sutor, Cornelia Fermuller, Yiannis Aloimonos Brain-inspired Cognition in Next Generation Racetrack Memories Asif A. Khan, Jerónimo Castrillón, Alex K. Jones |
| 2:30-3:25 | Session 4: Metacognition and Foundation Models Chair: Peter Sutor, UMD | Metacognitive Artificial Intelligence in Vision Foundation Models: Research Challenges Shahriar Rifat, A. Q. M. Sazzad Sayyed, Milin Zhang, Nathaniel D. Bastian, Francesco Restuccia Metacognitive Large Language Models for Feature Optimization and Scientific Discovery Chandan K. Reddy Metacognition in content-centric computational cognitive (C4) modeling Sergei Nirenburg, Marjorie McShane, Sanjay Oruganti Understanding the Uncertainty of LLM Explanations from Reasoning Topology Longchao Da, Xiaoou Liu, Jiaxin Dai, Lu Cheng, Yaqing Wang, Hua Wei |
| 3:25-3:30 | Closing Remarks Paulo Shakarian, ASU |
Organizing Committee:
Co-Chair: Paulo Shakarian, Arizona State University
Co-Chair: Nathaniel D. Bastian, Defense Advanced Research Projects Agency (DARPA) | United States Military Academy
Gerardo I. Simari, Universidad Nacional del Sur in Bahia Blanca, Argentina
Mario Leiva, Institute for Computer Science and Engineering, Argentina
Contact Information:
Paulo Shakarian, [email protected]; 699 S. Mill Ave., Tempe AZ 85281
Workshop Description:
As artificial intelligence (AI) becomes more prevalent in an increasing number of practical applications and systems, improved characterization of such systems will, in-turn, become important to ensure that such systems are resilient, safe, and reliable in the environments for which they are deployed – which may often differ from data used in training. However, while AI systems, often using supervised machine learning or reinforcement learning, have provided excellent results for a variety of applications, the reasons behind their failure modes – or anomalous behavior they engage in – are generally not well understood. The idea of metacognition, reasoning about an AI system itself, is a key avenue to understanding the behavior and performance of machine learning systems. Recently, a variety of methodologies have been explored in the literature, which include stress testing of robotic systems [1], model introspection [2], model certification [3], and performance prediction [4]. Moreover, researchers across multiple disciplines including computer science, control theory, mechanical engineering, human factors, and business schools have explored these problems from different angles. The objectives of the workshop are as follows:
- Survey various approaches to metacognition of AI systems
- Understand the requirements for various metacognitive approaches
- Identify novel methods for metacognition that drive improved AI performance in an operational or cross-domain setting
- Identify application areas suitable for the deployment of metacognitive methods
- Understand the relationship between AI metacognition and human operators