{"id":1186150,"date":"2026-09-16T08:40:40","date_gmt":"2026-09-16T15:40:40","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?post_type=msr-event&p=1186150"},"modified":"2026-09-16T21:14:20","modified_gmt":"2026-09-17T04:14:20","slug":"advancing-the-science-of-ai-agent-evaluation","status":"publish","type":"msr-event","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/event\/advancing-the-science-of-ai-agent-evaluation\/","title":{"rendered":"Advancing the science of AI agent evaluation\u00a0and governance"},"content":{"rendered":"\n\n\n\n\n

As AI agents move from research prototypes into systems that act on people’s behalf, our ability to evaluate them has not kept pace. Rigorous, shared measurement science is the foundation of accountability, and it is currently one of the field’s most significant gaps.\u202f<\/p>\n\n\n\n

On October 22\u201323, 2026, Microsoft Research and Carnegie Mellon University’s AI Measurement Science & Engineering Center (AIMSEC) (opens in new tab)<\/span><\/a> will convene 120 key leaders across academia, industry, civil society, and government<\/strong> to define shared standards and measurement science for evaluating and governing AI agents.\u202f<\/p>\n\n\n\n

The program combines keynote presentations, lightning talks, panel discussions, and hands-on working groups addressing AI agent evaluation frameworks, tooling, and domain-specific challenges. Attendees will participate directly in breakout discussions that will inform open public synthesis reports for the field.\u202f<\/p>\n\n\n\n

\n\t\n\t
\n\t\t
\n\t\t\t
\n\n\n\n
\n
\n
\"calendar<\/figure>\n\n\n\n
<\/div>\n\n\n\n

October 22\u201323, 2026<\/h3>\n<\/div>\n\n\n\n
\n
\"clock<\/figure>\n\n\n\n
<\/div>\n\n\n\n

9:00 AM \u2013 5:00 PM<\/h3>\n<\/div>\n\n\n\n
\n
\"globe<\/figure>\n\n\n\n
<\/div>\n\n\n\n

Microsoft Research New York City Lab<\/h3>\n<\/div>\n\n\n\n
\n
\"marker<\/figure>\n\n\n\n
<\/div>\n\n\n\n

300 Lafayette St, New York, NY 10012<\/h3>\n<\/div>\n<\/div>\n\n\n\n
<\/div>\t\t<\/div>\n\t<\/div>\n\n\t<\/div>\n\n\n\n
\n\t\n\t
\n\t\t
\n\t\t\t

Workshop goals<\/h2>\n\n\n\n
\n\n\n\n
\n
\n

Convening the people who will define evaluation standards<\/h3>\n\n\n\n

In addition to team members from Microsoft and CMU, the workshop brings together researchers from leading academic institutions\u202f\u2014 including Stanford, Berkeley, NYU, and Cornell Tech\u202f\u2014 alongside practitioners from frontier AI organizations including OpenAI, Anthropic, Google DeepMind, Meta, and Amazon. <\/p>\n<\/div>\n\n\n\n

\n

Surfacing the state of the art<\/h3>\n\n\n\n

The program combines lightning talks, panels, and structured working sessions. Presenters will share emerging evaluation frameworks and tooling, giving the research community early visibility into approaches being developed inside frontier labs, and giving those labs direct critique from independent experts. Sessions cover both the horizontal dimension, which addresses general challenges and requirements for agent evaluation, and the vertical dimension, which focuses on domain-specific and application-specific approaches.<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

Producing durable public outputs<\/h3>\n\n\n\n

Beyond the convening itself, the workshop is structured to generate artifacts of lasting value to the field: a synthesis of open problems in agent evaluation, documented consensus and disagreement on emerging best practices, and a map of where current methods fall short. These will be published openly following the workshop.<\/p>\n<\/div>\n\n\n\n

\n

Building the community that sustains this work<\/h3>\n\n\n\n

Measurement science advances through sustained relationships, not one-off events. This convening is intended as the start of an ongoing cross-sector collaboration on agent evaluation, seeding the working groups and partnerships that carry the work forward.<\/p>\n<\/div>\n<\/div>\t\t<\/div>\n\t<\/div>\n\n\t<\/div>\n\n\n\n

<\/div>\n\n\n\n

Organizing committee<\/h2>\n\n\n\n
\n
\n
\"Sarah<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Sarah Bird<\/a><\/h3>\n\n\n\n

Chief Product Officer of Responsible AI, Microsoft<\/p>\n<\/div>\n\n\n\n

\n
\"Emma<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Emma Bowers (opens in new tab)<\/span><\/a><\/h3>\n\n\n\n

Senior Business Manager, Microsoft<\/p>\n<\/div>\n\n\n\n

\n
\"2024<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Wesley Deng<\/a><\/h3>\n\n\n\n

Senior Researcher, Microsoft Research New York City<\/p>\n<\/div>\n\n\n\n

\n
\"Austin<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Austin Krause (opens in new tab)<\/span><\/a><\/h3>\n\n\n\n

AIMSEC Project Manager, Carnegie Mellon University<\/p>\n<\/div>\n\n\n\n

\n
\"Ramayya<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Ramayya Krishnan (opens in new tab)<\/span><\/a><\/h3>\n\n\n\n

Professor & Dean Emeritus, Carnegie Mellon University<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n
\"Hojun\u202fJin<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Hojun\u202fJin (opens in new tab)<\/span><\/a><\/h3>\n\n\n\n

Master of Information Systems Management student, Carnegie Mellon University<\/p>\n<\/div>\n\n\n\n

\n
\"Elaina<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Elaina Maffeo (opens in new tab)<\/span><\/a><\/h3>\n\n\n\n

Microsoft Research Community Initiatives<\/p>\n<\/div>\n\n\n\n

\n
\"Mehrnoosh<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Mehrnoosh Sameki (opens in new tab)<\/span><\/a><\/h3>\n\n\n\n

Principal Product Lead, Microsoft<\/p>\n<\/div>\n\n\n\n

\n
\"Jenn<\/figure>\n\n\n\n
<\/div>\n\n\n\n

Jenn Wortman Vaughan<\/a><\/h3>\n\n\n\n

Senior Principal Research Manager, Microsoft Research New York City<\/p>\n<\/div>\n\n\n\n

\n
\"Hanna<\/a><\/figure>\n\n\n\n
<\/div>\n\n\n\n

Hanna Wallach<\/a><\/h3>\n\n\n\n

Vice President and Distinguished Scientist, Microsoft Research New York City<\/p>\n<\/div>\n<\/div>\n\n\n\n

<\/div>\n\n\n\n\n\n

Microsoft\u2019s mission is to empower every person and every organization on the planet to achieve more. This includes virtual events Microsoft hosts and participates in, where we seek to create a respectful, friendly, and inclusive experience for all participants. As such, we do not tolerate harassing or disrespectful behavior, messages, images, or interactions by any event participant, in any form, at any aspect of the program including business and social activities, regardless of location. We do not tolerate any behavior that is degrading to any gender, race, sexual orientation or disability, or any behavior that would violate Microsoft\u2019s Anti-Harassment and Anti-Discrimination Policy, Equal Employment Opportunity Policy, or Standards of Business Conduct<\/a>. In short, the entire experience must meet our culture standards. We encourage everyone to assist in creating a welcoming and safe environment. Please report (opens in new tab)<\/span><\/a> any concerns, harassing behavior, or suspicious or disruptive activity. Microsoft reserves the right to ask attendees to leave at any time at its sole discretion.<\/p>\n\n\n\n

\n
Report a concern<\/a><\/div>\n<\/div>\n\n\n\n\n\n\n\n