Saleema Amershi

Partner Research Manager, Microsoft AI Frontiers

Publications

View by:

- SentinelBench: A Benchmark for Long-Running Monitoring Agents
  
  Matheus Kunzler Maldaner, Adam Fourney, Amanda Swearngin, Hussein Mozannar, Gagan Bansal, Maya Murad, Rafah Hosn, Saleema Amershi
  
  June 2026
  
  arXiv | June 2026
  
  Publication
- The Collaboration Gap
  
  Tim R. Davidson, Adam Fourney, Saleema Amershi, Robert West, Eric Horvitz, Ece Kamar
  
  COLM 2026 | November 2025
  
  Publication
- Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets
  
  Gagan Bansal, Wenyue Hua, Zezhou Huang, Adam Fourney, Amanda Swearngin, Will Epperson, Tyler Payne, Jake Hofman, Brendan Lucier, Chinmay Singh, Markus Mobius, Akshay Nambi, Archana Yadav, Kevin Gao, David Rothschild, Aleksandrs Slivkins, Daniel G. Goldstein, Hussein Mozannar, Nicole Immorlica, Maya Murad, Matthew Vogel, Subbarao Kambhampati, Eric Horvitz, Saleema Amershi
  
  MSR-TR-2025-50 | October 2025
  
  Published by Microsoft
  
  Publication Video Video Access
- Magentic-UI: Towards Human-in-the-loop Agentic Systems
  
  Hussein Mozannar, Gagan Bansal, Cheng Tan, Adam Fourney, Victor Dibia, Jingya Chen, Jack Gerrits, Tyler Payne, Matheus Kunzler Maldaner, Madeleine Grunde-McLaughlin, Eric Zhu, Griffin Bassman, Jacob Alber, Peter Chang, Ricky Loynd, Friederike Niedtner, Ece Kamar, Maya Murad, Rafah Hosn, Saleema Amershi
  
  MSR-TR-2025-40 | July 2025
  
  Published by Microsoft
  
  Publication Project
- Interactive Debugging and Steering of Multi-Agent AI Systems
  
  Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney (adamfo), Jack Gerrits, Erkang (Eric) Zhu, Saleema Amershi
  
  CHI 2025 | April 2025
  
  Publication
- Challenges in Human-Agent Communication
  
  Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
  
  MSR-TR-2024-53 | December 2024
  
  Published by Microsoft
  
  Publication Project
- Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
  
  Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang (Eric) Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, Saleema Amershi
  
  MSR-TR-2024-47 | November 2024
  
  Published by Microsoft
  
  Preprint Video Access
- AutoGen Studio: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems
  
  Victor Dibia, Jingya Chen, Gagan Bansal, Suff Syed, Adam Fourney, Erkang (Eric) Zhu, Chi Wang, Saleema Amershi
  
  August 2024
  
  Preprint
  
  PDF Publication Video Video Project
- Supporting Human-AI Collaboration in Auditing LLMs with LLMs
  
  Charvi Rastogi, Marco Tulio Ribeiro, Nicholas King, Harsha Nori, Saleema Amershi
  
  AIES | August 2023
  
  Publication
- Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions
  
  John Chung, Ece Kamar, Saleema Amershi
  
  ACL 2023 | July 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | June 2023
  
  Publication
- AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
  
  Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
  
  June 2023
  
  arXiv | June 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | October 2022
  
  Preprint
- How Different Groups Prioritize Ethical Values for Responsible AI
  
  Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
  
  FAccT 2022 | May 2022
  
  Publication DOI
- REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
  
  Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
  
  FAccT 2022 | May 2022
  
  DOI Project
- Assessing Human-AI Interaction Early through Factorial Surveys: A Study on the Guidelines for Human-AI Interaction
  
  Tianyi Li, Mihaela Vorvoreanu, Derek DeBellis, Saleema Amershi
  
  ACM Transactions on Computer-Human Interaction | April 2022
  
  Publication Project Project
- HINT: Integration Testing for AI-based Features with Humans in the Loop
  
  Quanze Chen, Tobias Schnabel, Besmira Nushi, Saleema Amershi
  
  International Conference on Intelligent User Interfaces | March 2022
  
  Publication Project
- Planning for Natural Language Failures with the AI Playbook
  
  Matthew K. Hong, Adam Fourney, Derek DeBellis, Saleema Amershi
  
  2021 CHI Conference on Human Factors in Computing Systems | May 2021
  
  Publication Video Project
- “Who doesn’t like dinosaurs?” Finding and Eliciting Richer Preferences for Recommendation
  
  Tobias Schnabel, Gonzalo Ramos, Saleema Amershi
  
  14th ACM Conference on Recommender Systems | September 2020
  
  PDF
- The Impact of More Transparent Interfaces on Behavior in Personalized Recommendation
  
  Tobias Schnabel, Saleema Amershi, Paul Bennett, Peter Bailey, Thorsten Joachims
  
  Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) | July 2020
  
  Publication
- Software Engineering for Machine Learning: A Case Study
  
  Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLIne, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, Tom Zimmermann
  
  International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track | May 2019
  
  ICSE 2019 Best Paper Award
  
  PDF
- People and AI See Things Different Implications of Mismatched Perception on HCI for AI Systems
  
  Saleema Amershi, Ece Kamar, Emre Kiciman
  
  Workshop on Human-Centered Machine Learning Perspectives at CHI 2019 | May 2019
- Guidelines for Human-AI Interaction
  
  Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz
  
  CHI 2019 | May 2019
  
  CHI 2019 Honorable Mention Award
  
  PDF Video Project Project
- Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
  
  Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi Iqbal, Jaime Teevan
  
  CHI 2019 | May 2019
  
  Publication Project
- Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems
  
  Rafal Kocielnik, Saleema Amershi, Paul Bennett
  
  CHI 2019 | May 2019
  
  Publication Project
- Machine Teaching: A New Paradigm for Building Machine Learning Systems
  
  Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
  
  MSR-TR-2017-26 | July 2017
  
  Preprint Project
- Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
  
  Joseph Chee Chang, Saleema Amershi, Ece Kamar
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2017) | May 2017
  
  PDF
- A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs
  
  Kristina Toutanova, Chris Brockett, Ke M. Tran, Saleema Amershi
  
  EMNLP | November 2016
  
  PDF PDF PDF Publication Access Project
- Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
  
  Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, Jason Williams
  
  IEEE TVCG, VAST | August 2016
  
  PDF Video Project
- The Label Complexity of Mixed-Initiative Classiï¬er Training
  
  Jina Suh, Xiaojin Zhu, Saleema Amershi
  
  Proceedings of the 33rd International Conference on Machine Learning | June 2016
  
  PDF PDF PDF
- Active Learning with Oracle Epiphany
  
  Tzu-Kuo Huang, Lihong Li, Ara Vartanian, Saleema Amershi, Xiaojin Zhu
  
  30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | May 2016
  
  PDF
- FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification
  
  Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, Patrice Simard
  
  IEEE VAST | October 2015
  
  PDF Project
- ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
  
  Saleema Amershi, Max Chickering, Steven Drucker, Bongshin Lee, Patrice Simard, Jina Suh
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2015) | April 2015
  
  PDF Publication Project Project
- Power to the People: The Role of Humans in Interactive Machine Learning
  
  Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza
  
  AI Magazine | December 2014
  
  PDF Publication
- ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
  
  Patrice Simard, Max Chickering, Apama Lakshmiratan, Denis Charles, Leon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, Jina Suh
  
  MSR-TR-2014-128 | September 2014
  
  PDF Project
- Structured Labeling for Facilitating Concept Evolution in Machine Learning
  
  Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2014) | May 2014
  
  Best Paper Award
  
  PDF PDF
- LiveAction: Automating Web Task Model Generation
  
  Saleema Amershi, Jalal Mahmud, Jeffrey Nichols, Tessa Lau, German Attanasio Ruiz
  
  ACM Trans. Interact. Intell. Syst | October 2013, pp. 23
  
  PDF Publication
- ReGroup: Interactive Machine Learning for On-Demand Group Creation in Social Networks
  
  Saleema Amershi, James Fogarty, Daniel Weld
  
  Proceeding CHI '12 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems | May 2012
  
  PDF Publication
- Designing for Effective End-User Interaction with Machine Learning
  
  Saleema Amershi
  
  October 2011
  
  October 2011
  
  PDF
- Effective End-User Interaction with Machine Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  AAAI (Nectar Track) | August 2011
  
  PDF
- Human-Guided Machine Learning for Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence | July 2011
  
  ISBN: 978-1-57735-515-1
  
  Invited from CHI 2011 to Best Papers Track
  
  PDF Publication Project
- CueT: Human-Guided Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  CHI | May 2011
  
  Best Paper Honorable Mention
  
  PDF Project
- Comparing Web Interaction Models in Developing Regions
  
  Jay Chen, Aditya Dhananjay, Saleema Amershi, Lakshmi Subramanian
  
  December 2010
  
  PDF
- Examining Multiple Potential Models in End-User Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  CHI 2010: Machine Learning and Web Interactions | April 2010
  
  PDF
- Multiple Mouse Text Entry for Single-Display Groupware
  
  Saleema Amershi, Meredith Ringel Morris, Neema Moraveji, Ravin Balakrishnan, Kentaro Toyama
  
  CSCW | February 2010
  
  Best Paper Honorable Mention
  
  PDF
- Overview-Based Example Selection in End-User Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  UIST '09 Proceedings of the 22nd annual ACM symposium on User interface software and technology | October 2009
  
  ISBN: 978-1-60558-745-5
  
  PDF Publication
- Overview-Based Examples Selection in Mixed-Initiative Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  UIST | October 2009
  
  PDF
- Combining Unsupervised and Supervised Classification to Build User Models for Exploratory Learning Environments
  
  Saleema Amershi, CRISTINA CONATI
  
  Journal of Educational Data Mining | September 2009
  
  PDF
- Designing for End-User Interactive Concept Learning in CueFlik
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  (2009) Neural Information Processing Systems (NIPS) Workshop on Analysis and Design of Algorithms for Interactive Machine Learning | August 2009
  
  PDF
- Amplifying Community Content Creation with Mixed-Initiative Information Extraction
  
  Raphael Hoffmann, Saleema Amershi, Kayur Patel, Fei Wu, James Fogarty, Daniel S. Weld
  
  In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2009) | April 2009
  
  Best Paper Honorable Mention
  
  PDF Publication
- CoSearch: A System for Co-located Collaborative Web Search
  
  Saleema Amershi, Meredith Ringel Morris
  
  Proceedings of CHI 2008 | April 2008
  
  ISBN: 978-1-60558-011-1
  
  PDF Video
- Shared Sensemaking: Enhancing the Value of Collaborative Web Search Tools
  
  Meredith Ringel Morris, Saleema Amershi
  
  MSR-TR-2008-205 | April 2008
  
  PDF
- Intelligence in Wikipedia
  
  Daniel S. Weld, Fei Wu, Eytan Adar, Saleema Amershi, James Fogarty, Raphael Hoffmann, Kayur Patel, Michael Skinner
  
  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 08) Senior Papers Track | January 2008
  
  PDF
- Pedagogy and usability in interactive algorithm visualizations: Designing and evaluating CIspace
  
  Saleema Amershi, Giuseppe Carenini, Cristina Conati, Alan K. Mackworth, David Poole
  
  Designing and Evaluating CIspace. Interacting with Computers - The Interdisciplinary Journal of Human-Computer Interaction | September 2007
  
  PDF
- Using Eye-tracking Data for High-Level User Modeling in Adaptive Interfaces
  
  Cristina Conati, Christina Merten, Saleema Amershi, Kasia Muldner
  
  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 07) Nectar Track | July 2007
  
  PDF
- Unsupervised and Supervised Machine Learning in User Modeling for Intelligent Learning Environments
  
  Saleema Amershi, Cristina Conati
  
  In Proceedings of the ACM/SIGCHI Conference on Intelligent User Interfaces (IUI 2007) | January 2007
  
  PDF
- Automatic Recognition of Learner Types in Exploratory Learning Environments
  
  Saleema Amershi, Cristina Conati
  
  ITS'06 Proceedings of the 8th international conference on Intelligent Tutoring Systems | June 2006
  
  PDF Publication
- Using Feature Selection and Unsupervised Clustering to Identify Affective Expressions in Educational Games
  
  Saleema Amershi, Cristina Conati, Heather Maclaren
  
  In Proceedings of The Intelligent Tutoring Systems Workshop on Motivational and Affective Issues in ITS (ITS 2006) | January 2006
  
  PDF
- Designing CIspace: Pedagogy and Usability in a Learning Environment for AI
  
  Saleema Amershi, N. Arksey, G. Carenini, C. Conati, A. Mackworth, H. Maclaren, D. Poole
  
  In Proceedings of the ACM/SIGCSE Conference on Innovation and Technology in Computer Science Education ITiCSE '05 | June 2005
  
  PDF

- SentinelBench: A Benchmark for Long-Running Monitoring Agents
  
  Matheus Kunzler Maldaner, Adam Fourney, Amanda Swearngin, Hussein Mozannar, Gagan Bansal, Maya Murad, Rafah Hosn, Saleema Amershi
  
  June 2026
  
  arXiv | June 2026
  
  Publication
- The Collaboration Gap
  
  Tim R. Davidson, Adam Fourney, Saleema Amershi, Robert West, Eric Horvitz, Ece Kamar
  
  COLM 2026 | November 2025
  
  Publication
- Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets
  
  Gagan Bansal, Wenyue Hua, Zezhou Huang, Adam Fourney, Amanda Swearngin, Will Epperson, Tyler Payne, Jake Hofman, Brendan Lucier, Chinmay Singh, Markus Mobius, Akshay Nambi, Archana Yadav, Kevin Gao, David Rothschild, Aleksandrs Slivkins, Daniel G. Goldstein, Hussein Mozannar, Nicole Immorlica, Maya Murad, Matthew Vogel, Subbarao Kambhampati, Eric Horvitz, Saleema Amershi
  
  MSR-TR-2025-50 | October 2025
  
  Published by Microsoft
  
  Publication Video Video Access
- Magentic-UI: Towards Human-in-the-loop Agentic Systems
  
  Hussein Mozannar, Gagan Bansal, Cheng Tan, Adam Fourney, Victor Dibia, Jingya Chen, Jack Gerrits, Tyler Payne, Matheus Kunzler Maldaner, Madeleine Grunde-McLaughlin, Eric Zhu, Griffin Bassman, Jacob Alber, Peter Chang, Ricky Loynd, Friederike Niedtner, Ece Kamar, Maya Murad, Rafah Hosn, Saleema Amershi
  
  MSR-TR-2025-40 | July 2025
  
  Published by Microsoft
  
  Publication Project
- Interactive Debugging and Steering of Multi-Agent AI Systems
  
  Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney (adamfo), Jack Gerrits, Erkang (Eric) Zhu, Saleema Amershi
  
  CHI 2025 | April 2025
  
  Publication
- Challenges in Human-Agent Communication
  
  Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
  
  MSR-TR-2024-53 | December 2024
  
  Published by Microsoft
  
  Publication Project
- Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
  
  Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang (Eric) Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, Saleema Amershi
  
  MSR-TR-2024-47 | November 2024
  
  Published by Microsoft
  
  Preprint Video Access
- AutoGen Studio: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems
  
  Victor Dibia, Jingya Chen, Gagan Bansal, Suff Syed, Adam Fourney, Erkang (Eric) Zhu, Chi Wang, Saleema Amershi
  
  August 2024
  
  Preprint
  
  PDF Publication Video Video Project
- Supporting Human-AI Collaboration in Auditing LLMs with LLMs
  
  Charvi Rastogi, Marco Tulio Ribeiro, Nicholas King, Harsha Nori, Saleema Amershi
  
  AIES | August 2023
  
  Publication
- Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions
  
  John Chung, Ece Kamar, Saleema Amershi
  
  ACL 2023 | July 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | June 2023
  
  Publication
- AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
  
  Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
  
  June 2023
  
  arXiv | June 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | October 2022
  
  Preprint
- How Different Groups Prioritize Ethical Values for Responsible AI
  
  Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
  
  FAccT 2022 | May 2022
  
  Publication DOI
- REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
  
  Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
  
  FAccT 2022 | May 2022
  
  DOI Project
- Assessing Human-AI Interaction Early through Factorial Surveys: A Study on the Guidelines for Human-AI Interaction
  
  Tianyi Li, Mihaela Vorvoreanu, Derek DeBellis, Saleema Amershi
  
  ACM Transactions on Computer-Human Interaction | April 2022
  
  Publication Project Project
- HINT: Integration Testing for AI-based Features with Humans in the Loop
  
  Quanze Chen, Tobias Schnabel, Besmira Nushi, Saleema Amershi
  
  International Conference on Intelligent User Interfaces | March 2022
  
  Publication Project
- Planning for Natural Language Failures with the AI Playbook
  
  Matthew K. Hong, Adam Fourney, Derek DeBellis, Saleema Amershi
  
  2021 CHI Conference on Human Factors in Computing Systems | May 2021
  
  Publication Video Project
- “Who doesn’t like dinosaurs?” Finding and Eliciting Richer Preferences for Recommendation
  
  Tobias Schnabel, Gonzalo Ramos, Saleema Amershi
  
  14th ACM Conference on Recommender Systems | September 2020
  
  PDF
- The Impact of More Transparent Interfaces on Behavior in Personalized Recommendation
  
  Tobias Schnabel, Saleema Amershi, Paul Bennett, Peter Bailey, Thorsten Joachims
  
  Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) | July 2020
  
  Publication
- Software Engineering for Machine Learning: A Case Study
  
  Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLIne, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, Tom Zimmermann
  
  International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track | May 2019
  
  ICSE 2019 Best Paper Award
  
  PDF
- People and AI See Things Different Implications of Mismatched Perception on HCI for AI Systems
  
  Saleema Amershi, Ece Kamar, Emre Kiciman
  
  Workshop on Human-Centered Machine Learning Perspectives at CHI 2019 | May 2019
- Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems
  
  Rafal Kocielnik, Saleema Amershi, Paul Bennett
  
  CHI 2019 | May 2019
  
  Publication Project
- Guidelines for Human-AI Interaction
  
  Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz
  
  CHI 2019 | May 2019
  
  CHI 2019 Honorable Mention Award
  
  PDF Video Project Project
- Machine Teaching: A New Paradigm for Building Machine Learning Systems
  
  Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
  
  MSR-TR-2017-26 | July 2017
  
  Preprint Project
- Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
  
  Joseph Chee Chang, Saleema Amershi, Ece Kamar
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2017) | May 2017
  
  PDF
- A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs
  
  Kristina Toutanova, Chris Brockett, Ke M. Tran, Saleema Amershi
  
  EMNLP | November 2016
  
  PDF PDF PDF Publication Access Project
- Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
  
  Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, Jason Williams
  
  IEEE TVCG, VAST | August 2016
  
  PDF Video Project
- The Label Complexity of Mixed-Initiative Classiï¬er Training
  
  Jina Suh, Xiaojin Zhu, Saleema Amershi
  
  Proceedings of the 33rd International Conference on Machine Learning | June 2016
  
  PDF PDF PDF
- Active Learning with Oracle Epiphany
  
  Tzu-Kuo Huang, Lihong Li, Ara Vartanian, Saleema Amershi, Xiaojin Zhu
  
  30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | May 2016
  
  PDF
- FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification
  
  Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, Patrice Simard
  
  IEEE VAST | October 2015
  
  PDF Project
- ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
  
  Saleema Amershi, Max Chickering, Steven Drucker, Bongshin Lee, Patrice Simard, Jina Suh
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2015) | April 2015
  
  PDF Publication Project Project
- Power to the People: The Role of Humans in Interactive Machine Learning
  
  Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza
  
  AI Magazine | December 2014
  
  PDF Publication
- ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
  
  Patrice Simard, Max Chickering, Apama Lakshmiratan, Denis Charles, Leon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, Jina Suh
  
  MSR-TR-2014-128 | September 2014
  
  PDF Project
- Structured Labeling for Facilitating Concept Evolution in Machine Learning
  
  Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2014) | May 2014
  
  Best Paper Award
  
  PDF PDF
- Effective End-User Interaction with Machine Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  AAAI (Nectar Track) | August 2011
  
  PDF
- Human-Guided Machine Learning for Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence | July 2011
  
  ISBN: 978-1-57735-515-1
  
  Invited from CHI 2011 to Best Papers Track
  
  PDF Publication Project
- CueT: Human-Guided Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  CHI | May 2011
  
  Best Paper Honorable Mention
  
  PDF Project
- Examining Multiple Potential Models in End-User Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  CHI 2010: Machine Learning and Web Interactions | April 2010
  
  PDF
- Overview-Based Examples Selection in Mixed-Initiative Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  UIST | October 2009
  
  PDF
- Intelligence in Wikipedia
  
  Daniel S. Weld, Fei Wu, Eytan Adar, Saleema Amershi, James Fogarty, Raphael Hoffmann, Kayur Patel, Michael Skinner
  
  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 08) Senior Papers Track | January 2008
  
  PDF
- Pedagogy and usability in interactive algorithm visualizations: Designing and evaluating CIspace
  
  Saleema Amershi, Giuseppe Carenini, Cristina Conati, Alan K. Mackworth, David Poole
  
  Designing and Evaluating CIspace. Interacting with Computers - The Interdisciplinary Journal of Human-Computer Interaction | September 2007
  
  PDF
- Using Eye-tracking Data for High-Level User Modeling in Adaptive Interfaces
  
  Cristina Conati, Christina Merten, Saleema Amershi, Kasia Muldner
  
  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 07) Nectar Track | July 2007
  
  PDF
- Using Feature Selection and Unsupervised Clustering to Identify Affective Expressions in Educational Games
  
  Saleema Amershi, Cristina Conati, Heather Maclaren
  
  In Proceedings of The Intelligent Tutoring Systems Workshop on Motivational and Affective Issues in ITS (ITS 2006) | January 2006
  
  PDF
- Designing CIspace: Pedagogy and Usability in a Learning Environment for AI
  
  Saleema Amershi, N. Arksey, G. Carenini, C. Conati, A. Mackworth, H. Maclaren, D. Poole
  
  In Proceedings of the ACM/SIGCSE Conference on Innovation and Technology in Computer Science Education ITiCSE '05 | June 2005
  
  PDF
- Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets
  
  Gagan Bansal, Wenyue Hua, Zezhou Huang, Adam Fourney, Amanda Swearngin, Will Epperson, Tyler Payne, Jake Hofman, Brendan Lucier, Chinmay Singh, Markus Mobius, Akshay Nambi, Archana Yadav, Kevin Gao, David Rothschild, Aleksandrs Slivkins, Daniel G. Goldstein, Hussein Mozannar, Nicole Immorlica, Maya Murad, Matthew Vogel, Subbarao Kambhampati, Eric Horvitz, Saleema Amershi
  
  MSR-TR-2025-50 | October 2025
  
  Published by Microsoft
  
  Publication Video Video Access
- Interactive Debugging and Steering of Multi-Agent AI Systems
  
  Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney (adamfo), Jack Gerrits, Erkang (Eric) Zhu, Saleema Amershi
  
  CHI 2025 | April 2025
  
  Publication
- Challenges in Human-Agent Communication
  
  Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
  
  MSR-TR-2024-53 | December 2024
  
  Published by Microsoft
  
  Publication Project
- AutoGen Studio: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems
  
  Victor Dibia, Jingya Chen, Gagan Bansal, Suff Syed, Adam Fourney, Erkang (Eric) Zhu, Chi Wang, Saleema Amershi
  
  August 2024
  
  Preprint
  
  PDF Publication Video Video Project
- Supporting Human-AI Collaboration in Auditing LLMs with LLMs
  
  Charvi Rastogi, Marco Tulio Ribeiro, Nicholas King, Harsha Nori, Saleema Amershi
  
  AIES | August 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | June 2023
  
  Publication
- AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
  
  Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
  
  June 2023
  
  arXiv | June 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | October 2022
  
  Preprint
- How Different Groups Prioritize Ethical Values for Responsible AI
  
  Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
  
  FAccT 2022 | May 2022
  
  Publication DOI
- Assessing Human-AI Interaction Early through Factorial Surveys: A Study on the Guidelines for Human-AI Interaction
  
  Tianyi Li, Mihaela Vorvoreanu, Derek DeBellis, Saleema Amershi
  
  ACM Transactions on Computer-Human Interaction | April 2022
  
  Publication Project Project
- HINT: Integration Testing for AI-based Features with Humans in the Loop
  
  Quanze Chen, Tobias Schnabel, Besmira Nushi, Saleema Amershi
  
  International Conference on Intelligent User Interfaces | March 2022
  
  Publication Project
- Planning for Natural Language Failures with the AI Playbook
  
  Matthew K. Hong, Adam Fourney, Derek DeBellis, Saleema Amershi
  
  2021 CHI Conference on Human Factors in Computing Systems | May 2021
  
  Publication Video Project
- “Who doesn’t like dinosaurs?” Finding and Eliciting Richer Preferences for Recommendation
  
  Tobias Schnabel, Gonzalo Ramos, Saleema Amershi
  
  14th ACM Conference on Recommender Systems | September 2020
  
  PDF
- The Impact of More Transparent Interfaces on Behavior in Personalized Recommendation
  
  Tobias Schnabel, Saleema Amershi, Paul Bennett, Peter Bailey, Thorsten Joachims
  
  Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) | July 2020
  
  Publication
- Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems
  
  Rafal Kocielnik, Saleema Amershi, Paul Bennett
  
  CHI 2019 | May 2019
  
  Publication Project
- Guidelines for Human-AI Interaction
  
  Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz
  
  CHI 2019 | May 2019
  
  CHI 2019 Honorable Mention Award
  
  PDF Video Project Project
- Machine Teaching: A New Paradigm for Building Machine Learning Systems
  
  Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
  
  MSR-TR-2017-26 | July 2017
  
  Preprint Project
- Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
  
  Joseph Chee Chang, Saleema Amershi, Ece Kamar
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2017) | May 2017
  
  PDF
- A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs
  
  Kristina Toutanova, Chris Brockett, Ke M. Tran, Saleema Amershi
  
  EMNLP | November 2016
  
  PDF PDF PDF Publication Access Project
- The Label Complexity of Mixed-Initiative Classiï¬er Training
  
  Jina Suh, Xiaojin Zhu, Saleema Amershi
  
  Proceedings of the 33rd International Conference on Machine Learning | June 2016
  
  PDF PDF PDF
- ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
  
  Saleema Amershi, Max Chickering, Steven Drucker, Bongshin Lee, Patrice Simard, Jina Suh
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2015) | April 2015
  
  PDF Publication Project Project
- Power to the People: The Role of Humans in Interactive Machine Learning
  
  Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza
  
  AI Magazine | December 2014
  
  PDF Publication
- ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
  
  Patrice Simard, Max Chickering, Apama Lakshmiratan, Denis Charles, Leon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, Jina Suh
  
  MSR-TR-2014-128 | September 2014
  
  PDF Project
- Structured Labeling for Facilitating Concept Evolution in Machine Learning
  
  Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2014) | May 2014
  
  Best Paper Award
  
  PDF PDF
- LiveAction: Automating Web Task Model Generation
  
  Saleema Amershi, Jalal Mahmud, Jeffrey Nichols, Tessa Lau, German Attanasio Ruiz
  
  ACM Trans. Interact. Intell. Syst | October 2013, pp. 23
  
  PDF Publication
- ReGroup: Interactive Machine Learning for On-Demand Group Creation in Social Networks
  
  Saleema Amershi, James Fogarty, Daniel Weld
  
  Proceeding CHI '12 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems | May 2012
  
  PDF Publication
- Designing for Effective End-User Interaction with Machine Learning
  
  Saleema Amershi
  
  October 2011
  
  October 2011
  
  PDF
- Effective End-User Interaction with Machine Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  AAAI (Nectar Track) | August 2011
  
  PDF
- Human-Guided Machine Learning for Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence | July 2011
  
  ISBN: 978-1-57735-515-1
  
  Invited from CHI 2011 to Best Papers Track
  
  PDF Publication Project
- CueT: Human-Guided Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  CHI | May 2011
  
  Best Paper Honorable Mention
  
  PDF Project
- Comparing Web Interaction Models in Developing Regions
  
  Jay Chen, Aditya Dhananjay, Saleema Amershi, Lakshmi Subramanian
  
  December 2010
  
  PDF
- Examining Multiple Potential Models in End-User Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  CHI 2010: Machine Learning and Web Interactions | April 2010
  
  PDF
- Multiple Mouse Text Entry for Single-Display Groupware
  
  Saleema Amershi, Meredith Ringel Morris, Neema Moraveji, Ravin Balakrishnan, Kentaro Toyama
  
  CSCW | February 2010
  
  Best Paper Honorable Mention
  
  PDF
- Overview-Based Example Selection in End-User Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  UIST '09 Proceedings of the 22nd annual ACM symposium on User interface software and technology | October 2009
  
  ISBN: 978-1-60558-745-5
  
  PDF Publication
- Overview-Based Examples Selection in Mixed-Initiative Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  UIST | October 2009
  
  PDF
- Combining Unsupervised and Supervised Classification to Build User Models for Exploratory Learning Environments
  
  Saleema Amershi, CRISTINA CONATI
  
  Journal of Educational Data Mining | September 2009
  
  PDF
- Designing for End-User Interactive Concept Learning in CueFlik
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  (2009) Neural Information Processing Systems (NIPS) Workshop on Analysis and Design of Algorithms for Interactive Machine Learning | August 2009
  
  PDF
- Amplifying Community Content Creation with Mixed-Initiative Information Extraction
  
  Raphael Hoffmann, Saleema Amershi, Kayur Patel, Fei Wu, James Fogarty, Daniel S. Weld
  
  In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2009) | April 2009
  
  Best Paper Honorable Mention
  
  PDF Publication
- CoSearch: A System for Co-located Collaborative Web Search
  
  Saleema Amershi, Meredith Ringel Morris
  
  Proceedings of CHI 2008 | April 2008
  
  ISBN: 978-1-60558-011-1
  
  PDF Video
- Using Eye-tracking Data for High-Level User Modeling in Adaptive Interfaces
  
  Cristina Conati, Christina Merten, Saleema Amershi, Kasia Muldner
  
  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 07) Nectar Track | July 2007
  
  PDF
- Unsupervised and Supervised Machine Learning in User Modeling for Intelligent Learning Environments
  
  Saleema Amershi, Cristina Conati
  
  In Proceedings of the ACM/SIGCHI Conference on Intelligent User Interfaces (IUI 2007) | January 2007
  
  PDF
- Automatic Recognition of Learner Types in Exploratory Learning Environments
  
  Saleema Amershi, Cristina Conati
  
  ITS'06 Proceedings of the 8th international conference on Intelligent Tutoring Systems | June 2006
  
  PDF Publication
- Using Feature Selection and Unsupervised Clustering to Identify Affective Expressions in Educational Games
  
  Saleema Amershi, Cristina Conati, Heather Maclaren
  
  In Proceedings of The Intelligent Tutoring Systems Workshop on Motivational and Affective Issues in ITS (ITS 2006) | January 2006
  
  PDF
- Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions
  
  John Chung, Ece Kamar, Saleema Amershi
  
  ACL 2023 | July 2023
  
  Publication
- Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
  
  Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi Iqbal, Jaime Teevan
  
  CHI 2019 | May 2019
  
  Publication Project
- Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | June 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | October 2022
  
  Preprint
- Software Engineering for Machine Learning: A Case Study
  
  Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLIne, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, Tom Zimmermann
  
  International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track | May 2019
  
  ICSE 2019 Best Paper Award
  
  PDF
- The Impact of More Transparent Interfaces on Behavior in Personalized Recommendation
  
  Tobias Schnabel, Saleema Amershi, Paul Bennett, Peter Bailey, Thorsten Joachims
  
  Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) | July 2020
  
  Publication
- Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems
  
  Rafal Kocielnik, Saleema Amershi, Paul Bennett
  
  CHI 2019 | May 2019
  
  Publication Project
- CoSearch: A System for Co-located Collaborative Web Search
  
  Saleema Amershi, Meredith Ringel Morris
  
  Proceedings of CHI 2008 | April 2008
  
  ISBN: 978-1-60558-011-1
  
  PDF Video
- Shared Sensemaking: Enhancing the Value of Collaborative Web Search Tools
  
  Meredith Ringel Morris, Saleema Amershi
  
  MSR-TR-2008-205 | April 2008
  
  PDF
- Pedagogy and usability in interactive algorithm visualizations: Designing and evaluating CIspace
  
  Saleema Amershi, Giuseppe Carenini, Cristina Conati, Alan K. Mackworth, David Poole
  
  Designing and Evaluating CIspace. Interacting with Computers - The Interdisciplinary Journal of Human-Computer Interaction | September 2007
  
  PDF
- Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
  
  Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, Jason Williams
  
  IEEE TVCG, VAST | August 2016
  
  PDF Video Project
- FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification
  
  Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, Patrice Simard
  
  IEEE VAST | October 2015
  
  PDF Project
- Human-Guided Machine Learning for Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence | July 2011
  
  ISBN: 978-1-57735-515-1
  
  Invited from CHI 2011 to Best Papers Track
  
  PDF Publication Project
- CueT: Human-Guided Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  CHI | May 2011
  
  Best Paper Honorable Mention
  
  PDF Project
- Examining Multiple Potential Models in End-User Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  CHI 2010: Machine Learning and Web Interactions | April 2010
  
  PDF
- Multiple Mouse Text Entry for Single-Display Groupware
  
  Saleema Amershi, Meredith Ringel Morris, Neema Moraveji, Ravin Balakrishnan, Kentaro Toyama
  
  CSCW | February 2010
  
  Best Paper Honorable Mention
  
  PDF

- SentinelBench: A Benchmark for Long-Running Monitoring Agents
  
  Matheus Kunzler Maldaner, Adam Fourney, Amanda Swearngin, Hussein Mozannar, Gagan Bansal, Maya Murad, Rafah Hosn, Saleema Amershi
  
  June 2026
  
  arXiv | June 2026
  
  Publication
- AHA! Facilitating AI Impact Assessment by Generating Examples of Harms
  
  Zana Buçinca, Chau Minh Pham, Maurice Jakesch, Marco Tulio Ribeiro, Alexandra Olteanu, Saleema Amershi
  
  June 2023
  
  arXiv | June 2023
  
  Publication
- The Collaboration Gap
  
  Tim R. Davidson, Adam Fourney, Saleema Amershi, Robert West, Eric Horvitz, Ece Kamar
  
  COLM 2026 | November 2025
  
  Publication
- Interactive Debugging and Steering of Multi-Agent AI Systems
  
  Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney (adamfo), Jack Gerrits, Erkang (Eric) Zhu, Saleema Amershi
  
  CHI 2025 | April 2025
  
  Publication
- Supporting Human-AI Collaboration in Auditing LLMs with LLMs
  
  Charvi Rastogi, Marco Tulio Ribeiro, Nicholas King, Harsha Nori, Saleema Amershi
  
  AIES | August 2023
  
  Publication
- Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions
  
  John Chung, Ece Kamar, Saleema Amershi
  
  ACL 2023 | July 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value for Code Generation Models
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | June 2023
  
  Publication
- Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers
  
  Victor Dibia, Adam Fourney, Gagan Bansal, Forough Poursabzi, Han Liu, Saleema Amershi
  
  ACL 2023 | October 2022
  
  Preprint
- How Different Groups Prioritize Ethical Values for Responsible AI
  
  Maurice Jakesch, Zana Buçinca, Saleema Amershi, Alexandra Olteanu
  
  FAccT 2022 | May 2022
  
  Publication DOI
- REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
  
  Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
  
  FAccT 2022 | May 2022
  
  DOI Project
- HINT: Integration Testing for AI-based Features with Humans in the Loop
  
  Quanze Chen, Tobias Schnabel, Besmira Nushi, Saleema Amershi
  
  International Conference on Intelligent User Interfaces | March 2022
  
  Publication Project
- Planning for Natural Language Failures with the AI Playbook
  
  Matthew K. Hong, Adam Fourney, Derek DeBellis, Saleema Amershi
  
  2021 CHI Conference on Human Factors in Computing Systems | May 2021
  
  Publication Video Project
- “Who doesn’t like dinosaurs?” Finding and Eliciting Richer Preferences for Recommendation
  
  Tobias Schnabel, Gonzalo Ramos, Saleema Amershi
  
  14th ACM Conference on Recommender Systems | September 2020
  
  PDF
- The Impact of More Transparent Interfaces on Behavior in Personalized Recommendation
  
  Tobias Schnabel, Saleema Amershi, Paul Bennett, Peter Bailey, Thorsten Joachims
  
  Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) | July 2020
  
  Publication
- Software Engineering for Machine Learning: A Case Study
  
  Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLIne, Harald Gall, Ece Kamar, Nachi Nagappan, Besmira Nushi, Tom Zimmermann
  
  International Conference on Software Engineering (ICSE 2019) - Software Engineering in Practice track | May 2019
  
  ICSE 2019 Best Paper Award
  
  PDF
- People and AI See Things Different Implications of Mismatched Perception on HCI for AI Systems
  
  Saleema Amershi, Ece Kamar, Emre Kiciman
  
  Workshop on Human-Centered Machine Learning Perspectives at CHI 2019 | May 2019
- Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
  
  Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi Iqbal, Jaime Teevan
  
  CHI 2019 | May 2019
  
  Publication Project
- Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems
  
  Rafal Kocielnik, Saleema Amershi, Paul Bennett
  
  CHI 2019 | May 2019
  
  Publication Project
- Guidelines for Human-AI Interaction
  
  Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz
  
  CHI 2019 | May 2019
  
  CHI 2019 Honorable Mention Award
  
  PDF Video Project Project
- Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets
  
  Joseph Chee Chang, Saleema Amershi, Ece Kamar
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2017) | May 2017
  
  PDF
- A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs
  
  Kristina Toutanova, Chris Brockett, Ke M. Tran, Saleema Amershi
  
  EMNLP | November 2016
  
  PDF PDF PDF Publication Access Project
- Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
  
  Donghao Ren, Saleema Amershi, Bongshin Lee, Jina Suh, Jason Williams
  
  IEEE TVCG, VAST | August 2016
  
  PDF Video Project
- The Label Complexity of Mixed-Initiative Classiï¬er Training
  
  Jina Suh, Xiaojin Zhu, Saleema Amershi
  
  Proceedings of the 33rd International Conference on Machine Learning | June 2016
  
  PDF PDF PDF
- Active Learning with Oracle Epiphany
  
  Tzu-Kuo Huang, Lihong Li, Ara Vartanian, Saleema Amershi, Xiaojin Zhu
  
  30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain | May 2016
  
  PDF
- FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification
  
  Michael Brooks, Saleema Amershi, Bongshin Lee, Steven Drucker, Ashish Kapoor, Patrice Simard
  
  IEEE VAST | October 2015
  
  PDF Project
- ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
  
  Saleema Amershi, Max Chickering, Steven Drucker, Bongshin Lee, Patrice Simard, Jina Suh
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2015) | April 2015
  
  PDF Publication Project Project
- Structured Labeling for Facilitating Concept Evolution in Machine Learning
  
  Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, Denis Charles
  
  Proceedings of the Conference on Human Factors in Computing Systems (CHI 2014) | May 2014
  
  Best Paper Award
  
  PDF PDF
- ReGroup: Interactive Machine Learning for On-Demand Group Creation in Social Networks
  
  Saleema Amershi, James Fogarty, Daniel Weld
  
  Proceeding CHI '12 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems | May 2012
  
  PDF Publication
- Effective End-User Interaction with Machine Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  AAAI (Nectar Track) | August 2011
  
  PDF
- Human-Guided Machine Learning for Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  IJCAI'11 Proceedings of the Twenty-Second international joint conference on Artificial Intelligence | July 2011
  
  ISBN: 978-1-57735-515-1
  
  Invited from CHI 2011 to Best Papers Track
  
  PDF Publication Project
- CueT: Human-Guided Fast and Accurate Network Alarm Triage
  
  Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, Blaine Christian
  
  CHI | May 2011
  
  Best Paper Honorable Mention
  
  PDF Project
- Comparing Web Interaction Models in Developing Regions
  
  Jay Chen, Aditya Dhananjay, Saleema Amershi, Lakshmi Subramanian
  
  December 2010
  
  PDF
- Examining Multiple Potential Models in End-User Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  CHI 2010: Machine Learning and Web Interactions | April 2010
  
  PDF
- Multiple Mouse Text Entry for Single-Display Groupware
  
  Saleema Amershi, Meredith Ringel Morris, Neema Moraveji, Ravin Balakrishnan, Kentaro Toyama
  
  CSCW | February 2010
  
  Best Paper Honorable Mention
  
  PDF
- Overview-Based Example Selection in End-User Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  UIST '09 Proceedings of the 22nd annual ACM symposium on User interface software and technology | October 2009
  
  ISBN: 978-1-60558-745-5
  
  PDF Publication
- Overview-Based Examples Selection in Mixed-Initiative Interactive Concept Learning
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  UIST | October 2009
  
  PDF
- Designing for End-User Interactive Concept Learning in CueFlik
  
  Saleema Amershi, James Fogarty, Ashish Kapoor, Desney Tan
  
  (2009) Neural Information Processing Systems (NIPS) Workshop on Analysis and Design of Algorithms for Interactive Machine Learning | August 2009
  
  PDF
- Amplifying Community Content Creation with Mixed-Initiative Information Extraction
  
  Raphael Hoffmann, Saleema Amershi, Kayur Patel, Fei Wu, James Fogarty, Daniel S. Weld
  
  In Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2009) | April 2009
  
  Best Paper Honorable Mention
  
  PDF Publication
- CoSearch: A System for Co-located Collaborative Web Search
  
  Saleema Amershi, Meredith Ringel Morris
  
  Proceedings of CHI 2008 | April 2008
  
  ISBN: 978-1-60558-011-1
  
  PDF Video
- Intelligence in Wikipedia
  
  Daniel S. Weld, Fei Wu, Eytan Adar, Saleema Amershi, James Fogarty, Raphael Hoffmann, Kayur Patel, Michael Skinner
  
  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 08) Senior Papers Track | January 2008
  
  PDF
- Using Eye-tracking Data for High-Level User Modeling in Adaptive Interfaces
  
  Cristina Conati, Christina Merten, Saleema Amershi, Kasia Muldner
  
  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 07) Nectar Track | July 2007
  
  PDF
- Unsupervised and Supervised Machine Learning in User Modeling for Intelligent Learning Environments
  
  Saleema Amershi, Cristina Conati
  
  In Proceedings of the ACM/SIGCHI Conference on Intelligent User Interfaces (IUI 2007) | January 2007
  
  PDF
- Automatic Recognition of Learner Types in Exploratory Learning Environments
  
  Saleema Amershi, Cristina Conati
  
  ITS'06 Proceedings of the 8th international conference on Intelligent Tutoring Systems | June 2006
  
  PDF Publication
- Using Feature Selection and Unsupervised Clustering to Identify Affective Expressions in Educational Games
  
  Saleema Amershi, Cristina Conati, Heather Maclaren
  
  In Proceedings of The Intelligent Tutoring Systems Workshop on Motivational and Affective Issues in ITS (ITS 2006) | January 2006
  
  PDF
- Designing CIspace: Pedagogy and Usability in a Learning Environment for AI
  
  Saleema Amershi, N. Arksey, G. Carenini, C. Conati, A. Mackworth, H. Maclaren, D. Poole
  
  In Proceedings of the ACM/SIGCSE Conference on Innovation and Technology in Computer Science Education ITiCSE '05 | June 2005
  
  PDF
- Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets
  
  Gagan Bansal, Wenyue Hua, Zezhou Huang, Adam Fourney, Amanda Swearngin, Will Epperson, Tyler Payne, Jake Hofman, Brendan Lucier, Chinmay Singh, Markus Mobius, Akshay Nambi, Archana Yadav, Kevin Gao, David Rothschild, Aleksandrs Slivkins, Daniel G. Goldstein, Hussein Mozannar, Nicole Immorlica, Maya Murad, Matthew Vogel, Subbarao Kambhampati, Eric Horvitz, Saleema Amershi
  
  MSR-TR-2025-50 | October 2025
  
  Published by Microsoft
  
  Publication Video Video Access
- Magentic-UI: Towards Human-in-the-loop Agentic Systems
  
  Hussein Mozannar, Gagan Bansal, Cheng Tan, Adam Fourney, Victor Dibia, Jingya Chen, Jack Gerrits, Tyler Payne, Matheus Kunzler Maldaner, Madeleine Grunde-McLaughlin, Eric Zhu, Griffin Bassman, Jacob Alber, Peter Chang, Ricky Loynd, Friederike Niedtner, Ece Kamar, Maya Murad, Rafah Hosn, Saleema Amershi
  
  MSR-TR-2025-40 | July 2025
  
  Published by Microsoft
  
  Publication Project
- Challenges in Human-Agent Communication
  
  Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
  
  MSR-TR-2024-53 | December 2024
  
  Published by Microsoft
  
  Publication Project
- Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
  
  Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang (Eric) Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, Saleema Amershi
  
  MSR-TR-2024-47 | November 2024
  
  Published by Microsoft
  
  Preprint Video Access
- Machine Teaching: A New Paradigm for Building Machine Learning Systems
  
  Patrice Simard, Saleema Amershi, Max Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Chris Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, John Wernsing
  
  MSR-TR-2017-26 | July 2017
  
  Preprint Project
- ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
  
  Patrice Simard, Max Chickering, Apama Lakshmiratan, Denis Charles, Leon Bottou, Carlos Garcia Jurado Suarez, David Grangier, Saleema Amershi, Johan Verwey, Jina Suh
  
  MSR-TR-2014-128 | September 2014
  
  PDF Project
- Shared Sensemaking: Enhancing the Value of Collaborative Web Search Tools
  
  Meredith Ringel Morris, Saleema Amershi
  
  MSR-TR-2008-205 | April 2008
  
  PDF
- AutoGen Studio: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems
  
  Victor Dibia, Jingya Chen, Gagan Bansal, Suff Syed, Adam Fourney, Erkang (Eric) Zhu, Chi Wang, Saleema Amershi
  
  August 2024
  
  Preprint
  
  PDF Publication Video Video Project
- Assessing Human-AI Interaction Early through Factorial Surveys: A Study on the Guidelines for Human-AI Interaction
  
  Tianyi Li, Mihaela Vorvoreanu, Derek DeBellis, Saleema Amershi
  
  ACM Transactions on Computer-Human Interaction | April 2022
  
  Publication Project Project
- Power to the People: The Role of Humans in Interactive Machine Learning
  
  Saleema Amershi, Maya Cakmak, W. Bradley Knox, Todd Kulesza
  
  AI Magazine | December 2014
  
  PDF Publication
- LiveAction: Automating Web Task Model Generation
  
  Saleema Amershi, Jalal Mahmud, Jeffrey Nichols, Tessa Lau, German Attanasio Ruiz
  
  ACM Trans. Interact. Intell. Syst | October 2013, pp. 23
  
  PDF Publication
- Combining Unsupervised and Supervised Classification to Build User Models for Exploratory Learning Environments
  
  Saleema Amershi, CRISTINA CONATI
  
  Journal of Educational Data Mining | September 2009
  
  PDF
- Pedagogy and usability in interactive algorithm visualizations: Designing and evaluating CIspace
  
  Saleema Amershi, Giuseppe Carenini, Cristina Conati, Alan K. Mackworth, David Poole
  
  Designing and Evaluating CIspace. Interacting with Computers - The Interdisciplinary Journal of Human-Computer Interaction | September 2007
  
  PDF
- Designing for Effective End-User Interaction with Machine Learning
  
  Saleema Amershi
  
  October 2011
  
  October 2011
  
  PDF

Saleema Amershi

Publications

SentinelBench: A Benchmark for Long-Running Monitoring Agents

The Collaboration Gap

Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets

Magentic-UI: Towards Human-in-the-loop Agentic Systems

Interactive Debugging and Steering of Multi-Agent AI Systems

Challenges in Human-Agent Communication

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

AutoGen Studio: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems

Supporting Human-AI Collaboration in Auditing LLMs with LLMs

Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

Aligning Offline Metrics and Human Judgments of Value for Code Generation Models

AHA! Facilitating AI Impact Assessment by Generating Examples of Harms

Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers

How Different Groups Prioritize Ethical Values for Responsible AI

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Assessing Human-AI Interaction Early through Factorial Surveys: A Study on the Guidelines for Human-AI Interaction

HINT: Integration Testing for AI-based Features with Humans in the Loop

Planning for Natural Language Failures with the AI Playbook

“Who doesn’t like dinosaurs?” Finding and Eliciting Richer Preferences for Recommendation

The Impact of More Transparent Interfaces on Behavior in Personalized Recommendation

Software Engineering for Machine Learning: A Case Study

People and AI See Things Different Implications of Mismatched Perception on HCI for AI Systems

Guidelines for Human-AI Interaction

Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence

Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems

Machine Teaching: A New Paradigm for Building Machine Learning Systems

Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets

A Dataset and Evaluation Metrics for Abstractive Compression of Sentences and Short Paragraphs

Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers

The Label Complexity of Mixed-Initiative Classiï¬er Training

Active Learning with Oracle Epiphany

FeatureInsight: Visual Support for Error-Driven Feature Ideation in Text Classification

ModelTracker: Redesigning Performance Analysis Tools for Machine Learning

Power to the People: The Role of Humans in Interactive Machine Learning

ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems

Structured Labeling for Facilitating Concept Evolution in Machine Learning

LiveAction: Automating Web Task Model Generation

ReGroup: Interactive Machine Learning for On-Demand Group Creation in Social Networks

Designing for Effective End-User Interaction with Machine Learning

Effective End-User Interaction with Machine Learning

Human-Guided Machine Learning for Fast and Accurate Network Alarm Triage

CueT: Human-Guided Fast and Accurate Network Alarm Triage

Comparing Web Interaction Models in Developing Regions

Examining Multiple Potential Models in End-User Interactive Concept Learning

Multiple Mouse Text Entry for Single-Display Groupware

Overview-Based Example Selection in End-User Interactive Concept Learning

Overview-Based Examples Selection in Mixed-Initiative Interactive Concept Learning

Combining Unsupervised and Supervised Classification to Build User Models for Exploratory Learning Environments

Designing for End-User Interactive Concept Learning in CueFlik

Amplifying Community Content Creation with Mixed-Initiative Information Extraction

CoSearch: A System for Co-located Collaborative Web Search

Shared Sensemaking: Enhancing the Value of Collaborative Web Search Tools

Intelligence in Wikipedia

Pedagogy and usability in interactive algorithm visualizations: Designing and evaluating CIspace

Using Eye-tracking Data for High-Level User Modeling in Adaptive Interfaces

Unsupervised and Supervised Machine Learning in User Modeling for Intelligent Learning Environments

Automatic Recognition of Learner Types in Exploratory Learning Environments

Using Feature Selection and Unsupervised Clustering to Identify Affective Expressions in Educational Games

Designing CIspace: Pedagogy and Usability in a Learning Environment for AI

SentinelBench: A Benchmark for Long-Running Monitoring Agents

The Collaboration Gap

Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets

Magentic-UI: Towards Human-in-the-loop Agentic Systems

Interactive Debugging and Steering of Multi-Agent AI Systems

Challenges in Human-Agent Communication

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

AutoGen Studio: A No-Code Developer Tool for Building and Debugging Multi-Agent Systems

Supporting Human-AI Collaboration in Auditing LLMs with LLMs

Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions

Aligning Offline Metrics and Human Judgments of Value for Code Generation Models

AHA! Facilitating AI Impact Assessment by Generating Examples of Harms

Aligning Offline Metrics and Human Judgments of Value of AI-Pair Programmers

How Different Groups Prioritize Ethical Values for Responsible AI

REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research

Assessing Human-AI Interaction Early through Factorial Surveys: A Study on the Guidelines for Human-AI Interaction

HINT: Integration Testing for AI-based Features with Humans in the Loop

Planning for Natural Language Failures with the AI Playbook

“Who doesn’t like dinosaurs?” Finding and Eliciting Richer Preferences for Recommendation

The Label Complexity of Mixed-Initiative Classiï¬er Training

The Label Complexity of Mixed-Initiative Classiï¬er Training

The Label Complexity of Mixed-Initiative Classiï¬er Training