<\/p>\n\n\n\n
Large language and multimodal models have been largely employed to reactively generate content and assistance. In contrast, we explore how models can be employed to provide proactive assistance that fluidly supports ongoing human activities. We identify and tackle challenges in such streaming scenarios\u2014where models must act continuously and responsively in real time, rather than waiting for explicit user requests. We explore solutions and evaluation metrics in the context of a mixed-reality AI agent designed to assist people in carrying out procedural physical tasks. Specifically, we focus on the challenge of identifying when each step of a procedure has been completed so that an agent can appropriately move on to the next step. Using a publicly available multimodal dataset of task-centric human-agent interactions, we study an array of prompting strategies across several frontier LLMs and VLMs, considering both the content and timing of model queries. Our findings demonstrate the criticality of developing application-driven metrics that enable meaningful comparisons and highlight key gaps in current model capabilities for proactive assistance. This work underscores the importance of temporally-aware evaluation in designing fluid real-time human-AI collaboration.<\/p>\n","protected":false},"excerpt":{"rendered":"
Large language and multimodal models have been largely employed to reactively generate content and assistance. In contrast, we explore how models can be employed to provide proactive assistance that fluidly supports ongoing human activities. We identify and tackle challenges in such streaming scenarios\u2014where models must act continuously and responsively in real time, rather than waiting […]<\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[{"type":"user_nicename","value":"Sean Andrist","user_id":"36443"},{"type":"user_nicename","value":"Dan Bohus","user_id":"31581"},{"type":"user_nicename","value":"Maia Stiber","user_id":"44072"},{"type":"text","value":"Yuwei Bao","user_id":0},{"type":"text","value":"Tim J. Schoonbeek","user_id":0},{"type":"user_nicename","value":"Eric Horvitz","user_id":"32033"}],"msr_publishername":"","msr_publisher_other":"","msr_booktitle":"Proceedings of the 28th International Conference On Multimodal Interaction","msr_chapter":"","msr_edition":"Proceedings of the 28th International Conference On Multimodal Interaction","msr_editors":"","msr_how_published":"","msr_isbn":"","msr_issue":"","msr_journal":"Proceedings of the 28th International Conference On Multimodal Interaction","msr_number":"","msr_organization":"","msr_pages_string":"","msr_page_range_start":"","msr_page_range_end":"","msr_series":"","msr_volume":"","msr_copyright":"","msr_conference_name":"Proceedings of the 28th International Conference On Multimodal Interaction","msr_doi":"10.1145\/3776574.3831228","msr_arxiv_id":"","msr_mag_id":"","msr_other_authors":"","msr_other_contributors":"","msr_speaker":"","msr_award":"","msr_affiliation":"","msr_institution":"","msr_host":"","msr_version":"","msr_duration":"","msr_release_tracker_id":"","msr_highlight_type":"","msr_date_display_format":"","msr_main_download_label":"","msr_external_link_label":"","msr_doi_label":"","msr_published_date":"2026-10-04","msr_startdate":"","msr_presentation_date":"","msr_highlight_text":"","msr_notes":"","msr_longbiography":"","msr_publicationurl":"","msr_external_url":"","msr_secondary_video_url":"","msr_conference_url":"","msr_journal_url":"","msr_year":2026,"msr_month":10,"msr_day":4,"msr_microsoftintellectualproperty":false,"msr_pub_id":"1c6d5c2d75e3486206b5dc3ac3186870a64bb64e","msr_publication_uploader":[],"msr_related_uploader":[{"type":"url","title":"https:\/\/dl.acm.org\/doi\/full\/10.1145\/3776574.3831228","label_id":270375,"id":false,"viewUrl":false}],"msr_original_fields_of_study":[],"msr_s2_paper_id":"1c6d5c2d75e3486206b5dc3ac3186870a64bb64e","msr_s2_pdf_url":"https:\/\/doi.org\/10.1145\/3776574.3831228","msr_citation_count_updated":"","msr_citation_count":0,"msr_influential_citations":0,"msr_reference_count":42,"msr_s2_open_access":true,"msr_s2_author_ids":[],"msr_pub_ids":[{"provider":"s2","id":"1c6d5c2d75e3486206b5dc3ac3186870a64bb64e"},{"provider":"doi","id":"10.1145\/3776574.3831228"},{"provider":"corpusid","id":"291320274"}],"msr_hide_image_in_river":0,"footnotes":""},"msr-research-highlight":[],"research-area":[13556,13554],"msr-publication-type":[193719,193716],"msr-publisher":[],"msr-publication-cta":[],"msr-focus-area":[],"msr-locale":[268875],"msr-post-option":[],"msr-field-of-study":[255217,254467],"msr-conference":[270703],"msr-journal":[270702],"msr-impact-theme":[],"msr-pillar":[],"class_list":["post-1191124","msr-research-item","type-msr-research-item","status-publish","hentry","msr-research-area-artificial-intelligence","msr-research-area-human-computer-interaction","msr-locale-en_us","msr-field-of-study-human-computer-interaction-4","msr-field-of-study-mixed-reality"],"msr_publishername":"","msr_edition":"Proceedings of the 28th International Conference On Multimodal Interaction","msr_affiliation":"","msr_published_date":"2026-10-04","msr_host":"","msr_duration":"","msr_version":"","msr_speaker":"","msr_other_contributors":"","msr_booktitle":"Proceedings of the 28th International Conference On Multimodal Interaction","msr_pages_string":"","msr_chapter":"","msr_isbn":"","msr_journal":"Proceedings of the 28th International Conference On Multimodal Interaction","msr_volume":"","msr_number":"","msr_editors":"","msr_series":"","msr_issue":"","msr_organization":"","msr_how_published":"","msr_notes":"","msr_highlight_text":"","msr_release_tracker_id":"","msr_original_fields_of_study":"","msr_download_urls":"","msr_external_url":"","msr_secondary_video_url":"","msr_longbiography":"","msr_microsoftintellectualproperty":0,"msr_main_download":"","msr_publicationurl":"","msr_doi":"10.1145\/3776574.3831228","msr_publication_uploader":[],"msr_related_uploader":[{"type":"url","title":"https:\/\/dl.acm.org\/doi\/full\/10.1145\/3776574.3831228","label_id":270375,"id":false,"viewUrl":false}],"msr_citation_count":0,"msr_citation_count_updated":"","msr_s2_paper_id":"1c6d5c2d75e3486206b5dc3ac3186870a64bb64e","msr_influential_citations":0,"msr_reference_count":42,"msr_arxiv_id":"","msr_s2_author_ids":[],"msr_s2_open_access":true,"msr_s2_pdf_url":"https:\/\/doi.org\/10.1145\/3776574.3831228","msr_attachments":[],"msr-author-ordering":[{"type":"user_nicename","value":"Sean Andrist","user_id":36443,"rest_url":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Sean Andrist"},{"type":"user_nicename","value":"Dan Bohus","user_id":31581,"rest_url":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Dan Bohus"},{"type":"user_nicename","value":"Maia Stiber","user_id":44072,"rest_url":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Maia Stiber"},{"type":"text","value":"Yuwei Bao","user_id":0,"rest_url":false},{"type":"text","value":"Tim J. Schoonbeek","user_id":0,"rest_url":false},{"type":"user_nicename","value":"Eric Horvitz","user_id":32033,"rest_url":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/microsoft-research\/v1\/researchers?person=Eric Horvitz"}],"msr_impact_theme":[],"msr_research_lab":[],"msr_event":[],"msr_group":[],"msr_project":[],"publication":[],"video":[],"msr-tool":[],"msr_publication_type":"book","related_content":[],"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1191124","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-research-item"}],"version-history":[{"count":3,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1191124\/revisions"}],"predecessor-version":[{"id":1191986,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-research-item\/1191124\/revisions\/1191986"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1191124"}],"wp:term":[{"taxonomy":"msr-research-highlight","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-research-highlight?post=1191124"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1191124"},{"taxonomy":"msr-publication-type","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-type?post=1191124"},{"taxonomy":"msr-publisher","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-publisher?post=1191124"},{"taxonomy":"msr-publication-cta","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-publication-cta?post=1191124"},{"taxonomy":"msr-focus-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-focus-area?post=1191124"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1191124"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1191124"},{"taxonomy":"msr-field-of-study","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-field-of-study?post=1191124"},{"taxonomy":"msr-conference","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-conference?post=1191124"},{"taxonomy":"msr-journal","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-journal?post=1191124"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=1191124"},{"taxonomy":"msr-pillar","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-pillar?post=1191124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}