{"id":1162754,"date":"2026-02-23T22:18:02","date_gmt":"2026-02-24T06:18:02","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?post_type=msr-research-item&p=1162754"},"modified":"2026-04-02T12:59:42","modified_gmt":"2026-04-02T19:59:42","slug":"interaction-augmented-instruction-modeling-the-synergy-of-prompts-and-interactions-in-human-genai-collaboration","status":"publish","type":"msr-research-item","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/publication\/interaction-augmented-instruction-modeling-the-synergy-of-prompts-and-interactions-in-human-genai-collaboration\/","title":{"rendered":"Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration"},"content":{"rendered":"

\"A<\/p>\n

Text prompt is the most common way for human-generative AI (GenAI) communication. Though convenient, it is challenging to convey fine-grained and referential intent. One promising solution is to combine text prompts with precise GUI interactions, like brushing and clicking. However, there lacks a formal model to capture synergistic designs between prompts and interactions, hindering their comparison and innovation. To fill this gap, via an iterative and deductive process, we develop the Interaction-Augmented Instruction (IAI) model, a compact entity\u2013relation graph formalizing how the combination of interactions and text prompts enhances human-GenAI communication. With the model, we distill twelve recurring and composable atomic interaction paradigms from prior tools, verifying our model\u2019s capability to facilitate systematic design characterization and comparison. Four usage scenarios further demonstrate the model\u2019s utility in applying, refining, and innovating these paradigms. These results illustrate the IAI model\u2019s descriptive, discriminative, and generative power for shaping future GenAI systems.<\/p>\n