{"id":1190869,"date":"2026-09-05T09:00:00","date_gmt":"2026-09-05T16:00:00","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/?post_type=msr-blog-post&p=1190869"},"modified":"2026-10-28T00:36:05","modified_gmt":"2026-10-28T07:36:05","slug":"beyond-the-chat-window-huabu","status":"publish","type":"msr-blog-post","link":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/articles\/beyond-the-chat-window-huabu\/","title":{"rendered":"Beyond the chat window: Huabu reimagines the desktop for the agent era\u00a0"},"content":{"rendered":"\n

Today, the chat window is still the main way people interact with large models and agents: users ask a question, the AI answers, and each follow-up extends the thread. This is simple and natural, but fundamentally linear. When a question opens several directions worth exploring, people can usually pursue only one, and the rest is buried in an ever-longer chat history. Human thinking works differently: people jump between ideas and put related materials side by side to compare them. They set a question aside and come back to it later. Sometimes simply placing two pieces of information next to each other reveals a hidden connection. <\/p>\n\n\n\n

Huabu, recently introduced by Microsoft Research Asia, aims to change this. Instead of stacking the web pages, images, notes, conversations, sketches, and final plans from an agent\u2019s research in a chat thread, it lays them out in an unbounded two-dimensional space. Information can be placed side by side, moved, resized, grouped, and linked, and users can edit content directly and ask the agent to keep working on it at any time. <\/p>\n\n\n\n

Beyond a new interface, Huabu attempts to answer a fundamental question: as AI gets better at handling complex information, how should people think together with it?<\/p>\n\n\n\n

Built for the way people naturally think<\/h2>\n\n\n\n

Unlike an online whiteboard, which mainly offers drawing and layout, Huabu instead provides project workspaces, called Spaces, designed for people and AI to work together. A Space connects the whole process in one environment, from gathering information and organizing ideas to having AI carry out tasks and capturing the results. <\/p>\n\n\n\n

In Huabu, the basic unit of work is a project Space rather than a round of chat, so thinking no longer has to follow a single conversational path. Users can create a Space for each project and gather everything related to it in one place: PDFs, web pages, images, notes, chat histories, sketches, and agent-generated content. Materials can sit side by side for comparison, and each idea can become a node to expand and refine at any time. Content can be linked, and items on the same topic can be grouped, so scattered information grows into a project Space that can be explored over time. <\/p>\n\n\n\n

Work in Huabu moves through three stages: Collect, Organize, and Act. But this is not a fixed, one-way process. \u201cThe three stages form a cycle,\u201d said Luna Qiu, senior program manager at Microsoft Research Asia \u2013 Shanghai. \u201cNew results from the agent flow back into Huabu and become material for the next round of thinking and work. People can reorganize based on the new information and then hand the agent new tasks.\u201d<\/p>\n\n\n\n

Collect: Freeing users from playing information courier<\/strong><\/p>\n\n\n\n

In a conventional chat, users act as \u201ccontext couriers.\u201d Web pages, local documents, past conversations, and team discussions sit in separate windows, and getting AI to draw on them means copying and pasting them in piece by piece, a tedious process in which information is easily missed.  <\/p>\n\n\n\n

In Huabu, any material related to a task can be uploaded directly and automatically becomes its own node. More importantly, ideas set aside for later, unexplored lines of thought, and past discussions all stay in the Space as context the agent can read and use. <\/p>\n\n\n\n

Organize: Helping AI understand how information connects <\/strong><\/p>\n\n\n\n

Organizing is also a joint effort between people and AI. Users can arrange material themselves or leave the job to the agent. The agent places the material it retrieves on Huabu as separate nodes, each with an automatically generated summary. Users can then zoom in on anything they want to look at more closely, keep what matters, and remove what does not, avoiding the cognitive overload of a linear chat that delivers a large block of text all at once. <\/p>\n\n\n\n

Given dozens of papers, for example, the agent can arrange them by research timeline and academic lineage, draw lines to mark citations and opposing views, and produce a visual map of the research.  Given access to a code repository, it can break down the system architecture, with nodes for modules and lines for the dependencies between them, showing the full technical structure at a glance. For a proposal, it can sort material into core ideas, alternatives, and questions still to be verified, place related items together, and frame each group as a block of ideas. <\/p>\n\n\n\n

\"diagram\"
Figure 1. The agent organizes research papers into a visual map, showing research lineages and connections across topics.<\/figcaption><\/figure>\n\n\n\n

\u201cHuabu doesn\u2019t just pile up material,\u201d Qiu said. \u201cThrough semantic understanding, it organizes scattered information automatically and turns the implicit logical connections in people\u2019s minds into a visible spatial structure. At the same time, AI can recognize the relationships between ideas that people express through position, grouping, and links.\u201d <\/p>\n\n\n\n

Zooming lets users move between the big picture and the details: zoomed out, nodes show only the AI\u2019s summaries; zoomed in, they show the full text, high-resolution images, and handwritten annotations. In internal testing by the team of Nan Chen, a research software development engineer at Microsoft Research Asia \u2013 Shanghai, zooming greatly reduced the chance of losing track of ideas during multi-branch exploration. \u201cOverview and detail are no longer an either-or choice,\u201d Chen said. \u201cZooming solves the problem of large projects becoming cluttered and hard to manage.\u201d <\/p>\n\n\n\n

\"graphical
Figure 2. Zooming in reveals more detail while preserving the overall spatial context.<\/figcaption><\/figure>\n\n\n\n

Act: Letting agents step out of the chat box and into a shared space<\/strong><\/p>\n\n\n\n

Once information is collected and organized, users can call on local or cloud agents at any time. Their output flows back onto the shared canvas instead of staying in a chat box, becoming new material for the next round of thinking and agent tasks. <\/p>\n\n\n\n

Revising an AI-generated image or slide deck, for example, used to take long, complicated prompts, and even a slight misstep could mean losing the original style. In Huabu, users can express intent through spatial relationships. Telling the agent, \u201cMake this image match the style of the one next to it,\u201d is enough to align the styles, and a simple instruction is all it takes for the agent to adjust spacing while leaving everything else untouched. <\/p>\n\n\n\n

\u201cHuabu changes more than how people interact with agents,\u201d Qiu said. \u201cIt lets people tell AI what they want in a way that comes naturally.\u201d <\/p>\n\n\n\n

\"diagram\"
Figure 3. The user iteratively develops a slide with the agent while intermediate versions and relevant content remain available as context in the workspace.<\/figcaption><\/figure>\n\n\n\n

Huabu understands spatial instructions while keeping context continuous, and it saves each round of edits as a version. Users need not re-explain earlier steps, and the agent keeps working from what is on the canvas. This what-you-see-is-what-you-get interaction turns creation into a continuous loop of iteration and cuts the repeated trial and error that unpredictable results can cause. <\/p>\n\n\n\n

The logic behind Huabu: How agents understand spatial thinking <\/h2>\n\n\n\n

The three-stage workflow of Collect, Organize, and Act is more than a set of features. It rests on a new understanding of how people think and is built around three core principles: Externalized Thinking, Shared Cognitive Space, and Natural Interaction.<\/p>\n\n\n\n

Externalized Thinking: Layout itself is information <\/strong><\/p>\n\n\n\n

People naturally use space to organize information: they enlarge what matters, keep related material close, put distance between opposing views, draw lines to show logical relationships, and give scattered guesses an area of their own. These moves convey intent, but conventional AI tools cannot recognize them. <\/p>\n\n\n\n

Huabu designs its data storage and the agent\u2019s tool calls so that the agent reads not just a node\u2019s content but also its position, size, and connections. \u201cIn Huabu, position is information, size is information, and what something sits next to is information too,\u201d Chen said. \u201cEnlarge a passage of text, and the agent knows it matters to the user. Connect two nodes, and it knows they are closely related. That is far more intuitive than repeating \u2018this part is important\u2019 in a prompt.\u201d \u202fWith thinking externalized this way, people no longer need long prompts to \u201ctranslate\u201d their intent for AI; spatial relationships express it directly. <\/p>\n\n\n\n

\"diagram\"
Figure 4. Spatial layout and connections make relationships among materials explicit, allowing related information to be organized into a visible structure.<\/figcaption><\/figure>\n\n\n\n

Shared Cognitive Space: From text context to spatial context <\/strong><\/p>\n\n\n\n

In a chat window, context is chronological text, and focusing the AI on specific material means copying it in and spelling out its scope. Huabu introduces a spatial context mechanism instead, in which the space itself organizes context and points to what is meant. When an agent is placed in a particular area of Huabu, it reads the nodes around it and follows their links to related material, so users do not have to set boundaries for what it should reference. For example, with a hand-drawn sketch placed nearby, the agent interprets the task from the surrounding text, images, and other nodes. <\/p>\n\n\n\n

People and agents thus read the same \u201cthinking map,\u201d cutting the cost of passing and explaining information back and forth. That is the core of a Shared Cognitive Space: a unified workspace where people and AI share a view of the whole project and keep collaborating. <\/p>\n\n\n\n

Natural Interaction: Many ways to give input, with people in control <\/strong><\/p>\n\n\n\n

Beyond typing, Huabu supports touchpads, freehand drawing on graphics tablets, and voice to suit different stages of thinking. It will improve handwriting recognition so that agents can turn rough drafts annotations scribbled on the fly into structured content. <\/p>\n\n\n\n

Huabu also follows a human-in-the-loop principle: every change the AI makes to the Huabu is recorded and shown. \u201cThe agent doesn\u2019t force users to review every change, but a complete record is necessary,\u201d Qiu explained. \u201cIf the AI\u2019s edits were hidden in the background, users would lose their sense of control over the project. Showing every change builds trust in human-AI collaboration.\u201d <\/p>\n\n\n\n

This reflects another dimension of how Huabu understands natural interaction: natural collaboration does not mean handing everything to AI, but keeping its actions transparent and adjustable at any time. AI\u2019s job is to push the limits of what people can do, while people always have the final say on decisions and changes. In Qiu\u2019s words, \u201cTruly natural interaction means AI\u2019s actions always serve human thinking, not the other way around.\u201d <\/p>\n\n\n\n

Shared Cognitive Space: From text context to spatial context <\/strong><\/p>\n\n\n\n

In a chat window, context is chronological text, and focusing the AI on specific material means copying it in and spelling out its scope. Huabu introduces a spatial context mechanism instead, in which the space itself organizes context and points to what is meant. When an agent is placed in a particular area of Huabu, it reads the nodes around it and follows their links to related material, so users do not have to set boundaries for what it should reference. For example, with a hand-drawn sketch placed nearby, the agent interprets the task from the surrounding text, images, and other nodes. <\/p>\n\n\n\n

People and agents thus read the same \u201cthinking map,\u201d cutting the cost of passing and explaining information back and forth. That is the core of a Shared Cognitive Space: a unified workspace where people and AI share a view of the whole project and keep collaborating.<\/p>\n\n\n\n

Natural Interaction: Many ways to give input, with people in control <\/strong><\/p>\n\n\n\n

Beyond typing, Huabu supports touchpads, freehand drawing on graphics tablets, and voice to suit different stages of thinking. It will improve handwriting recognition so that agents can turn rough drafts annotations scribbled on the fly into structured content. <\/p>\n\n\n\n

Huabu also follows a human-in-the-loop principle: every change the AI makes to the Huabu is recorded and shown. \u201cThe agent doesn\u2019t force users to review every change, but a complete record is necessary,\u201d Qiu explained. \u201cIf the AI\u2019s edits were hidden in the background, users would lose their sense of control over the project. Showing every change builds trust in human-AI collaboration.\u201d <\/p>\n\n\n\n

This reflects another dimension of how Huabu understands natural interaction: natural collaboration does not mean handing everything to AI, but keeping its actions transparent and adjustable at any time. AI\u2019s job is to push the limits of what people can do, while people always have the final say on decisions and changes. In Qiu\u2019s words, \u201cTruly natural interaction means AI\u2019s actions always serve human thinking, not the other way around.\u201d<\/p>\n\n\n\n

From \u201cmy agent\u201d to \u201cour Space\u201d: The next desktop for the agent era <\/h2>\n\n\n\n

Huabu already supports one person working with multiple agents: the agents share a Space, each handles its own part of the work, and all their output flows back into Huabu. The researchers see this as only a starting point, with much more room for Huabu to grow.  <\/p>\n\n\n\n

In the workspace they envision, \u201cme\u201d and \u201cmy agent\u201d will be joined by colleagues and their agents. Several people will be able to work in the same Space at once, each bringing their own agent. People will talk in real time while their agents work together, and a higher-level agent can coordinate roles and assign tasks, creating a more natural network of collaboration among people, between people and agents, and among agents.  <\/p>\n\n\n\n

Huabu is one attempt at a new human-AI interface for the agent era, and there is no standard answer yet. Today\u2019s AI chat is not the final form, the researchers note, just as the command line was not the final form of early computers. Huabu is open source so that users, developers, and researchers can explore the question together.  <\/p>\n\n\n\n

As Huabu moves from a personal AI assistant toward an intelligent space shared by many people, what it is exploring may well be the next \u201cdesktop\u201d of the agent era. <\/p>\n\n\n\n

Huabu on GitHub:<\/strong> https:\/\/github.com\/microsoft\/Huabu (opens in new tab)<\/span><\/a><\/p>\n\n\n\n

Huabu project page:<\/strong> https:\/\/microsoft.github.io\/Huabu\/ (opens in new tab)<\/span><\/a><\/p>\n\n\n\n

<\/p>\n","protected":false},"excerpt":{"rendered":"

Today, the chat window is still the main way people interact with large models and agents: users ask a question, the AI answers, and each follow-up extends the thread. This is simple and natural, but fundamentally linear. When a question opens several directions worth exploring, people can usually pursue only one, and the rest is […]<\/p>\n","protected":false},"author":44093,"featured_media":1190878,"template":"","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-content-parent":199560,"msr_hide_image_in_river":0,"footnotes":""},"research-area":[13556],"msr-locale":[268875],"msr-post-option":[],"class_list":["post-1190869","msr-blog-post","type-msr-blog-post","status-publish","has-post-thumbnail","hentry","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_assoc_parent":{"id":199560,"type":"lab"},"_links":{"self":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1190869","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post"}],"about":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/types\/msr-blog-post"}],"author":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/users\/44093"}],"version-history":[{"count":2,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1190869\/revisions"}],"predecessor-version":[{"id":1190950,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-blog-post\/1190869\/revisions\/1190950"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media\/1190878"}],"wp:attachment":[{"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/media?parent=1190869"}],"wp:term":[{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=1190869"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=1190869"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/cm-edgetun.pages.dev\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=1190869"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}