Today, the chat window is still the main way people interact with large models and agents: users ask a question, the AI answers, and each follow-up extends the thread. This is simple and natural, but fundamentally linear. When a question opens several directions worth exploring, people can usually pursue only one, and the rest is buried in an ever-longer chat history. Human thinking works differently: people jump between ideas and put related materials side by side to compare them. They set a question aside and come back to it later. Sometimes simply placing two pieces of information next to each other reveals a hidden connection.
Huabu, recently introduced by Microsoft Research Asia, aims to change this. Instead of stacking the web pages, images, notes, conversations, sketches, and final plans from an agent’s research in a chat thread, it lays them out in an unbounded two-dimensional space. Information can be placed side by side, moved, resized, grouped, and linked, and users can edit content directly and ask the agent to keep working on it at any time.
Beyond a new interface, Huabu attempts to answer a fundamental question: as AI gets better at handling complex information, how should people think together with it?
Built for the way people naturally think
Unlike an online whiteboard, which mainly offers drawing and layout, Huabu instead provides project workspaces, called Spaces, designed for people and AI to work together. A Space connects the whole process in one environment, from gathering information and organizing ideas to having AI carry out tasks and capturing the results.
In Huabu, the basic unit of work is a project Space rather than a round of chat, so thinking no longer has to follow a single conversational path. Users can create a Space for each project and gather everything related to it in one place: PDFs, web pages, images, notes, chat histories, sketches, and agent-generated content. Materials can sit side by side for comparison, and each idea can become a node to expand and refine at any time. Content can be linked, and items on the same topic can be grouped, so scattered information grows into a project Space that can be explored over time.
Work in Huabu moves through three stages: Collect, Organize, and Act. But this is not a fixed, one-way process. “The three stages form a cycle,” said Luna Qiu, senior program manager at Microsoft Research Asia – Shanghai. “New results from the agent flow back into Huabu and become material for the next round of thinking and work. People can reorganize based on the new information and then hand the agent new tasks.”
Collect: Freeing users from playing information courier
In a conventional chat, users act as “context couriers.” Web pages, local documents, past conversations, and team discussions sit in separate windows, and getting AI to draw on them means copying and pasting them in piece by piece, a tedious process in which information is easily missed.
In Huabu, any material related to a task can be uploaded directly and automatically becomes its own node. More importantly, ideas set aside for later, unexplored lines of thought, and past discussions all stay in the Space as context the agent can read and use.
Organize: Helping AI understand how information connects
Organizing is also a joint effort between people and AI. Users can arrange material themselves or leave the job to the agent. The agent places the material it retrieves on Huabu as separate nodes, each with an automatically generated summary. Users can then zoom in on anything they want to look at more closely, keep what matters, and remove what does not, avoiding the cognitive overload of a linear chat that delivers a large block of text all at once.
Given dozens of papers, for example, the agent can arrange them by research timeline and academic lineage, draw lines to mark citations and opposing views, and produce a visual map of the research. Given access to a code repository, it can break down the system architecture, with nodes for modules and lines for the dependencies between them, showing the full technical structure at a glance. For a proposal, it can sort material into core ideas, alternatives, and questions still to be verified, place related items together, and frame each group as a block of ideas.
“Huabu doesn’t just pile up material,” Qiu said. “Through semantic understanding, it organizes scattered information automatically and turns the implicit logical connections in people’s minds into a visible spatial structure. At the same time, AI can recognize the relationships between ideas that people express through position, grouping, and links.”
Zooming lets users move between the big picture and the details: zoomed out, nodes show only the AI’s summaries; zoomed in, they show the full text, high-resolution images, and handwritten annotations. In internal testing by the team of Nan Chen, a research software development engineer at Microsoft Research Asia – Shanghai, zooming greatly reduced the chance of losing track of ideas during multi-branch exploration. “Overview and detail are no longer an either-or choice,” Chen said. “Zooming solves the problem of large projects becoming cluttered and hard to manage.”
Act: Letting agents step out of the chat box and into a shared space
Once information is collected and organized, users can call on local or cloud agents at any time. Their output flows back onto the shared canvas instead of staying in a chat box, becoming new material for the next round of thinking and agent tasks.
Revising an AI-generated image or slide deck, for example, used to take long, complicated prompts, and even a slight misstep could mean losing the original style. In Huabu, users can express intent through spatial relationships. Telling the agent, “Make this image match the style of the one next to it,” is enough to align the styles, and a simple instruction is all it takes for the agent to adjust spacing while leaving everything else untouched.
“Huabu changes more than how people interact with agents,” Qiu said. “It lets people tell AI what they want in a way that comes naturally.”
Huabu understands spatial instructions while keeping context continuous, and it saves each round of edits as a version. Users need not re-explain earlier steps, and the agent keeps working from what is on the canvas. This what-you-see-is-what-you-get interaction turns creation into a continuous loop of iteration and cuts the repeated trial and error that unpredictable results can cause.
The logic behind Huabu: How agents understand spatial thinking
The three-stage workflow of Collect, Organize, and Act is more than a set of features. It rests on a new understanding of how people think and is built around three core principles: Externalized Thinking, Shared Cognitive Space, and Natural Interaction.
Externalized Thinking: Layout itself is information
People naturally use space to organize information: they enlarge what matters, keep related material close, put distance between opposing views, draw lines to show logical relationships, and give scattered guesses an area of their own. These moves convey intent, but conventional AI tools cannot recognize them.
Huabu designs its data storage and the agent’s tool calls so that the agent reads not just a node’s content but also its position, size, and connections. “In Huabu, position is information, size is information, and what something sits next to is information too,” Chen said. “Enlarge a passage of text, and the agent knows it matters to the user. Connect two nodes, and it knows they are closely related. That is far more intuitive than repeating ‘this part is important’ in a prompt.” With thinking externalized this way, people no longer need long prompts to “translate” their intent for AI; spatial relationships express it directly.
Shared Cognitive Space: From text context to spatial context
In a chat window, context is chronological text, and focusing the AI on specific material means copying it in and spelling out its scope. Huabu introduces a spatial context mechanism instead, in which the space itself organizes context and points to what is meant. When an agent is placed in a particular area of Huabu, it reads the nodes around it and follows their links to related material, so users do not have to set boundaries for what it should reference. For example, with a hand-drawn sketch placed nearby, the agent interprets the task from the surrounding text, images, and other nodes.
People and agents thus read the same “thinking map,” cutting the cost of passing and explaining information back and forth. That is the core of a Shared Cognitive Space: a unified workspace where people and AI share a view of the whole project and keep collaborating.
Natural Interaction: Many ways to give input, with people in control
Beyond typing, Huabu supports touchpads, freehand drawing on graphics tablets, and voice to suit different stages of thinking. It will improve handwriting recognition so that agents can turn rough drafts annotations scribbled on the fly into structured content.
Huabu also follows a human-in-the-loop principle: every change the AI makes to the Huabu is recorded and shown. “The agent doesn’t force users to review every change, but a complete record is necessary,” Qiu explained. “If the AI’s edits were hidden in the background, users would lose their sense of control over the project. Showing every change builds trust in human-AI collaboration.”
This reflects another dimension of how Huabu understands natural interaction: natural collaboration does not mean handing everything to AI, but keeping its actions transparent and adjustable at any time. AI’s job is to push the limits of what people can do, while people always have the final say on decisions and changes. In Qiu’s words, “Truly natural interaction means AI’s actions always serve human thinking, not the other way around.”
Shared Cognitive Space: From text context to spatial context
In a chat window, context is chronological text, and focusing the AI on specific material means copying it in and spelling out its scope. Huabu introduces a spatial context mechanism instead, in which the space itself organizes context and points to what is meant. When an agent is placed in a particular area of Huabu, it reads the nodes around it and follows their links to related material, so users do not have to set boundaries for what it should reference. For example, with a hand-drawn sketch placed nearby, the agent interprets the task from the surrounding text, images, and other nodes.
People and agents thus read the same “thinking map,” cutting the cost of passing and explaining information back and forth. That is the core of a Shared Cognitive Space: a unified workspace where people and AI share a view of the whole project and keep collaborating.
Natural Interaction: Many ways to give input, with people in control
Beyond typing, Huabu supports touchpads, freehand drawing on graphics tablets, and voice to suit different stages of thinking. It will improve handwriting recognition so that agents can turn rough drafts annotations scribbled on the fly into structured content.
Huabu also follows a human-in-the-loop principle: every change the AI makes to the Huabu is recorded and shown. “The agent doesn’t force users to review every change, but a complete record is necessary,” Qiu explained. “If the AI’s edits were hidden in the background, users would lose their sense of control over the project. Showing every change builds trust in human-AI collaboration.”
This reflects another dimension of how Huabu understands natural interaction: natural collaboration does not mean handing everything to AI, but keeping its actions transparent and adjustable at any time. AI’s job is to push the limits of what people can do, while people always have the final say on decisions and changes. In Qiu’s words, “Truly natural interaction means AI’s actions always serve human thinking, not the other way around.”
From “my agent” to “our Space”: The next desktop for the agent era
Huabu already supports one person working with multiple agents: the agents share a Space, each handles its own part of the work, and all their output flows back into Huabu. The researchers see this as only a starting point, with much more room for Huabu to grow.
In the workspace they envision, “me” and “my agent” will be joined by colleagues and their agents. Several people will be able to work in the same Space at once, each bringing their own agent. People will talk in real time while their agents work together, and a higher-level agent can coordinate roles and assign tasks, creating a more natural network of collaboration among people, between people and agents, and among agents.
Huabu is one attempt at a new human-AI interface for the agent era, and there is no standard answer yet. Today’s AI chat is not the final form, the researchers note, just as the command line was not the final form of early computers. Huabu is open source so that users, developers, and researchers can explore the question together.
As Huabu moves from a personal AI assistant toward an intelligent space shared by many people, what it is exploring may well be the next “desktop” of the agent era.
Huabu on GitHub: https://github.com/microsoft/Huabu (opens in new tab)
Huabu project page: https://microsoft.github.io/Huabu/ (opens in new tab)
Artificial intelligence