Vibe Gaming

What if, instead of controlling every move your game character made, you could tell them what you wanted and watch them live, act, and connect with others on their own in the changing game world?

This is the idea behind VEGA, our exploration of vibe gaming powered by agentic AI. Characters can act on their own, pursue goals, react to a changing world, and make decisions independently while remaining connected to the player. Instead of controlling every action, the player can offer guidance through natural conversation when they choose, and the character can seek or incorporate that guidance when appropriate.

That connection continues even when the player leaves the game. Characters can keep living in the world, send updates to the player, and ask for input when they need it. In a shared multiplayer world, they can also meet, react to, and talk with other players’ characters, allowing relationships and stories to emerge without continuous direction from their players.

We aim to design VEGA as a flexible agent framework for 3D embodied games, where characters can perceive, act, and interact in persistent single-player or multiplayer worlds. For the prototypes and demonstrations in this article, our testbed is Luanti (opens in new tab), an open-source voxel game platform.

Every Character Is Unique

In VEGA, every character begins with a soul shaped by its player. A soul is open-ended guidance about who the character is and how it might live in the world. It can be brief or detailed, follow any structure, or simply be written as free text. The soul does not prescribe every action. Instead, it gives the character a distinct perspective that influences what it notices, which goals it pursues, and how it responds to players and other characters.

That identity continues to develop through experience. As characters explore the world, interact with others, and overcome challenges, they accumulate their own memories and skills. Even characters that begin with similar souls can grow in different directions, shaped by where they have been, what they have learned, and the relationships they have formed.

In the examples below, we describe each soul through an identity, personality, behavioral tendencies, communication style, and long-term dream, but these are possibilities rather than predefined categories. Choose Nova, Milo, or Ares below to see how a distinct soul becomes the foundation for a unique journey.

Living on Their Own

Once a character enters the world, they do not wait for the player to direct every move. They observe what is happening around them, choose what to do next, turn broad goals into concrete actions, and adapt when a plan does not work.

This ongoing experience also changes the character. They remember meaningful events, learn reusable skills, and apply what they have learned to future situations. The following demo opens this inner loop, showing how a character moves from observation and intention to action, reflection, and growth.

Influence, Not Control

Living on their own does not mean living beyond the player’s reach. Instead of taking back direct control, players can influence their characters through natural conversation: asking what they see, suggesting a direction, pointing out a problem, or sharing a preference. The character interprets that input in context and decides how to respond while continuing to pursue their own goals.

This creates a collaboration rather than a sequence of commands. Player guidance can reshape a plan, correct a misunderstanding, or introduce a new constraint, but the character can also explain their reasoning, ask for clarification, or resist a request that conflicts with who they are. The following demo shows how a conversation can influence the selected character’s next actions without taking away their agency.

Living Together

Autonomous living becomes more interesting when characters share a world. They can talk with one another, exchange observations, divide responsibilities, and coordinate around common goals. Each character contributes a different perspective shaped by their soul, memories, and skills, allowing the group to organize without waiting for a player to direct every step.

In the following demo, the characters work together to improve their new home. They identify what is missing, decide who should gather materials and who should build, report progress, and warn one another about danger. Their shared channel turns individual actions into team knowledge, helping them adapt and accomplish more together while the player is away.

Putting It All Together

Vibe gaming brings autonomous living, player influence, and character collaboration into the same shared experience. The player does not need to control every action or coordinate every detail. Instead, they can step in when their perspective matters: warning a character about danger, changing a priority, or helping the team make a difficult decision. The characters interpret that guidance, communicate with one another, and decide how to carry it forward together.

In the final demo, Nova discovers a threat, the player encourages her to retreat, and the team responds according to their distinct strengths. Nova scouts, Ares protects the group, and Milo builds what they need to escape. As the situation changes, they share observations, reuse learned skills, form new memories, and adapt their plan. The result is a multiplayer experience in which players and unique, evolving characters shape the story together.

Evolving Through Experience

Across these demos, the characters do more than complete isolated tasks. Each experience leaves something behind. A discovery can become a memory, a successful action can become a reusable skill, and a conversation can change how a character understands its priorities. What begins as a soul gradually develops into a history: not only who the character was designed to be, but what they have seen, learned, and chosen to do.

This growth also changes how the characters participate in the world. Nova’s exploration gives the team knowledge it can act on. Milo’s building experience becomes practical know-how that can be reused when the group needs a way forward. Ares’s commitment to protection shapes how he responds when danger appears. Their identities remain recognizable, but their behavior becomes grounded in accumulated experience rather than a fixed script.

For players, this means progression can be more than unlocking stronger abilities or collecting better equipment. It can also be the gradual formation of a character with continuity: one who remembers shared moments, carries lessons into new situations, and develops relationships through living alongside others. The demos offer an early view of that possibility, where growth is expressed not only in what a character can do, but in who they are becoming.

Looking to the Future

VEGA offers an early glimpse of a future in which every player could have a character that grows alongside them. These characters could continue exploring, learning, and forming relationships while players are away, then share their progress and memorable moments. Players would become not only controllers, but also coaches and companions who shape their characters’ journeys through conversation.

VEGA also suggests a new paradigm for multi-agent learning. Rather than learning only from predefined objectives, characters with distinct inner drives can develop memories, skills, and resources through self-directed experience. When shared challenges arise, they can form teams and collaborate according to their evolving strengths. This could open a new design space for persistent game worlds shaped jointly by players and autonomous characters.

---

Acknowledgements

The Microsoft VEGA team consists of Xianliang Yang, Kaixin Wang, Zhan Cheng, Chang Xu, Chuheng Zhang, Ling Zhang, Adrian Brown, and Li Zhao. Key areas of contribution include:

We also thank our colleagues, including Tim Pearce from MSR NYC, for their valuable feedback on this research project and article.