When a production incident occurs, the last thing engineers want to do is burn critical time gathering context when they’re better served solving the problem.
Our Microsoft Digital engineers saw a better way forward: a single intelligent workspace, powered by an AI agent that makes identifying problems, troubleshooting them, and sharing context much quicker and simpler.
What began as a side project, inspired by a hackathon, quickly grew into a cross-team initiative.
“People were passionate about this project,” says Payal Awasthi, a senior software engineer in Microsoft Digital, the company’s IT organization. “Whenever people had some free capacity, they were coming in and contributing to it.”
The tool, called Smart DRI Agent, now supports over 40 teams at Microsoft, delivering more than 7,800 AI-assisted interactions and nearly 6,000 proactive incident notifications. This has saved the company an estimated 600 engineering hours per month, with bigger gains projected in the future.
“It saves a lot of time on context switching,” says Bhupesh Guptha Muthiyalu, a principal engineering manager in Microsoft HR. “The DRIs have ready-made information in hand, and based on that they’re typically able to resolve the incident quickly.”
Bringing all incident info into one place
In Microsoft Digital, our engineers serve as designated responsible individuals, or DRIs, who oversee production scenarios on a rotating basis. Their responsibilities include investigating incidents, monitoring service health, reviewing security work, and responding to other operational issues—work that can be both time sensitive and stressful.
“Whenever a new incident comes in, a DRI might have to elevate a lot of production permissions before they can get any insight into it. That’s one of the reasons we wanted to build something that can bring all the information into one place.”
Payal Awasthi, senior software engineer, Microsoft Digital
Meanwhile, the information that our DRIs need to take action on is typically distributed across several different portals and dashboards, along with troubleshooting guides, SharePoint sites, OneNote notebooks, and other engineering documentation.
Finding and assembling that information takes a while. For engineers who are new to a service or take over an unfamiliar DRI rotation, it can take even longer, as they may have to seek out the right subject matter expert to understand the situation.
“Whenever a new incident comes in, a DRI might have to elevate a lot of production permissions before they can get any insight into it,” Awasthi says. “That’s one of the reasons we wanted to build something that can bring all the information into one place; you don’t have to spend a lot of time switching between portals and elevating permissions.”
In some cases, important context can get lost in the shuffle, which can bog down the process and make it harder for DRIs to take the necessary next steps to resolve an issue.
“We depend on DRIs to be very proactive, and monitor these services 24/7 to make sure that everything is running smoothly,” says Sudhakar Sadasivuni, a principal group engineering manager in Microsoft Digital. “As part of our company’s native AI transformation, our idea was to make live site monitoring more proactive than reactive. We thought, why not use AI to make these decisions more intelligent, so that when DRIs come into the live site situation they’re already well informed.”
“We created goals and a 30-60-90-day plan and started making it more generic, where any service could simply onboard and start using that intelligence. It’s been one of the most successful open-source projects I’ve ever seen internally.”
Sudhakar Sadasivuni, a principal group engineering manager, Microsoft Digital
Teams in Microsoft Digital and the Microsoft HR organization were exploring related ideas for helping their DRIs, although the solutions they were considering were not identical. Once the teams recognized that they were addressing the same underlying operational problem, they decided to join forces.
Engineers from the two organizations created the first iteration of the agent during a Microsoft hackathon. As the prototype developed, they decided to treat it as a more structured effort.
“Later, we realized this was something that we should seriously invest in,” Sadasivuni says. “So we did a reset: We created goals and a 30-60-90-day plan and started making it more generic, where any service could simply onboard and start using that intelligence. It’s been one of the most successful open-source projects I’ve ever seen internally.”
Fine-tuning the product
One particularly challenging part of developing the agent was taking a scattered system and making it more uniform.
“Some services respond in different ways, and the reports that DRIs generate during handoffs can differ,” Sadasivuni says. “As we started interviewing various DRI teams, the requirements were piling up. We struggled with that for two or three months; eventually, we made some design changes to make it more generic.”
The initial implementation included more hard-coded behavior, but the team later separated configuration from the core logic and allowed teams to supply templates for workflows, such as DRI handoff reporting.
“We leveraged the framework functionality, but we also built a lot of custom code to support good results and good navigation through the system,” Awasthi says.
The team used Microsoft Semantic Kernel as an orchestrator. Different operational domains, such as incident management and Service 360, use separate sets of functions. The orchestrator determines which functions to call in response to a request.
“It’s easy, and there’s no limit on the number of services that I can query. That’s one of the biggest advantages that this agent brings.”
Lakshmi Kothamasu, principal engineering manager, Microsoft Digital
For similar-incident retrieval, the solution indexes incident information in Azure AI Search. The team found that search indexing alone didn’t always return useful matches, so developers added custom logic to assess similarity and improve the results supplied to DRIs.
“It’s easy, and there’s no limit on the number of services that I can query,” says Lakshmi Kothamasu, a principal engineering manager in Microsoft Digital. “That’s one of the biggest advantages that this agent brings.”
When an incident comes in, the agent automatically provides an analysis that includes suggested troubleshooting steps and historical context from similar issues in the past, which speeds up the resolution process.
“It will connect to your telemetry and get all the relevant traces, logs, and exceptions that happened in the last few minutes.”
Bhupesh Guptha Muthiyalu, principal engineering manager, Microsoft HR
Based on the incident details, title, description, and other data, the agent will search your troubleshooting guide, identify all the matching patterns, explain how to troubleshoot this issue, and give you needed context.
“It will connect to your telemetry and get all the relevant traces, logs, and exceptions that happened in the last few minutes,” Guptha Muthiyalu says.
It can also function proactively, such as delivering a summary first thing in the morning that identifies what an engineer needs to complete that day. All of the information is presented directly in Teams, helping it to seamlessly fit into engineers’ workflows.
Looking ahead: Agentic incident management
In future iterations, the team hopes to train the agent to actually take action and mitigate some incidents for the DRIs, keeping them in the loop and enabling them to sign off on the steps the agent takes.
For teams considering a similar project, the starting point is the DRI workflow itself. Identify where engineers spend time, where information is fragmented, and which repeated steps are strong candidates for assistance. Then, connect the workflow from detection and context gathering through to mitigation and continuous improvement, while keeping real people responsible for operational decisions.
“Without a strong live site culture, where the reliability and health of production services are a priority, this agent is of no use at all,” Sadasivuni says. “They go hand in hand; otherwise, it’s an unnecessary use of computing and artificial intelligence. This agent was designed with a single goal: to empower the human DRI. So, the more the DRI is demanding, the more the system will respond.”
End-to-end coverage is also key. DRIs should be able to turn to the agent anytime, at any stage for guidance, helping engineers to move faster and with more confidence. And with even more proactive measures in place, DRI work will only become more seamless in the future.
Key takeaways
If you’re considering building an AI agent to streamline your engineering operations, you can apply the following lessons from our Smart DRI Agent project:
- Start with a specific operational problem. Identify where engineers spend time gathering information or repeating manual work before selecting your technology.
- Bring relevant context together in one place. Connect the systems, documentation, telemetry, and operational data that engineers already rely on so they can spend less time searching for information and more time resolving issues.
- Design for proactive and interactive use. Support questions from engineers while also delivering important summaries and notifications automatically when the workflow requires them.
- Make team-specific behavior configurable. Use templates and configuration rather than assuming every service follows the same process.
- Test retrieval against real operational cases. Refine indexing and add domain-specific logic when general search results don’t provide relevant matches.
- Pair the technology with established live site practices. Assign clear ownership for reviewing and acting on the information the agent provides.
- Plan for reuse. Separate operational capabilities so they can be extended or made available to other agent experiences.
