{"id":16333,"date":"2024-08-29T08:00:00","date_gmt":"2024-08-29T15:00:00","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/insidetrack\/blog\/?p=16333"},"modified":"2026-08-14T15:17:06","modified_gmt":"2026-08-14T22:17:06","slug":"finding-and-fixing-network-outages-in-minutes-not-hours-with-real-time-telemetry-at-microsoft","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/insidetrack\/blog\/finding-and-fixing-network-outages-in-minutes-not-hours-with-real-time-telemetry-at-microsoft\/","title":{"rendered":"Finding and fixing network outages in minutes\u2014not hours\u2014with real-time telemetry at Microsoft"},"content":{"rendered":"\n
\"\"\/<\/figure>\n\n\n\n

With more than 600 physical worksites around the world, Microsoft has one of the largest network infrastructure footprints on the planet.<\/p>\n\n\n\n

Managing the thousands of devices that keep those locations connected demands constant attention from a global team of network engineers. It\u2019s their job to monitor and maintain those devices. And when outages occur, they lead the charge to repair and remediate the situation.<\/p>\n\n\n\n

To support their work, our Real Time Telemetry team at Microsoft Digital, the company\u2019s IT organization, has introduced new capabilities that help engineers identify network device outages and capture data faster and more extensively than ever before. Through real-time telemetry, network engineers can isolate and remediate issues in minutes\u2014not hours\u2014to keep their colleagues productive and our technology running smoothly.<\/p>\n\n\n\n

Immediacy is everything<\/strong><\/strong><\/h2>\n\n\n\n
\"Dave,
Aayush Dave, Astha Sinha, Abhijit Vijay, Daniel Menten, and Martin O\u2019Flaherty (not pictured) are part of the Microsoft Digital Real Time Telemetry team enabling more up-to-date and extensive network device data.<\/figcaption><\/figure>\n\n\n\n

Conventional network monitoring uses the Simple Network Management Protocol (SNMP) architecture, which retrieves network telemetry through periodic, pull-based polls and other legacy technologies. At Microsoft, that polling interval typically ranges between five minutes and six hours.<\/p>\n\n\n\n

SNMP is a foundational telemetry architecture with decades of legacy. It\u2019s ubiquitous, but it doesn\u2019t allow for the most up-to-date data possible.<\/p>\n\n\n\n

\n

\u201cThe biggest pain point we\u2019ve always heard from network engineers is latency in the data,\u201d says Astha Sinha, senior product manager for the Infrastructure and Engineering Services team in Microsoft Digital. \u201cWhen data is stale, engineers can\u2019t react quickly to outages, and that has implications for security and productivity.\u201d<\/p>\n<\/blockquote>\n\n\n\n

Serious vulnerabilities and liabilities arise when a network device outage occurs. But because of lags between polling intervals, a network engineer might not receive information or alerts about the situation until long after it happens.<\/p>\n\n\n\n

We assembled the Real Time Telemetry team as part of our Infrastructure and Engineering Services to close that gap.<\/p>\n\n\n\n

\n

\u201cWe build the tools and automations that network engineers use to better manage their networks,\u201d says Martin O\u2019Flaherty, principal product manager for the Infrastructure and Engineering Services team in Microsoft Digital. \u201cTo do that, we need to make sure they have the right signals as early and as consistently as possible.\u201d<\/p>\n<\/blockquote>\n\n\n\n

The technology that powers these possibilities is known as streaming telemetry. It relies on network devices compatible with the more modern gRPC Network Management Interface (gNMI) telemetry protocol and other technologies to support a push-based approach to network monitoring where network devices stream data constantly.<\/p>\n\n\n\n

This architecture isn\u2019t new, but our team is scaling and programmatizing how that data becomes available by creating a real-time telemetry apparatus that collects, stores, and delivers network information to service engineers. These capabilities offer several benefits.<\/p>\n\n\n\n

The advantages of real-time network device telemetry<\/h3>\n\n\n\n
\n
\n
\n

Prevention<\/strong><\/p>\n<\/div>\n\n\n\n

\n

Superior anomaly detection, reduced intent and configuration drift, the foundation for large-scale automation and less network downtime.<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

Security and compliance<\/strong><\/p>\n<\/div>\n\n\n\n

\n

Better detection of breaches, vulnerabilities, and bugs through automated scans of OS stalls, lateral device hijacking, malware, and other common vulnerabilities.<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

Observability<\/strong><\/p>\n<\/div>\n\n\n\n

\n

Visibility into real-time utilization data on network device stats, as well as steady replacement of current data collection technology and more scalable network growth and evolution.<\/p>\n<\/div>\n<\/div>\n\n\n\n

\n
\n

Service quality<\/strong><\/p>\n<\/div>\n\n\n\n

\n

More rapid network fixes, leading to a reduction in the baselines for time-to-detection and time-to-migration for incidents.<\/p>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n

\u201cDevices are proactively sending data without having to wait for requests, so they function more efficiently and facilitate timely troubleshooting and optimization,\u201d says Abhijit Vijay, principal software engineering manager with the Infrastructure and Engineering Services team in Microsoft Digital. \u201cSince this approach pushes data continuously rather than at specific intervals, it also reduces the additional network traffic and scales better in larger, more complex environments.<\/p>\n\n\n\n

At any given time, Microsoft operates 25,000 to 30,000 network devices, managed by engineers working across 10 different service lines. Accounting for all their needs while keeping data collection manageable and efficient requires extensive collaboration and prioritization.<\/p>\n\n\n\n

We also had to account for compatibility. With so many network devices in operation, replacement lifecycles vary. Not all of them are currently gNMI-compatible.<\/p>\n\n\n\n

Working with our service lines, we identified the use cases that would provide the best possible ROI, largely based on where we would find the greatest benefits for security and where networks offered a meaningful number of gNMI-compatible devices. We also zeroed in on the types of data that would be the most broadly useful. Being selective helped us preserve resources and avoid overwhelming engineers with too much data.<\/p>\n\n\n\n

We built our internal solution entirely using Azure components, including Azure Functions and Azure Kubernetes Service (AKS), Azure Cosmos DB, Redis, and Azure Data Lake. The result is a platform that network engineers can use to access real-time telemetry data.<\/p>\n\n\n\n

With key service lines, use cases, and a base of technology in place, we worked with network engineers to onboard the relevant devices. From there, their service lines were free to experiment with our solution on real-world incidents.<\/p>\n\n\n\n

Better response times, greater network reliability<\/h2>\n\n\n\n

Service lines are already experiencing big wins.<\/p>\n\n\n\n

In one case, a heating and cooling system went offline for a building in the company\u2019s Millennium Campus in Redmond, Washington. A lack of environmental management has the potential to cause structural damage to buildings if left unchecked, so it was important to resolve this issue as quickly as possible. The service line for wired onsite connections sprang into action as soon as they received a network support ticket.<\/p>\n\n\n\n

With real-time telemetry enabled, the team created a Kusto query to compare DOT1X access-session data for the day of the outage with a period before the outage started. Almost immediately, they spotted problematic VLAN switching, including the exact time and duration of the outage. By correlating the timestamps, they determined that the RADIUS registrations of the device owner had expired, which caused the devices to switch into the guest network as part of the zero-trust network implementation.<\/p>\n\n\n\n

As a result, the team was able to resolve the registration issues and restore the heating and cooling systems in 10 minutes\u2014a process that might have taken hours using other collection methods due to the lag-time between polling intervals.<\/p>\n\n\n\n

\n

\u201cThis has the potential to improve alerting, reduce outages, and enhance security,\u201d says Daniel Menten, senior cloud network engineer for site infrastructure management on the Site Wired team. \u201cOne of the benefits of real-time telemetry is that it lets us capture information that wasn\u2019t previously available\u2014or that we received too slowly to take action.\u201d<\/p>\n<\/blockquote>\n\n\n\n

It\u2019s about speeding up how we identify issues and how we then respond to them.  <\/p>\n\n\n\n

\u201cWith this level of observability, engineers that monitor issues and outages benefit from enhanced experiences,\u201d says Aayush Dave, a product manager on the Infrastructure and Engineering Services team in Microsoft Digital. \u201cAnd that\u2019s going to make our network more reliable and performant in a world where security issues and outages can have a global impact.”<\/p>\n\n\n\n

The future is in real time<\/h2>\n\n\n\n

Now that real-time telemetry has demonstrated its value, our efforts are focused on broadening and deepening the experience.<\/p>\n\n\n\n

\n

\u201cMore devices mean more impact,\u201d Dave says. \u201cBy increasing the number of network devices that facilitate real-time telemetry, we\u2019re giving our engineers the tools to accelerate their response to these incidents and outages, all leading to enhanced performance and a more robust network reliability posture.\u201d<\/p>\n<\/blockquote>\n\n\n\n

It\u2019s also about layering on new ways of accessing and using the data.<\/p>\n\n\n\n

We\u2019ve just released a preview UI that provides a quick look at essential data, as well as an all-up view of devices in an engineer\u2019s service line. This dashboard will enable a self-service model that makes it even easier to isolate essential telemetry without the need for engineers to create or integrate their own interfaces.<\/p>\n\n\n\n

That kind of observability isn\u2019t only about outages. It also enables optimization by helping engineers understand and influence how devices work together.<\/p>\n\n\n\n

The depth and quality of real-time telemetry data also provides a wealth of information for training AI models. With enough data spread across enough devices, predictive analysis might be able to provide preemptive alerts when the kinds of network signals that tend to accompany outages appear.<\/p>\n\n\n\n

\u201cWe\u2019re paving the way for an AIOps future where the system won\u2019t just predict potential issues, but initiate self-healing actions,\u201d says Rob Beneson, partner director of software engineering on the Infrastructure and Engineering Services team in Microsoft Digital.<\/p>\n\n\n\n

It\u2019s work that aligns with our company mission.<\/p>\n\n\n\n

\n

\u201cThis transformation is enhancing our internal user experience and maintaining the network connectivity that\u2019s critical for our ultimate goal,\u201d Beneson says. \u201cWe want to empower every person and organization on the planet to achieve more.\u201d<\/p>\n<\/blockquote>\n\n\n\n

\n
\"Key<\/figure>\n\n\n\n

Here are some tips for getting started with real-time telemetry at your company:<\/p>\n\n\n\n