ProfessionalReliability automation

Automated remediation of HTTP 503 failures on Azure

Application Insights telemetry and KQL-based Azure Monitor alert rules detect HTTP 503 degradation and trigger an Azure Automation runbook that restarts the affected virtual machines — no human in the loop.

Role
Designer and implementer
Context
ALIANDO · Managed Services area
Period
2025
View source on GitHub(opens in a new tab)

01Overview

A recurring failure mode with a known fix is a candidate for automation. I designed and implemented a closed loop on Azure that detects HTTP 503 errors from application telemetry and restarts the affected VMs automatically.

02Context

Azure-hosted applications running on virtual machines, operated by ALIANDO’s Managed Services area, where HTTP 503 (Service Unavailable) failures required an operator to notice the degradation and restart the service.

03Problem statement

Each 503 incident depended on a human — someone had to see the alert, diagnose and restart the VM. Time to recovery was bounded by response time, not by how long the fix takes.

04My responsibilities

  • Designed the end-to-end detection and remediation flow.
  • Wrote the KQL queries behind the alert rules.
  • Implemented the Azure Automation runbook that restarts the affected VMs.

05Architecture overview

Application Insights collects request telemetry from the application. An Azure Monitor log alert rule evaluates a KQL query for HTTP 503 responses; when it fires, an action group invokes an Azure Automation runbook written in PowerShell, which restarts the affected virtual machines.

Architecture overviewClosed-loop detection and remediation
Closed-loop detection and remediationThe application on Azure VMs sends telemetry to Application Insights. An Azure Monitor alert rule runs a KQL query for HTTP 503 responses and, when it fires, an action group invokes an Azure Automation PowerShell runbook that restarts the affected VMs.Azure MonitortelemetryKQLfiresinvokesrestart affected VMsApplicationAzure VMsApplication Insightsrequest telemetryLog alert ruleKQL · HTTP 503Action groupAutomation runbookPowerShell
  • Azure service
  • Observability
  • Delivery & automation
Text description of the diagram

The application on Azure VMs sends telemetry to Application Insights. An Azure Monitor alert rule runs a KQL query for HTTP 503 responses and, when it fires, an action group invokes an Azure Automation PowerShell runbook that restarts the affected VMs.

Flows

  • Application → Application Insights (telemetry)
  • Application Insights → Log alert rule (KQL)
  • Log alert rule → Action group (fires)
  • Action group → Automation runbook (invokes)
  • Automation runbook → Application (restart affected VMs)

Components

Application · Application Insights · Log alert rule · Action group · Automation runbook

06Technology stack

Detection
  • Application Insights
  • Azure Monitor alert rules
  • KQL
Remediation
  • Azure Automation
  • PowerShell runbook
Target
  • Azure Virtual Machines

07Implementation approach

  • Detection based on application telemetry, so the signal is what users actually experience — 503 responses — rather than an indirect host metric.
  • Alert logic expressed in KQL, which keeps the detection rule explicit, reviewable and easy to tune.
  • Remediation implemented as an Azure Automation runbook, so the fix is the same scripted action every time.
  • Platform-native wiring — alert rule → action group → runbook — with no extra infrastructure to run.

08Challenges & engineering decisions

  1. Challenge

    Distinguishing real service degradation from noise.

    Decision

    Base the alert on a KQL query over request telemetry, where conditions and thresholds can be expressed precisely and adjusted without changing the remediation.

  2. Challenge

    Making the remediation repeatable and safe to run unattended.

    Decision

    Encapsulate it in a single runbook scoped to restarting the affected VMs, rather than a broader set of ad-hoc actions.

09Security considerations

  • An unattended runbook should run under an identity with only the permissions needed to restart the target VMs.
  • Keeping the remediation narrow (restart only) limits the impact of a false positive.

10Operational considerations

  • Recovery no longer waits for an operator to respond to the alert.
  • Alert history and runbook job history give a record of every automated intervention.
  • Restart is a mitigation, not a root-cause fix — automated recoveries remain visible for follow-up.

11Results & impact

  • HTTP 503 failures detected and remediated automatically, without human intervention.
  • Faster incident response for a known, recurring failure mode.

12Lessons learned

  • Automate known fixes for known failures, and keep the automation as narrow as the fix.
  • Alerting on user-facing signals produces more meaningful automation than alerting on infrastructure proxies.