AI for DevOps: Faster Triage, Human Hands on Production

TL;DR: Operations work splits into reading, writing, and doing. AI is strong on the first two, compressing a wall of logs into a triage summary, drafting the runbook nobody wrote, producing the postmortem skeleton while memories are fresh, and must be kept on a leash for the third. The teams getting real MTTR improvements use AI as an always-awake junior analyst during incidents and a tireless technical writer between them, while every production-touching action stays behind existing change controls with a human executing. This guide covers the four workflows, the prompts, and exactly where the execution line sits.

Part of the AI for Engineering hub. The ticket-deflection patterns here overlap with the IT helpdesk guide; runbook drafting connects to AI for technical documentation.

Where AI actually helps on-call

At 3 a.m., the on-call engineer’s scarce resource isn’t typing speed, it’s working memory. Fifteen dashboards, four alert channels, a deploy log, and a Slack thread all compete for a brain running at half capacity. A large language model doesn’t get tired, doesn’t skim, and summarizes arbitrarily large text quickly. That maps to four workflows:

Workflow 1: incident triage

The pattern that works: when an incident opens, feed the model the alert payload, the last 30-60 minutes of relevant logs (scrubbed, see below), and the recent change log, and ask for a structured readout.

You are assisting incident triage. Inputs: the firing alerts [paste], application logs from the affected window [paste], and deploys/config changes from the last 24h [paste]. Produce: (1) a five-line summary of what the signals show, without speculation; (2) the top 3 hypotheses for the cause, each with the specific evidence for it, evidence against it, and one check that would confirm or kill it; (3) anything in the logs that looks anomalous but doesn’t fit any hypothesis. Rank hypotheses by evidence, not by how common the failure type is. If the signals are insufficient to form a hypothesis, say so.

The design choices matter. Forcing evidence against each hypothesis counters the model’s agreeable streak. Asking for a disconfirming check turns each guess into a next action. And the explicit permission to say “insufficient signal” reduces hallucinated certainty, the failure mode where a model, asked for a cause, invents one, fluently, because that is what it was asked to produce. Treat every AI diagnosis as a hypothesis until a human confirms it against real telemetry. Plausible-but-wrong is more dangerous mid-incident than no answer, because it burns the clock in the wrong direction.

Modern observability platforms (Datadog, New Relic, Grafana, PagerDuty and peers) now ship these summarization features natively; general assistants like Claude, ChatGPT, or Gemini do the same job when you bring the context yourself. The native versions see more telemetry; the general ones reason more flexibly. Either way, the confirm-before-acting rule is identical.

Workflow 2: remediation, where the line is

Draw the line by action class, not by enthusiasm:

1. AI drafts, human reviews, human runs, the default for everything. The model proposes steps and exact commands, citing which runbook or doc each comes from. The engineer reads each command before running it. A model will occasionally produce a command that is syntactically perfect and operationally destructive, the wrong namespace, a missing --dry-run, a deletion where a stop was meant.
2. Pre-approved automation, AI-triggered with human confirmation, for actions that were safe to automate before AI existed: restart a stateless service, scale a pool, roll back to the previous known-good deploy. The AI agent pattern is fine here because the action itself is bounded and reversible; the human click is the gate.
3. Never unattended, anything state-destroying, data-touching, security-relevant, or novel. No model-composed command runs against production without human eyes, regardless of how many months of good behavior the tooling has accumulated.

Widen autonomy one action type at a time, backed by months of logged proposals you can audit. The question is never “is the AI usually right?”, it’s “what happens the time it’s wrong?”

Workflow 3: runbooks, the tribal-knowledge extractor

Most teams’ runbook coverage is a handful of polished documents and a vast oral tradition. AI attacks this from two directions:

• Drafting from artifacts. Feed the model the last five incident tickets for a recurring issue, plus the Slack thread and shell history from the most recent occurrence, and ask for a runbook draft: symptoms, confirmation checks, step-by-step remediation, verification, escalation path. Twenty minutes of senior-engineer editing turns that draft into a document that previously never got written at all, the economics, not the prose quality, are the breakthrough.
• Answering from runbooks. Once runbooks exist, a retrieval-augmented generation setup, the model answers from your documents rather than its training data, gives on-call a conversational interface to them: “payments-api is throwing connection-pool exhaustion, what’s the procedure?” pulls the relevant runbook instead of a plausible improvisation. Grounding answers in retrieved docs is the single biggest hallucination reducer available for this workflow.

The one hard rule: a human who has actually performed the procedure validates every runbook before publication. On-call engineers follow runbooks precisely in the moments they lack context to question them. An unvalidated AI step in a trusted document is a trap you set for your own team.

Workflow 4: postmortems

The postmortem is the highest-value document nobody has energy to write. AI handles the mechanical 70%: assembling the timeline from Slack timestamps, alert history, and the deploy log; drafting the summary; structuring the document. Humans own the 30% that is the actual point, why the contributing factors lined up, what the systemic fixes are, which action items are real. A useful prompt:

Build an incident timeline from these sources: the incident channel export [paste], the alert history [paste], the deploy log [paste]. Every entry needs a timestamp and a source. Mark gaps where the record is silent rather than inferring what happened. Then draft a postmortem skeleton: summary, impact, timeline, and empty sections for contributing factors and action items, leave those empty; humans will write them.

“Mark gaps rather than inferring” is load-bearing: a timeline with confident invented entries is worse than one with visible holes.

The data rule for all four workflows

Logs and incident channels are dense with secrets: tokens, session IDs, customer emails, internal hostnames, connection strings. Before any of it reaches a model: sanctioned tools only, business/enterprise tiers with contractual no-training terms, and automated scrubbing of secrets and PII in the pipeline (your secret-scanning tooling can usually be repurposed). Write log handling explicitly into your AI acceptable use policy, it is the most common gap in otherwise-decent policies.

A 30-day pilot

1. Week 0: baseline MTTR, time-to-diagnosis if you track it, and runbook coverage for your top ten alert types. Pick one service’s on-call rotation.
2. Weeks 1-2: triage summaries only. On-call runs the triage prompt (or the platform feature) on real incidents; log each hypothesis and whether it was right, wrong, or unverifiable.
3. Weeks 3-4: add runbook drafting, target the five highest-frequency incident types without runbooks. Senior engineers validate before publication.
4. Day 30: review the hypothesis log (hit rate, and any harmful misses), runbooks shipped, and on-call’s subjective load. Expand to remediation proposals only if the triage hit rate earned it.

FAQ

Can AI diagnose production incidents? It generates ranked hypotheses from logs and change history fast, which compresses time-to-diagnosis, but every hypothesis needs human confirmation against telemetry before action. Confident wrong explanations are the standard failure mode.

Should AI be allowed to run remediation automatically? Only pre-approved, bounded, reversible actions that were already automation-safe, restarts, scaling, known-good rollbacks, with a human confirmation gate. Novel or destructive actions stay human-executed.

Is it safe to paste production logs into an AI tool? Only sanctioned tools on no-training business tiers, and only after scrubbing secrets and PII. Logs are a leak path; put them in your acceptable use policy explicitly.

Will AI-written runbooks be trusted by the team? They earn trust the same way human-written ones do: someone who has performed the procedure validates each one. AI changes the economics of drafting, not the requirement for validation.

Related: the IT helpdesk guide applies the same triage-and-deflect pattern to internal support, and AI for technical documentation extends the runbook workflow to docs generally. Back to the AI for Engineering hub.

Wondering whether on-call or another workflow is your best first AI project? The free AI readiness assessment will tell you in ten minutes.