First On-Call

Shift 4 — Blameless reviews + status updates

Aisha drafts first, you coach with questions

What you need

The checkout-errors incident from shift 3. Aisha knows it. Use it as the example throughout.

How it goes

Ask Aisha to write a blameless review on her own first: impact, timeline, what went well, contributing factors (as system gaps, not names), actions.

Read her draft. Check: does it name people? Do the contributing factors describe conditions and missing safeguards?

For any person-blame sentence, ask a question instead of rewriting: "What was missing in the system that let this reach production?"

Work through a status update together. She fills in the fields; you check it: what's affected, what we're doing, when's the next update.

Wrap up: ask what she'd do differently if this exact incident happened again next week.

Say this

"Walk me through your review. Which contributing factor do you think did the most damage before we caught it?"

"Not wrong — but can you write that as a gap in the system, not a gap in a person?"

Don't say this

"Sam shouldn't have merged that." Even as an example of what to avoid, it puts Sam in the room. Use the system frame instead: "the change reached production without a check on the payment path."

Your turn to lead

Aisha found the bad commit in two minutes using the AI log-search tool in shift 3. You hadn't used it before. After the review, ask her to show you the query she ran. You're the learner here — tell her that.

What we've learned

About Aisha

Shift 1

She froze when the pager went and didn't know where to look first.

So: built the first-five-minutes card together. She used it in shift 2 and acknowledged in under a minute.

Shift 2

Her instinct under pressure was to read logs and find the bug before mitigating. She spent 25 minutes on the stack trace while errors were live.

So: "roll back or dig in?" drill. The rule: if a deploy matches the spike, roll back first, investigate after.

Shift 3

She rolled back in 6 minutes and posted a clear status update. Her blameless review named the person who merged the change.

So: blameless reviews are shift 4's focus. The rollback instinct is solid now.

About Dave

Shift 2

When Aisha was slow, you grabbed the terminal. She stopped driving.

So: narrate and ask questions. "What would you check next?" not "here, let me show you."

Shift 3

Aisha used the AI log-search tool to find the bad commit in two minutes. You hadn't used it.

So: she demos it to you in shift 4. You are behind her on this one specific thing. Sit with that.

The on-call shifts

Shadowing. Small incident; Aisha froze. Built the first-five-minutes card together.

First solo page. Checkout errors after a deploy. She debugged instead of rolling back. You narrated; she typed. Built the "roll back or dig in?" drill.

Clean rollback. Six minutes. Clear status update. Review named a person. She showed you the AI log tool for the first time.

Blameless review + status update. (This shift.) She drafts. You coach with questions. She demos the log tool for you.

Where this could go: Aisha guides the next new hire through their first on-call rotation. Then a game-day drill together — a practice incident, controlled, on purpose.

When it's tricky

IF: her review names a person

Ask: "What was missing in the system that let this change reach production?" Don't rewrite it for her.

IF: she gets defensive about naming the person

Acknowledge it: "You're right that someone made the change. Blameless means we fix what the system was missing — not that we pretend the change didn't happen." Then ask the system question.

IF: she's stuck and doesn't know how to start

Start with what went well: she rolled back in 6 minutes, the status update was clear. That unlocks the rest.

IF: you want to rewrite her review

Ask instead: "What would a canary check have caught here?" You're not the author. She is.

IF: the review is good but the status update is vague

Ask: "What did customers see? What are we doing right now? When will we update again?" Those three answers are the update.

IF: Aisha asks Dave how to run a log query

Flip it: "Show me how you ran it in shift 3." You don't know this tool better than she does.

Tell us & ask

Note to copy back to MetaDAX after shift 4

Shift 4 done. [Did Aisha draft the review before you gave feedback? yes / no] The review [stayed system-focused / named a person and we redirected]. Status update: [clear first draft / needed one round of questions]. Log-tool demo: [happened / skipped]. What surprised me: [your words here].

Ask your own AI tool

Copy this prompt into your AI chat when you need to think through a coaching moment.

I'm coaching a junior engineer on writing blameless incident reviews. She tends to name the person who made a change instead of the system gap that let it reach production. How do I redirect that without dismissing what she observed? Give me one reframing question I can ask her. Don't write the review for me.

Did this help?

Shift 4 — Your turn

Three shifts done · blameless reviews and status updates today

What you're doing

Write a full blameless review of the checkout-errors incident from shift 3. Then write a status update for it. Dave will read both — but you write the first draft on your own.

The key idea

Blameless means you look at what the system was missing, not who made the mistake. Someone merged a change. The question is: what safeguard was absent that would have caught it before it reached production?

Your shift 3 review named the person. That's the one thing to change today.

The incident

HighCheckout errors after deploy

Impact: Checkout error rate ~3× baseline. Many customers could not complete purchases from about 09:09 to 09:26 (17 minutes).

Timeline: Deploy at 09:00 · Errors detected 09:09 · Alert acknowledged 09:12 · Rollback at 09:18 · Resolved 09:26

What went well: Fast acknowledgment, clear status post, rollback in 6 minutes once you saw the pattern.

Contributing factors (write these as system gaps):

No automated canary check on the payment path before the deploy went live
The test suite didn't cover the checkout flow with the new payment method
No automatic rollback trigger on a sustained error-rate spike

Actions to write: Add a payment-path smoke test to CI · Add a canary stage to the deploy pipeline

Your turn to lead

You found the bad commit in two minutes in shift 3 using the AI log-search tool. Dave hadn't used it before. After you finish the review today, show him your query. He asked you to demo it to the team. You know something he doesn't — that's yours.

Where this is going

Once this lands: you guide the next new hire through their first on-call rotation. After that, a game-day drill with the team — a practice incident, controlled, on purpose.

Pager Drill

A simulated incident. Pick the best next action at each step. Wrong answers show you the pattern behind the mistake.

Four decisions, in order. Each wrong answer shows the pattern behind the mistake.

The pager fires. Dashboard shows checkout error rate climbing — 3× baseline. A deploy went out 8 minutes ago. What do you do first?

You've acknowledged. Dashboard confirmed: checkout errors 3× baseline, started 9 minutes ago. What's the right next move?

Status posted. The deploy at 09:00 and the error spike match almost exactly. What do you do?

Error rate is back to baseline. Incident resolved. What do you write in the resolution update?

Blame or system?

For each sentence from a draft incident review, pick whether it's blame language or system language. Wrong answers explain why.

Status update builder

Fill in the four fields and watch your update take shape. Copy it when it looks right.

State

What's affected (service + customer impact)

What we're doing right now

When the next update comes

Preview

Fill in the fields above to see your update.

Questions you might have

What if I genuinely know who wrote the bad code?

You can know. In the blameless review, the question is different: what made it possible for that code to reach production? That's the system gap.

A person making a mistake is not a safeguard. The test that didn't cover the payment path — that's a safeguard you can fix.

But what if there was no system gap — someone just skipped the checklist?

Then the system gap is: the checklist isn't enforced automatically. Why not? Could it be? That's the action item — not "person X needs to follow the checklist."

You can't build a reliable system by expecting individuals to never skip steps. You build the system so skipping a step is harder than doing it.

What if rolling back would break something else?

Check the rollback target before you run it. If you're unsure, post a quick update: "Assessing whether rollback is safe. Next update in 5 min." Then decide.

If rollback is genuinely risky, the path is a forward fix or a feature flag. But default to rollback — it's the fastest route to a known-good state.

How do I know if a rollback is safe?

Quick test: did this deploy touch the database schema or any shared state? If yes, rollback is riskier — check with a second engineer first. If no, rollback is usually safe to run.

How long should a blameless review take to write?

For an incident like shift 3, 30–45 minutes for a first draft is about right. Timeline and what-went-well are fast. Contributing factors take the most time — that's where the real work is.

If you're past an hour on a single incident review, you're probably going too deep. The review is a starting point for action items, not a forensic investigation.

What makes a good action item vs a weak one?

Good: specific, ownable, and tied to a system gap. "Add a payment-path smoke test to CI" — yes. "Be more careful next time" — no. If you can't assign it to a system or process, it's not an action item.

What if a contributing factor sounds like an excuse?

Good instinct. There's a real difference. "The test suite didn't cover the payment path" is a system gap. "We were moving fast" is a context note, not a gap you can fix.

Test: can you write a concrete action item for it? If yes, it's a contributing factor. If no, move it to context and keep going.