PowerShell Genie

PowerShell Operational runbooks

A runbook is a procedure someone can follow at three in the morning without knowing what you know. Written well, it is also the thing you automate from.

Lesson 3 min read Windows PowerShell 5.1 and PowerShell 7

Documentation: Microsoft Learn

Overview

The structure that works has five parts. What triggers it, so nobody runs it speculatively. Pre-checks that confirm the situation matches, because the same alert has more than one cause. The action, with a dry run first where that is possible. Verification that the fix worked, which is different from the action having completed. And an escalation path with a condition, so nobody has to decide alone whether this is now someone else's problem.

The verification step is the one most often left out. Restarting a service succeeds; the service being healthy afterwards is a separate fact. A runbook that stops at the action tells you the button was pressed, not that the problem is gone.

Write commands out in full, with real parameter names. A runbook that says check the service assumes the reader knows how, which defeats the purpose.

Escalation needs a condition, not a suggestion. Twice in an hour is actionable; if it keeps happening is not.

When the same runbook has been followed a few times without judgement being needed, that is the signal to automate it. Runbooks that still need judgement should stay manual, and saying so is part of writing a good one.

Inspect this example’s structure

Read the command and its supplied arguments. This example requires real PowerShell or its module; this diagram does not execute it.

$runbook = [pscustomobject]@{ Name = 'Restart failed print spooler' Trigger = 'monitoring alert PRN-001' Precheck = 'Get-Service Spooler; Get-WinEvent -FilterHashtable @{LogName="System";Id=7031}' Action = 'Restart-Service Spooler -WhatIf, then without -WhatIf' Verify = 'Get-Service Spooler | Where-Object Status -eq Running' Escalate = 'if it fails twice in an hour, raise to platform team' } $runbook | Format-List
  1. Stage 1$runbookPositional values or no named parameters
  2. Stage 2Format-ListPositional values or no named parameters

    An example to inspect

    Preview the example. Sign in for the full lesson and activities.

    $runbook = [pscustomobject]@{
      Name = 'Restart failed print spooler'
      Trigger = 'monitoring alert PRN-001'
      Precheck = 'Get-Service Spooler; Get-WinEvent -FilterHashtable @{LogName="System";Id=7031}'
      Action = 'Restart-Service Spooler -WhatIf, then without -WhatIf'
      Verify = 'Get-Service Spooler | Where-Object Status -eq Running'
      Escalate = 'if it fails twice in an hour, raise to platform team'
    }
    $runbook | Format-List

    What happens

    This prints one runbook as a list. Precheck includes both the service state and the specific event id, because the alert alone does not tell you which failure this is. Action shows -WhatIf before the real run. Verify is separate from Action deliberately. Escalate carries a numeric condition rather than a judgement call.

    Worth remembering

    • Five parts: trigger, pre-checks, action, verification, escalation.
    • Verification is separate from the action succeeding.
    • Write commands in full; do not assume the reader knows the cmdlet.
    • Escalation needs a measurable condition.

    Primary documentation

    Where-Object reference · Get-Service reference · Restart-Service reference

    Try it

    Illustrative output · values vary by environment

    1PS> $runbook = [pscustomobject]@{
    2PS> Name = 'Restart failed print spooler'
    3PS> Trigger = 'monitoring alert PRN-001'
    4PS> Precheck = 'Get-Service Spooler; Get-WinEvent -FilterHashtable @{LogName="System";Id=7031}'
    5PS> Action = 'Restart-Service Spooler -WhatIf, then without -WhatIf'
    6PS> Verify = 'Get-Service Spooler | Where-Object Status -eq Running'
    7PS> Escalate = 'if it fails twice in an hour, raise to platform team'
    8PS> }
    9PS> $runbook | Format-List

    Example reader

    Inspect this page's example

    Fictional browser data when supported

    Run supported commands against the practice data, or inspect the example without executing it.

    Ready to inspect.

      Prints a complete runbook with all five sections.

      Show reference output

      Illustrative output · values vary by environment

      Name     : Restart failed print spooler
      Trigger  : monitoring alert PRN-001
      Precheck : Get-Service Spooler; Get-WinEvent -FilterHashtable @{LogName="System";Id=7031}
      Action   : Restart-Service Spooler -WhatIf, then without -WhatIf
      Verify   : Get-Service Spooler | Where-Object Status -eq Running
      Escalate : if it fails twice in an hour, raise to platform team

      Practise this

      Export an incident report

      Export Id, LevelDisplayName and Message for Error events to C:\Lab\report.csv.

      Review the lesson

      Keep going

      Next: Choose a PowerShell path for the work you do
      Genie guidePowerShell Operational runbooks

      A runbook is a procedure someone can follow at three in the morning without knowing what you know. Written well, it is also the thing you automate from.