Learn / Go Service Engineering
Deploys, Health, and Incident Runbooks
Ship Go services with health endpoints, graceful shutdown, smoke checks, and incident notes a teammate can follow.
Course: Go Service Engineering. Level: Intermediate. Topic: Reliable services.
Stage: pro - Pro service ownership - Deploys, health checks, and incident runbooks. Finish the Go path with deploy smoke checks, readiness gates, rollback signals, and incident response notes.
Outcomes
- Split liveness and readiness endpoints.
- Drain HTTP and worker work during shutdown.
- Write smoke, rollback, and incident review steps.
Concepts
- health endpoint
- graceful shutdown
- signal handling
- incident review
Concept flow
Show how deploys, health checks, and incident runbooks moves from trigger to implementation outcome in Go Services.
- Release
- Readiness probe
- Smoke request
- Shutdown signal
- Incident note
Session flow
- Model health endpoint (concept, 12 min) — Name the decisions behind health endpoint before writing code.
- Split liveness and readiness endpoints.
- Explain where health endpoint belongs in ticket processing platform.
- Build the vertical slice (walkthrough, 22 min) — Implement the smallest useful slice in cmd/api/shutdown.go.
- Drain HTTP and worker work during shutdown.
- Connect graceful shutdown to a working example.
- Verify and harden (exercise, 15 min) — Write incident review bullets for a failed deploy.
- Write smoke, rollback, and incident review steps.
- Record one risk or follow-up before moving on.
Code example
Go in cmd/api/shutdown.go.
package main
import (
"context"
"log/slog"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
func run(server *http.Server) error {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
errs := make(chan error, 1)
go func() {
errs <- server.ListenAndServe()
}()
select {
case err := <-errs:
if err == http.ErrServerClosed {
return nil
}
return err
case <-ctx.Done():
shutdownCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := server.Shutdown(shutdownCtx); err != nil {
slog.Error("server shutdown failed", "error", err)
return err
}
return nil
}
}
Walkthrough examples
- Deploys, Health, and Incident Runbooks in a ticket processing platform — A team is extending a small Go service plus worker and needs this lesson's pattern to be clear enough for review, testing, and future maintenance.
- File: cmd/api/shutdown.go
- File: tests/deploys-health-and-incident-runbooks.spec
- File: docs/go-services/deploys-health-and-incident-runbooks.md
- Start from the provided Go snippet and make the intent visible in names and boundaries.
- Apply the checklist item "Readiness reflects dependencies" before adding extra behavior.
- Write down how the implementation changes when graceful shutdown fails or becomes slow.
Practice
- Add separate live and ready handlers.
- Prove shutdown finishes in a test or local script.
- Write incident review bullets for a failed deploy.
Checklist
- Readiness reflects dependencies
- Shutdown has a timeout
- Smoke checks hit a real route
- Incident notes include follow-up owner
Quiz prompts
- Why should graceful shutdown use a timeout? — A bounded shutdown protects users and deploy automation at the same time.
- A teammate wants to hide health endpoint inside a convenient helper. What should you check first? — Place health endpoint at the boundary that keeps ticket processing platform behavior explicit, testable, and reviewable.
- Which artifact best proves this Go Services lesson is ready for review? — Production-ready learning needs evidence: a test, trace, command, screenshot, or log that catches the risk again.
- Pro service ownership: a teammate says the happy path works, but "Health and readiness gates" is still implicit. What should you ask for before merging? — Health and readiness gates belongs in the pro stage only when the decision is visible, testable, and tied to a realistic failure mode.
- A reviewer has five minutes to evaluate this pro Go Services slice. Which evidence is strongest? — Produce a pro Go service ownership packet: readiness gates, deploy smoke commands, rollback thresholds, and an incident runbook.
Flashcards
- Pro service ownership: what decision does "Health and readiness gates" force you to make? Separate process health from dependency readiness and worker backlog pressure. Evidence prompt: Add a readiness output that reports database and worker queue state.
- Pro service ownership: what decision does "Deploy rollback thresholds" force you to make? Define the error-rate, latency, readiness, and worker signals that stop a release. Evidence prompt: Write a deploy checklist with exact commands and threshold values.
- Pro service ownership: what decision does "Incident runbook" force you to make? Give the next responder symptoms, first checks, mitigation steps, and follow-up evidence. Evidence prompt: Draft one incident note for a stuck worker queue or dependency timeout.
- In Go Services, what should you remember about health endpoint? health endpoint matters here because it supports "Split liveness and readiness endpoints.".
- In Go Services, what should you remember about graceful shutdown? graceful shutdown matters here because it supports "Drain HTTP and worker work during shutdown.".
- In Go Services, what should you remember about signal handling? signal handling matters here because it supports "Write smoke, rollback, and incident review steps.".
- In Go Services, what should you remember about incident review? incident review matters here because it supports "Split liveness and readiness endpoints.".
Labs
- Ship a deploys, health, and incident runbooks slice — Extend a small Go service plus worker with a small but reviewable feature that proves the lesson's architecture in code.
- Produce a pro Go service ownership packet: readiness gates, deploy smoke commands, rollback thresholds, and an incident runbook.
- Add a readiness output that reports database and worker queue state.
- Write a deploy checklist with exact commands and threshold values.
- Draft one incident note for a stuck worker queue or dependency timeout.
- Add separate live and ready handlers.
- Prove shutdown finishes in a test or local script.
- The lab demonstrates the pro service ownership outcome without skipping earlier contract evidence.
- Each subtopic has a concrete artifact: Health and readiness gates, Deploy rollback thresholds, Incident runbook.
- The review notes explain how this level changes ownership, verification, or operational risk.
Challenge
- Review-ready deploys, health, and incident runbooks (Capstone) — Produce a pro Go service ownership packet: readiness gates, deploy smoke commands, rollback thresholds, and an incident runbook.
- All checklist items are either implemented or documented with a reason.
- The change can be understood from cmd/api/shutdown.go plus one short note.
- The concept diagram names ownership, failure handling, and verification points.
- A teammate could run the verification steps without asking for hidden context.
Canonical lesson URL