Go + Concurrency: Fluency + ship pulse

Fifth phase. Go fluency + concurrency patterns at the principle level. The language Year 4 llm-gateway and Year 5 platform-ctl are written in. ~5 weeks, ~60 hrs.

Phase 4 gave you Python fluency for ops work; Phase 5 gives you Go fluency for systems work. The two languages aren’t redundant — they have non-overlapping strengths, and the program ships projects in both for that exact reason. What unifies them in Year 1 is the same discipline: fluency, not mastery, plus a real shipped artifact.

The deeper reason Go matters here is that almost every cloud-native control plane you’ll operate from Phase 7 onward is written in Go — Kubernetes, ArgoCD, Prometheus, Terraform, Crossplane, Helm, etcd. Reading that source code is the path to truly understanding those systems, and reading it requires Go fluency you don’t have yet.

pulse is the artifact, but the real lesson is the concurrency model — goroutines + channels + context implement backpressure and fault-isolation patterns that survive whatever language replaces Go in 2035.

---

Prerequisites

• Phase 4 complete — Python fluent, rxp shipped
• You accept: fluency + concurrency patterns. Go’s concurrency model is the real lesson. Ship pulse as proof of fluency.

---

Why this phase exists

Several ROOT projects are written in Go — pulse (this phase), triage backend (Phase 7), platform-ctl (Year 2-5), llm-gateway core (Year 4), mlship CLI core (Year 5). Go’s concurrency primitives (goroutines, channels, contexts) implement patterns that survive the language: bounded parallelism, fault isolation, structured concurrency.

This phase teaches you those patterns by building pulse, an infrastructure scanner — a real concurrent IO-bound tool you’ll use to monitor your homelab.

Go is the lingua franca of platform engineering — read the source

The single most valuable habit this phase builds is “I can read Kubernetes source.” Once you can, every later phase that touches K8s (Phase 7, Year 2’s mesh, Year 4’s KServe + Kubeflow, Year 5’s agents-on-K8s) gets dramatically less mysterious. The 30-minute “read ArgoCD’s main loop” item in the operational checklist is the highest-leverage 30 minutes of the entire phase.

---

1. PROBLEM

You want to write systems software: small daemons, CLIs, and services with predictable performance, low memory overhead, and concurrent IO. Python’s GIL makes this awkward; Go was built for it.

But Go is also the language of platform engineering — Kubernetes, ArgoCD, Prometheus, Terraform, Crossplane, Helm — all written in Go. Reading their source is the path to deep platform understanding.

---

2. PRINCIPLES

2.1 Goroutines + channels + context (the Go concurrency model)

Goroutines are M:N green threads. Channels are typed message queues. Context propagates cancellation + deadlines through the call tree.

Investigate:

• Spin up 10,000 goroutines; observe memory + CPU
• Build a fan-out / fan-in pipeline with channels
• Write a context-aware HTTP handler that cancels downstream calls when client disconnects

The “10,000 goroutines” exercise is deliberate culture shock if you’re coming from Python threads — 10,000 OS threads will exhaust your RAM, but 10,000 goroutines barely register. That’s not a benchmark to brag about; it’s the proof that Go’s concurrency model is qualitatively different, which is why every later concurrency pattern in this phase makes sense.

2.2 Bounded parallelism (worker pool)

Unbounded concurrency = OOM + downstream-overload. Worker pools cap concurrency.

→ Pattern: backpressure

Investigate:

• Implement a worker pool with golang.org/x/sync/errgroup + a semaphore
• Why is “spawn a goroutine per request” wrong? When is it OK?

This is the same pattern that limits Kubernetes scheduler concurrency, that bounds Prometheus scrape parallelism, and that you’ll explicitly configure on every consumer in Year 3’s stream-processing tier. The worker-pool implementation in pulse is your first concrete encounter with it.

2.3 Fault isolation

Each goroutine should fail without taking the program down. Panics are recoverable; errors are values.

→ Pattern: fault-isolation

Investigate:

• Why does Go return errors instead of throwing exceptions?
• Build a worker pool where one worker’s panic doesn’t kill the others
• Read golang.org/x/sync/errgroup source — it’s tiny; the pattern is the lesson

2.4 Idiomatic Go (the small set that matters)

Stdlib first. Small interfaces. Errors as values. No over-engineering. gofmt is the law.

Investigate:

• Read “Effective Go” + the “Code Review Comments” wiki
• Compare a 10-file Go service to a 10-file Python one. What’s different about the file/folder structure?

2.5 Testing + benchmarks (table-driven, idiomatic)

Go’s testing package + table-driven tests + *testing.T.Run for sub-tests + *testing.B for benchmarks.

Investigate:

• Write a table-driven test for a probe function
• Write a benchmark; use pprof to find the hot path

---

3. TRADE-OFFS

Decision	Option A	Option B	Cost
CLI lib	cobra (kubectl-style)	urfave/cli	flag (stdlib)
Logging	slog (stdlib, since 1.21)	zap (Uber)	zerolog
Config	viper (cobra companion)	env vars + flag	viper for sprawling configs; env+flag for tiny CLIs
Concurrency primitives	channels	sync.Mutex	atomic

The last row deserves a beat — the Go community’s “share memory by communicating” mantra is aspirational, not absolute. A sync.Mutex around a shared map is the right tool half the time. The pattern lesson is “pick the smallest primitive that fits the problem”, not “always use channels.”

---

4. TOOLS (as of 2025-10)

• Go 1.23+
• cobra + viper (CLIs)
• slog (stdlib structured logging)
• golangci-lint (linter)
• pprof (profiler — built in)
• errgroup, singleflight (golang.org/x/sync)
• prometheus/client_golang (metrics)
• goreleaser (multi-platform binary releases)

---

5. MASTERY

5.1 Reading list

Required	Why
”The Go Programming Language” (Donovan & Kernighan) Ch. 1-9	The book
Go Tour (online) — quick warm-up	Syntax familiarity
”Effective Go” (golang.org)	Style + patterns
”Concurrency in Go” (Cox-Buday) Ch. 1-4	Concurrency depth

5.2 Operational depth checklist

[ ] Set up a Go project with go.mod + cobra + slog + golangci-lint
[ ] Build a worker pool with errgroup + semaphore; benchmark; observe with pprof
[ ] Implement context-aware shutdown (signal handler → cancel context → wait for goroutines to drain)
[ ] Write a small HTTP server with stdlib net/http; add structured logging via slog
[ ] Expose Prometheus metrics; scrape with curl
[ ] Use pprof to profile a CPU-bound function; identify the hot path; optimize
[ ] Use the race detector (go test -race) on concurrent code; force a real race; observe the report
[ ] Set up GoReleaser for multi-platform binary releases; tag v0.1.0; verify GitHub release
[ ] Read 1 hour of Kubernetes source (cmd/kubelet or pkg/scheduler); note 3 patterns
[ ] Read ArgoCD's main loop (controller/argocd-application-controller) for 30 min

The “context-aware shutdown” item is the operational pattern you’ll re-implement five more times across the program — every long-running Go service in basecamp needs this. SIGTERM → cancel context → goroutines drain → bounded grace period → exit. Get it right once in pulse and copy the shape forever after.

5.3 Project: pulse

A concurrent infrastructure scanner. Probes N hosts (ICMP/TCP/HTTP), exposes Prometheus metrics, exports JSON.

pulse scan --targets ./hosts.txt --workers 50
# → concurrent ICMP + TCP + HTTP probes; output JSON

pulse serve --port 9100 --targets ./hosts.txt
# → long-running mode; exposes /metrics for Prometheus

pulse check https://api.example.com --timeout 5s
# → one-shot check

pulse deliverables:

• cobra CLI (scan, serve, check)
• Probe types: ICMP, TCP, HTTP (status + content match)
• Concurrency: worker pool, bounded by --workers
• Outputs: JSON, YAML, Prometheus exposition format
• Per-probe timeout (context-driven)
• 80%+ test coverage; race-detector-clean
• GoReleaser multi-platform binaries
• README + examples
• Quiet ship — same as rxp/konfig. No launch energy.

pulse integrates back into the platform: from Year 2 onward, basecamp’s monitoring stack scrapes its /metrics endpoint. It’s not just a fluency artifact — it’s a real homelab probe. By Phase 7 you’ll deploy it onto K3s alongside triage, and by Year 3 it’ll be one of the first inputs to your observability tier.

See projects/pulse/plan.

---

6. COMPARE: Same probe, three concurrency models

Same workload (probe 100 hosts), three implementations:

• Python asyncio
• Python threads + queue
• Go goroutines + channels

Compare: code complexity, throughput, memory, ergonomics under failure.

300 words.

This is the COMPARE step that proves the concurrency pattern transferred — three implementations, the same underlying problem (bounded fan-out with timeouts and per-task error handling). If you can articulate “the worker pool is the same shape in all three; the syntax and runtime support differ”, you’ve internalized the pattern. If you can only articulate “Go is faster”, you haven’t.

---

7. OPERATE

• 2+ runbooks (Go service deploy, pprof workflow)
• Weekly log

---

8. CONTRIBUTE

Approachable Go OSS: cobra docs, slog examples, kubectl plugins, prometheus client_golang, any small CNCF tool.

---

Validation criteria

[ ] All 10 operational depth checks
[ ] `pulse` shipped to GitHub + GoReleaser binaries published
[ ] Three-concurrency-models comparison written up
[ ] 2+ runbooks; 5+ weekly log entries
[ ] Pattern entries deepened:
    - backpressure → OUTLINE (worker-pool as concrete example)
    - fault-isolation → OUTLINE (per-probe timeout + context cancellation)
[ ] Exit Test passed

---

Exit Test

Time: 2 hours.

1. Build (60 min) — write a small Go service from scratch: a health-aggregator that probes N HTTP endpoints concurrently with a worker pool, exposes per-target Prometheus gauges. Race-detector-clean.
2. Articulate (60 min) — 600 words: “Why does Go have channels when locks already exist? When do you reach for each?”

---

Anti-patterns

Anti-pattern	Why
Spawning a goroutine per request unbounded	OOM + downstream overload
Catching panics broadly	Hides bugs; recover only at goroutine boundaries
Shared mutable state without -race testing	Subtle races hit at 3am, not in dev
Over-using channels for everything	Channels for communication, mutex for shared state
Loud-launching pulse	Quiet ship. Reserve launch energy.

Run go test -race in CI from day one

The “race” anti-pattern is the most expensive bug class Go has. A race won’t fail your tests reliably — it’ll fail under production load, at 3am, in a way that looks like a corrupted database or a flaky network. The cost of -race is roughly 2-3x test time. The cost of not -race-ing is a sleepless on-call rotation. Wire it into pulse CI in week 1 and don’t think about it again.

---

Patterns touched this phase

• backpressure — OUTLINE
• fault-isolation — OUTLINE

---

→ Next: Phase 6: Containers