IaC: Terraform + Crossplane

Second phase of Year 2. Two implementations of declarative infrastructure with reconciliation loops. Ship terralabs publicly — your first launched OSS artifact. ~8 weeks, ~80 hrs.

---

Phase 8 made you accept that distributed systems are uncertain. Phase 9 makes you accept that infrastructure — the substrate the distributed systems run on — has to be expressed in code that converges, not in clicks that drift. Terraform and Crossplane are two answers to the same question; this phase asks you to learn both well enough to swap between them based on the team and the workload, not the resume bullet.

The phase also marks a shift in relationship with the broader ecosystem. terralabs ships publicly — your first launched OSS artifact, and the moment you stop being someone-with-a-homelab and start being someone-who-ships. The differentiator isn’t another Terraform module repo; it’s two implementations of the same shape (TF modules and Crossplane Compositions for VPC + cluster + DB) sitting in one repo as a teaching artifact. That’s the side-by-side proof that you’re reasoning in patterns, not memorizing tools.

By phase end, basecamp’s homelab K3s is provisioned by terralabs/terraform/modules/proxmox-k3s-cluster and the workloads on it are still managed by ArgoCD. That’s the first time the platform stack is self-coherent: terralabs makes the substrate, basecamp deploys onto it, and both live in git.

Two tools, one pattern — that’s the whole point

If you ship a terralabs that has TF modules but no Crossplane Compositions, you’ve shipped a portfolio piece. If you ship one with both for the same shape, you’ve shipped a teaching artifact — the kind of thing that demonstrates pattern-fluency over tool-fluency at a glance. That’s the differentiator. Don’t skip the Crossplane half thinking “I’ll do it later.” Later is what makes it ordinary.

---

Prerequisites

• Phase 8 complete — distributed-systems theory internalized
• You accept: Terraform and Crossplane are two implementations of the same pattern (declarative infrastructure with reconciliation). The pattern is durable; the tools rotate every 5 years.

---

Why this phase exists

basecamp deploys workloads onto K8s clusters. Those clusters need to exist first — and the VMs/networks/cloud resources they run on. That’s IaC.

Terraform is the industry-standard imperative-feeling DSL with declarative semantics. Crossplane is the K8s-native take: cloud resources as Custom Resources, reconciled by controllers — IaC and workload deployment in the same control plane.

Both implement the same pattern: declarative-vs-imperative-infrastructure. Both demonstrate control-loops you saw in Year 1’s K8s. terralabs ships both implementations side by side — that side-by-side teaching artifact is its differentiator.

---

1. PROBLEM

You want to provision cloud + on-prem infrastructure (VPCs, subnets, K8s clusters, databases, object storage) in a way that’s:

• Declarative — say what, not how.
• Reproducible — same code, same infrastructure, anywhere.
• Reviewable — diffs in a PR before they hit production.
• Reconciling — the world drifts; the IaC tool pulls it back.

Terraform solves this with HCL + provider plugins + state files. Crossplane solves it with K8s Custom Resources + reconciler controllers + Compositions. Same problem category — managed convergence to declared desired state — two stacks, two failure modes, two operational surfaces.

---

2. PRINCIPLES

2.1 Declarative resources + reconciliation

Both tools work the same way: declare desired state; tool diffs against actual state; tool applies the diff.

→ Pattern: declarative-vs-imperative-infrastructure → Pattern: control-loops

Investigate:

• Read Terraform’s plan/apply lifecycle in the docs.
• Read one Crossplane Composition; trace how a CompositeResource produces ManagedResources.
• Both have a “drift detection + reconcile” loop. Where does each store actual-state? (TF: state file. Crossplane: K8s API.)

2.2 State management + drift

Terraform: state file (local, S3, or Terraform Cloud). Crossplane: K8s etcd. Both must be authoritative + recoverable.

Investigate:

• TF state corruption — what does it look like? How do you recover? (Hint: S3 versioning + DynamoDB lock + terraform state pull/push.)
• Crossplane reconciliation loop interval; what happens if you kubectl edit a managed resource? (It reverts on the next reconcile — drift detection is automatic, not manual.)

2.3 Modules + Compositions

Reusable units. TF modules are functions over HCL. Crossplane Compositions are templates over CR + ManagedResources.

→ Pattern: layering-and-abstraction (revisited)

Investigate:

• Build a TF module: aws-vpc that takes CIDR + region, outputs subnet IDs.
• Build a Crossplane Composition: eks-with-rds that produces an EKS cluster + an RDS Postgres.
• Why do both have a “minimum viable interface” rule? (A module with 40 inputs is a leaky abstraction; a module with 4 inputs that sensibly defaults the rest is a real one.)

2.4 GitOps for IaC

Same pattern as basecamp’s app GitOps: PRs proposed; CI runs plan; reviewer approves; apply runs in CI.

→ Pattern: gitops (reinforced)

Investigate:

• Set up Atlantis or tf-cd for PR-based TF apply.
• Compare with Crossplane’s flow: just kubectl apply (or basecamp’s ArgoCD reconciliation) — no separate IaC pipeline. That’s the structural payoff: one control loop, one PR review process, one observability surface.

2.5 Provider hierarchy + immutability

Cloud-side immutability: most managed resources can’t be edited; create + replace is the norm.

→ Pattern: immutable-infrastructure

Investigate:

• Why does TF replace an EC2 instance when you change AMI? What’s lifecycle { create_before_destroy = true }?
• How does Crossplane handle “immutable fields” on a managed resource? (It surfaces an error in the resource status; you delete + recreate, or you use a Composition that does it for you.)

---

3. TRADE-OFFS

Decision	Terraform	Crossplane	When
Mental model	DSL outside K8s	K8s CRs inside cluster	TF for “ops team owns infra”; Crossplane for “platform team owns infra-as-app”
State	state file	K8s etcd	TF: simple but separate. CP: integrated but K8s-shaped
Lifecycle	plan/apply	reconcile loop	TF: explicit. CP: continuous
Drift handling	manual terraform apply	automatic re-reconcile	CP wins for “always converge”
Ecosystem	huge (1000+ providers)	growing	TF wins for breadth
Multi-cloud abstraction	per-provider	Compositions can abstract clouds	CP wins for “same shape across clouds”
OpenTofu	the post-license-change fork	n/a	If you want to keep the OSS pure, OpenTofu

Pick by team shape, not by hype

Crossplane is structurally elegant — one reconciler model, one observability surface, one PR review process for both apps and infra. Terraform is structurally pragmatic — every cloud provider lands a TF provider before they land anything else, the ecosystem is enormous, and most engineers can already read it. The decision usually isn’t “which is better” but “which does my org’s existing skill set already cover?” terralabs ships both so you can answer that honestly in either direction when the question lands in an interview.

---

4. TOOLS (as of Q1 2026)

• Terraform 1.9+ OR OpenTofu 1.8+ (the OSS fork; same HCL)
• Crossplane 1.18+
• tflint — linter
• tfsec / checkov / trivy config — security scanning
• terragrunt — TF wrapper for DRY environments (optional)
• atlantis — PR-based TF apply (optional)
• AWS CLI / GCP CLI — backend auth + state buckets

---

5. MASTERY

5.1 Reading list

Required	Why
Terraform Up & Running (Yevgeniy Brikman, 4th ed.) Ch. 1-7	The book
Crossplane docs — Concepts + Compositions	The K8s-native shape
HashiCorp’s “Module style guide”	Idioms that compound

Recommended	Why
GitOps Days talks on Crossplane	Real-world patterns
Upbound’s blog	Crossplane patterns

5.2 Operational depth checklist

[ ] Set up TF backend on S3 (with DynamoDB lock); migrate from local state
[ ] Build TF modules: aws-vpc, aws-eks, aws-rds-postgres, aws-s3-secure-bucket
[ ] Build TF module: proxmox-k3s-cluster (for the homelab)
[ ] Add CI: tflint + tfsec + terraform fmt + terraform plan on PR
[ ] Install Crossplane on basecamp's K3s
[ ] Build a Crossplane Composition: eks-with-rds; deploy via basecamp ArgoCD
[ ] Compare hands-on: same outcome (EKS + RDS) via TF and Crossplane; document trade-offs
[ ] Configure drift detection: edit a TF-managed VPC manually; observe `terraform plan` diff
[ ] Configure Crossplane to detect drift: edit a CP-managed resource manually; observe re-reconcile
[ ] Recover from a corrupted TF state (simulate; restore from backup)

5.3 Project: terralabs (first public launch)

Ship terralabs to GitHub publicly. This is your first launched OSS artifact — the moment you stop being someone-with-a-homelab and start being someone-who-ships.

terralabs scope this phase:
  github.com/abukix/terralabs (PUBLIC from day 1)

  terraform/modules/
    aws-vpc, aws-eks, aws-rds-postgres, aws-s3-secure-bucket
    proxmox-k3s-cluster

  crossplane/compositions/
    eks-with-rds.yaml (one example to start)

  examples/
    end-to-end demos: "spin up EKS + RDS via TF" and "same via Crossplane"

  CI: terraform fmt + tflint + tfsec on PR
  README + architecture doc + per-module docs
  Tagged v0.1.0 release
  Blog post: "terralabs — declarative infra, two implementations" on abukix.dev/blog
  LinkedIn announcement

The differentiator: Terraform and Crossplane Compositions for the same shape, side-by-side, as a teaching artifact. That’s rare in OSS and genuinely useful for engineers learning both tools.

See the terralabs plan.

5.4 basecamp deploys onto terralabs

By phase end, basecamp’s homelab K3s cluster is provisioned by proxmox-k3s-cluster from terralabs. The chain is:

abukix/terralabs (TF)  →  Proxmox VMs + K3s cluster
abukix/basecamp (Argo) →  workloads on the cluster

This is the first time the platform stack is self-coherent: terralabs makes the substrate, basecamp deploys onto it. The homelab the substrate runs on is the same one specced in homelab/hardware — 32GB DDR5 by Year 2, with the substrate now declared in code instead of clicked into Proxmox by hand.

---

6. COMPARE: Terraform vs Crossplane (the writeup)

You did the hands-on. Now write 600 words: which would you reach for in three scenarios?

1. A 2-person startup with one cloud account.
2. A platform team in a 1000-person org with policy enforcement requirements.
3. A homelab building toward Year 5 capstone.

Different answers per scenario; defend each.

---

7. OPERATE

• 4+ runbooks (terraform-state-recovery, crossplane-comp-debug, tf-plan-review-checklist, iac-apply-rollback)
• 1+ ADR (e.g., “TF for cloud, Crossplane for K8s-native services in basecamp”)
• Weekly log

---

8. CONTRIBUTE

terralabs itself is your contribution. Plus: terraform-aws-modules ecosystem (huge community), Crossplane providers, OpenTofu CLI, tflint rules.

---

Validation criteria

[ ] All 10 operational depth checks
[ ] terralabs publicly launched: GitHub + README + blog post + LinkedIn
[ ] basecamp's homelab K3s now provisioned by terralabs
[ ] At least 1 Crossplane Composition working end-to-end
[ ] TF-vs-Crossplane writeup
[ ] 4+ runbooks; 1+ ADR; 8+ weekly log entries
[ ] Pattern entries deepened:
    - declarative-vs-imperative-infrastructure → DEEP
    - control-loops → reinforced (TF + Crossplane both implement)
    - gitops → reinforced (PR-based TF apply)
    - immutable-infrastructure → OUTLINE
[ ] Exit Test passed

---

Exit Test

Time: 3 hours.

1. Build (90 min) — given a fresh AWS account, use terralabs to provision: VPC + EKS + RDS Postgres + S3 bucket. Via TF first; same shape via Crossplane Composition second. Verify both deployments work.
2. Diagnose (60 min) — scenario from Phase 9 catalog (TF state corrupted; Crossplane Composition stuck; provider quota hit).
3. Articulate (30 min) — 600 words: “When does a platform team pick TF vs Crossplane? Defend with examples.”

---

Anti-patterns

Anti-pattern	Why
Manual edits to cloud resources	Drift; future apply will revert your fix or fail
Storing TF state in git	Plaintext credentials; concurrency bugs
Skipping terraform plan review	Plans catch the surprises before they’re real
--auto-approve on prod	Self-explanatory
Wrapping terraform apply in shell scripts	Atlantis / Terraform Cloud / GitHub Actions exist

---

Patterns deepened this phase

• declarative-vs-imperative-infrastructure → DEEP
• control-loops → reinforced
• gitops → reinforced
• immutable-infrastructure → OUTLINE

Browse the full category at patterns/infrastructure-and-platform/.

---

→ Next: Phase 10: AWS Deep Dive