Notes from the field

Writing

DevOps, SRE, and platform engineering — mostly the unglamorous decisions that quietly save money or prevent outages.

Adventures in S3 Cost Optimization

Digging in to S3 to find some spare change.

Fish Stick: A Stateless Incident Management Bot for Slack

No database. No web UI. Just Slack. A new take on incident management that sits in the sweet spot between too simple and too complex.

AI Layoffs

Paycom laid off a bunch of people, blamed AI.

Monitor Available IPs with Lambda and CloudWatch

Building a custom monitoring solution for tracking VPC subnet IP availability in AWS EKS clusters when off-the-shelf tooling doesn't exist.

Do NOT use wildcard alternate domains in AWS CloudFront

A CloudFront gotcha: wildcard alternate domains silently override your DNS, routing traffic to the most specific name match across all your distributions regardless of what the CNAME says.

Adventures in tuning unicorn for Kubernetes

Hard-won notes on tuning a Ruby on Rails app running Unicorn under Kubernetes in production: resource config, autoscaling behavior, and the throughput dead-ends the sparse #kubernetes-ruby community won't warn you about.

You probably shouldn’t be using Kubernetes for your new startup

Kubernetes is great and also the wrong default for most new companies: its hidden complexity is binary (trivially easy or molecular-level hard), and a young product rarely has the k8s-shaped problems to justify the cost.

How to Live-Rotate PostgreSQL Credentials

A practical guide to rotating database credentials without downtime, navigating PostgreSQL's confusing role system along the way.

Kubernetes EC2 autoscaling for fun and profit

A distributed-systems debugging story: an async-worker failure a few weeks after migrating to AWS EKS turned into a lesson about how EC2 autoscaling really behaves under Kubernetes.

What I Love About SRE

A childhood soaked in science, and how site reliability engineering scratches the same itch: 'What do we think is true, and how would we know we're wrong?'

Hype Isn't a Use Case

Chasing the newest tech you read about this morning is how production ends up on fire. For the stuff that has to stay up, pick the boring tools you can count on.

Design for Recovery, Not Prevention

There are two ways to read 'how do we make sure this never happens again?' At cloud scale, stop chasing 100% prevention and design for fast recovery instead.

Empathy in Dev and Ops

The 'WTF was wrong with whoever wrote this?' reflex is cheap and unfair. The useful work is figuring out the why behind the bad code.

An Appeal for Simple Infrastructure Architectures

Every tool you bolt on isn't one dependency, it's a family of them. Pride in an architecture should come from how little you use, not how much.

The Dreyfus Model of Ops Engineering

The arc of an ops engineer: from crayon drawings, to Rube Goldberg machines, to spending more time removing boxes from the diagram than adding them.