Automating a 10,000-device fleet until it was boring.

context

Laptops are production too.

Splunk’s endpoint fleet was 10,000+ devices across macOS, Windows, and Linux — the machines the people who build production use to build production. If an engineer’s laptop is out of security policy, that’s not an IT inconvenience; it’s attack surface, and it’s an audit finding. The fleet sat squarely inside Splunk’s SOC 2, ISO 27001, and UK Cyber Essentials scope, which means every compliance claim we made had to be provable, continuously, with evidence.

constraints

A distributed system with the worst properties.

A laptop fleet is a distributed system that breaks most of the assumptions server-side tooling gets to make. Nodes are offline whenever a lid closes. Networking is whatever coffee-shop Wi-Fi the machine happens to be on. And every node has a human with admin rights and a strong opinion about their machine rebooting during a demo. On top of that: zero downtime was a hard requirement — there is no maintenance window for someone’s laptop — and compliance had to be continuously provable rather than reconstructed by hand at audit time.

design — observability first

Define “healthy” before automating anything.

The first thing we built wasn’t automation — it was the definition of done. We wrote SLIs for what a compliant, healthy device actually means, as measurable signals: patch currency, security-agent liveness, configuration drift, enforcement latency. Telemetry flowed from a custom OpenTelemetry Collector on every device into Splunk, where SLO-backed alerting replaced the ticket queue. The difference matters: a ticket says “something looked wrong once”; an SLO says “this is trending toward breach in N hours, act now.”

design — remediation

Hours to minutes, without breaking anyone’s demo.

Enforcement used to mean polling: a node checks in eventually, notices it’s out of policy eventually, converges eventually. “Eventually” was hours. We rebuilt it event-driven: an SLO breach or an emergency policy push (think P0/P1 incident response) triggers Python remediation services directly against the configuration layer, so convergence starts on the event instead of on the next check-in.

That got enforcement under 15 minutes, and 15 minutes was not an accident — it was the management platform’s check-in interval, the floor the architecture imposed. Security wanted three. Closing that gap meant getting off the check-in cycle entirely, so I started moving detection onto the endpoint itself with local polling daemons that could act without waiting to be asked. That work was still in flight when the acquisition redirected the roadmap, but the shape of the answer was clear: if your latency target is shorter than your control plane’s heartbeat, the agent has to decide locally.

The fleet was global; the rollouts deliberately weren’t. A new OS update went first to an opt-in beta group for UAT; once they signed off, the forced rollout moved in waves targeted by IP range — addresses are allocated geographically, so an IP range is effectively a region. Australia went first and had to come back clean before the next region started, which meant every wave got a full business day of soak from real users in one timezone rather than a thin scatter of signal across all of them. Each wave applied while machines were idle, so nobody lost work mid-task. A bad change could reach one region; it could never reach all ten thousand machines at once.

Along the way we retired a pile of legacy Perl and shell scripts into Python running in GitLab CI — not for the aesthetics, but because the failure rate of the old automation was itself a reliability problem.

1. beta      opt-in UAT group — sign-off required to proceed
2. wave      target by IP range → region · apply while idle
3. soak      one full business day, real users, one timezone
4. gate      SLIs clean? yes → next region · no → stop here
↻  repeat until the fleet is current — blast radius stays one region

the hard part

Trust is the actual constraint.

The technical problems here are the easy half. The hard half is that every node in this distributed system belongs to a person who can disable your agent, defer your patch, and file a ticket about you. Automation that fights its users loses, no matter how correct the control loop is.

I inherited the proof. Patching ran on a legacy Perl script that asked people to quit their applications and, failing that, quit the applications for them — including while they were presenting on a video call. It technically worked. It also guaranteed that every patch cycle produced a fresh round of angry tickets, and the political cost of that made the whole compliance program harder to run.

The replacement was a Python patching service that inverted the model: stage the update in the background, then apply it the next time the user relaunches the application. Nothing is interrupted, because nothing has to be. On top of that it checks whether someone is in a meeting and simply waits if they are. The engineering isn’t exotic — background staging, apply-on-relaunch, typed error handling around every failure path — but the result was 97–99% patch compliance fleet-wide, which in practice meant every machine that came online got patched. The remainder was hardware that never checked in — spares, machines in transit, boxes sitting powered off — not devices the system had failed to reach. The recurring audit findings for stale software stopped.

The same principle covered the always-on security agents. They had to be running at all times, so a machine whose agent had died was both a compliance gap and a support ticket waiting to happen. Rather than chase those by hand, I made the two management planes able to repair each other: if one agent went down, the other could reinstall it, and vice versa. A dead agent became a self-correcting event instead of an escalation.

outcomes

The fleet got boring. That was the point.

• Security state-change enforcement: hours → under 15 minutes (team goal ~3; emergency P0/P1 rollouts included).
• 97–99% macOS patch compliance across 10,000+ endpoints, sustained — effectively every device that came online.
• Zero-downtime updates as the norm; compliance evidence generated continuously for SOC 2, ISO 27001, and UK Cyber Essentials audits.
• Legacy Perl/shell automation retired into tested Python services in CI.

The telemetry layer that made all of this possible has its own story — including the hole I found in the OpenTelemetry Collector on macOS: a custom OTel Collector, fleet-wide →