A custom OpenTelemetry Collector for 10,000 laptops.

context

The fleet’s SLOs needed eyes.

The fleet control loop only works if the telemetry underneath it is trustworthy. We needed host and compliance signals from every laptop in the company, and the usual answer — a pile of vendor agents, each with its own format, footprint, and bill — is how endpoint fleets end up running five overlapping daemons. OpenTelemetry offered the alternative: one collector, one pipeline, open schema, no lock-in on the backend.

constraints

The watcher must not hurt the watched.

A collector on a developer’s laptop lives under a resource budget — if the telemetry agent is the thing making the fans spin, you lose the room. macOS adds its own tax: everything must be signed and notarized, or Gatekeeper ends the conversation. And the hardest constraint is recursive: updating the agent that monitors the fleet, across the fleet, without blinding yourself while you do it.

design

A minimal distribution, built like software.

The OpenTelemetry Collector Builder (ocb) lets you compile a collector containing exactly the components you need and nothing else. Ours carried the host-metrics receiver, a small set of processors for batching and memory protection, and the Splunk exporter — a deliberately boring manifest. The manifest lived in git; CI built, signed, and packaged it; Puppet rolled it out cohort by cohort, watching the same SLIs the collector itself reports. When the telemetry from a canary cohort stays healthy, the wave continues.

the darwin gap

The metrics that weren’t there.

Rolling this out surfaced an uncomfortable fact: the Collector’s host-metrics receiver couldn’t report CPU or disk metrics on macOS — the scrapers needed CGO and simply weren’t supported on Darwin. On a fleet that was heavily macOS, that’s not a footnote; it’s a hole in the middle of the health model.

The fork was the tempting answer — patch it privately, ship it, move on. But a private fork of a telemetry agent is a maintenance debt you pay forever, on every upstream release. So I took it upstream instead: I authored the proposal (opentelemetry-collector-contrib #33393) making the case for Darwin CPU/disk support with CGO, pushed for it internally and in the issue, and dug into the receiver’s code to understand what the fix required. The implementation PR that landed was authored by the component’s maintainers — the feature ships in the Collector today, and I’ll take that outcome. Upstream is a contact sport: an earlier PR of mine to Installomator, the Mac-admin install framework, got beaten to the merge by a parallel fix after ten commits of work. You file anyway. The commons is how you delete work forever.

$ otelcol query host.cpu.utilization   # 2024, darwin
no data — scraper unsupported on this platform
$ otelcol query host.cpu.utilization   # after #33393
host.cpu.utilization  0.42  {cpu="0", state="user"} ✓
$ 

lessons

What I’d tell the next fleet.

• Upstream-first beats forever-fork. Filing the issue was a one-time cost; a private fork would have been a bill due on every upstream release, forever.
• Minimal distributions win. Every component you don’t compile in is attack surface, memory, and upgrade risk you don’t carry.
• Collector config is code. The manifest and config lived in git and went through CI like everything else — telemetry outages from hand-edited configs are self-inflicted.

outcomes

One pipeline, all the way down.

The distribution ran on 10,000+ endpoints and fed every SLI in the fleet control loop. macOS host metrics now ship in the upstream Collector for everyone. The same discipline runs this very website — live SLOs, real-user vitals, and an error budget you can inspect — and carried into the security product I built on my own time: Copywarden →