A home screen renders 1.5 seconds slower. An alert fires. Four hours later, the cause is finally clear. Almost none of that time was spent computing — it was spent reconstructing relationships that already existed across five different tools with no shared vocabulary. Netflix published the architecture they built to solve this, and it’s worth reading carefully.
The Problem Was Never the Data
Netflix processes 2 billion requests per day and 38 million log events per second. They are not short on observability data. The issue was that none of it was connected. Applications, incidents, deployments, alerts, and infrastructure all lived in separate systems — Atlas for applications and time series, Spinnaker for deployments, Incident.io for incidents, PagerDuty for on-call, and Slack for everything the other tools missed. When an engineer got paged, their first hour wasn’t investigation. It was archaeology.
The standard AIOps response to this is better dashboards and smarter alerting. Netflix went a different direction: they built a knowledge graph that treats operational relationships as first-class data, not presentation-layer concerns.
The Graph: W3C Standards and a Custom Rust Database
The technical stack is built on W3C semantic web standards — RDF for representing facts as subject-predicate-object triples, OWL for defining the ontology, SHACL for validating writes, and SPARQL for querying across domains. For storage, Netflix built QuipuDB, a custom distributed quad store backed by Rust’s redb embedded key-value engine.
The data model is strict by design. Netflix refused generic Service entity types because every tool used the word differently. Instead, they created a single ops:Application type validated against a software catalog of roughly 49,000 entries. Any entity that can’t be verified against the catalog doesn’t make it into the graph. Dependency edges are capped bidirectionally so popular services don’t structurally dominate every query.
A simple triple looks like this: api-gateway ops:ownedBy “Team Bedrock”. At Netflix’s scale, these triples span 13 signal types: log errors, metric anomalies, slow traces, alerts, incidents, deployments, configuration changes, experiments, autoscaling events, commits, end-to-end bottlenecks, error patterns, and incident discussion threads. The last one matters more than it sounds.
AutoRCA: AI Agents That Talk Through the Graph
AutoRCA is Netflix’s agentic root-cause analysis system. It’s composed of specialist agents coordinated around an investigation question. The key design decision: the agents don’t message each other. They communicate through the graph. One agent writes findings about operational entities; others consume those typed triples without needing to understand upstream logic. The ontology is the interface.
This matters for anyone building multi-agent systems. Direct agent-to-agent communication creates tight coupling and schema drift. Using a shared typed vocabulary as the communication layer means agents can be replaced, upgraded, or added without breaking the others. Every hypothesis is also queryable after the fact.
The correlation engine analyzes all 13 signal types within a 10-minute temporal window, scores chains on proximity and severity, and filters anything below a 0.6 confidence threshold. Accepted chains retain their scores and supporting evidence. AutoRCA doesn’t resolve incidents autonomously — it surfaces ranked hypotheses with human approval gates on all remediation actions. That’s deliberate.
Turning LLM Output Into Deterministic Facts
LLMs are used throughout the pipeline — for summarization, fuzzy name matching, and hypothesis generation. But their outputs are constrained before reaching storage. Thread categories can only be incident, question, discussion, or decision. Sentiment values come from a closed enumeration. Any triple outside the contract is rejected.
This is the pattern worth stealing: use LLMs to interpret, use type systems to constrain. The result is non-deterministic model output converted into a deterministic, SPARQL-queryable fact store. This is what separates production AI systems from demos — the boundary conditions, not the model.
Human Reasoning as Data
When a Slack incident thread ends, most organizations lose everything discussed in it. Hypotheses tested, mitigations tried, theories rejected — gone. Netflix captures it as ordered triples and writes it back into the graph. The result, as one analysis of their system noted, is that “the ontology gradually stops being a schema and starts becoming a memory.”
This institutional knowledge powers the correlation engine for future incidents. The flywheel: Observe → Enrich → Infer → Adapt. Each resolved incident makes the next investigation faster.
Should Your Team Build This?
Netflix’s architecture is worth understanding even if you never deploy RDF. The honest signal: if your incident delays are from context assembly — figuring out who owns what, what changed recently, what broke similarly before — a knowledge graph pays off. If your delays are from bad alerting or slow debugging tools, this won’t help.
For teams ready to try it, the adoption path doesn’t require building QuipuDB. Start with rdflib and pyshacl for a proof of concept. Move to a hosted triple store with a SPARQL endpoint when you need team-level sharing. Only build toward indexed, scheduled enrichers when you’re operating at a scale where the rdflib approach breaks down — the documented bottleneck is 42 seconds for 360,000 triples in-memory.
AutoRCA in its current form is decision support, not autonomous resolution. Netflix is working toward targeted rollbacks and traffic shifts for high-confidence diagnoses, but that’s future research. The practical result today is that engineers start investigations with cross-system context that previously took 1–2 hours to assemble manually.
The full technical writeup is on the Netflix Tech Blog, with detailed analysis from InfoQ and the ZenML LLMOps database.
