By ByteBot
Most teams reach the same realization about six months into production AI: every service is calling a different model with a different API key, there is no unified view of what anything costs, and when one provider goes down, the outage cascades straight to users. ARBR, which launched today as a fully open-source project, is a self-hosted AI gateway built to fix exactly this. Deploy one instance, point your existing clients at it, and every LLM request your apps make immediately gains routing, budget enforcement, prompt-injection protection, and structured observability — without changing a line of application code.
One Endpoint, Every Provider
ARBR exposes a single OpenAI-compatible endpoint. Swap your OPENAI_BASE_URL to point at the ARBR instance and your existing SDK, agent framework, or chat UI keeps working as-is. Behind that endpoint, ARBR routes to any major provider or OpenAI-compatible self-hosted model, with pricing and benchmark data covering more than 3,000 models. The compatibility is not a thin shim — the same gateway seat that routes traffic also governs it, observes it, and evaluates it.
The Differentiator: Routing That Actually Thinks
Most gateways route on static rules: latency threshold, cost cap, availability. ARBR adds something the field has been missing — difficulty-aware routing. The gateway classifies each incoming request by complexity and routes accordingly: a simple summarization task goes to Gemini Flash or Muse Spark; a complex multi-step reasoning request goes to GPT-6 Astra or Claude Fable. The cheapest model that does the job gets the job.
The more interesting part is the evaluation loop. ARBR scores a sample of live traffic using an LLM judge, confirms which models are clearing your quality bar, and adjusts routing to favor the cheapest option that passes. This is canary testing for model selection, running continuously in production. Competitors like LiteLLM and Portkey offer routing, but neither does live eval-based routing as a first-class feature. That gap matters when you are paying per token and the price delta between “good enough” and “frontier” is 10x.
Governance in the Request Path, Not After It
The governance model is where ARBR takes a clear position: controls act in the request path, not post-hoc. Budget limits, rate limits, prompt-injection checks, and kill switches all fire before a request leaves your infrastructure. Output guardrails run on responses before they reach callers.
This matters more than it sounds. Discovering that a runaway agent spent $3,000 in tokens overnight is a postmortem problem. Stopping it mid-session is a governance problem. ARBR handles the second one. You set a budget by app, team, model, or user — when the budget is hit, the kill switch fires. Every decision is logged as a structured event attributable to the exact source.
Where It Sits in the Landscape
The AI gateway space in 2026 has a few main options:
- LiteLLM: Best self-hosted routing with zero vendor dependency, strong virtual keys, but no live eval and real ops burden to run reliably.
- Portkey: The cleanest managed option — 250+ models, PII redaction, semantic caching out of the box — but prompt data transits Portkey’s servers, which matters for healthcare and finance.
- Helicone: Observability-focused and lightweight, but thin on routing and governance; teams usually layer it on top of LiteLLM.
- ARBR: Positions as the convergence point — routing, governance, and live eval in one self-hosted binary. Newest of the group with the smallest community, but the architecture is the most complete.
If you are already on LiteLLM and happy with the ops overhead, there is no immediate reason to migrate. If you are evaluating your first gateway or you want live eval-based routing, ARBR is the place to start.
Getting Started
ARBR is self-hosted and open source. Deploy the gateway, set OPENAI_BASE_URL in your applications to point at it, and traffic starts flowing through immediately — no code changes in the calling applications. The project site at projectarbr.org has deployment documentation and configuration reference. For agentic governance context, OpenRouter’s guide on enforcing AI governance at the API layer is a useful primer on what problems you are solving.
The Broader Shift
ARBR is not an isolated launch. September 4, 2026 has brought several agent-infrastructure tools to market simultaneously — a consistent signal that the industry has moved from experimenting with AI agents to deploying them in production. An AI gateway is now the same category of non-negotiable infrastructure that an API gateway became for microservices around 2018. The teams that add governance before they need it will have a significantly easier time than those reverse-engineering it from production incidents.
