By ByteBot
AI agents don’t throw stack traces when they hallucinate. They return a confident wrong answer, your user gets burned, and your APM dashboard shows green. New Relic shipped something on October 6 that tries to close that gap.
The company introduced AI Evaluation, a new layer inside its AI Observability platform that scores live LLM responses using an asynchronous LLM-as-a-judge service — and critically, attaches those scores as attributes to your existing distributed traces. Not a separate AI dashboard. The same trace where you’d debug a database timeout now surfaces whether your model was hallucinating.
How the Scoring Works
The mechanism is straightforward: AI Evaluation samples live prompts and completions, runs them through a judge model, and produces three things for each evaluated interaction — a normalized score from 0 to 1, a pass/fail status based on a threshold you configure, and a reasoning string explaining the verdict.
Built-in evaluators cover hallucinations, prompt injection attempts, data leaks, bias, and toxicity. The scores attach directly to your existing distributed trace as attributes, so when a transaction fails quality thresholds, you can see exactly where in the call chain the AI output went wrong alongside whatever else your service was doing at that moment.
That context is what separates this from Datadog’s LLM monitoring, which evaluates isolated model calls with no APM integration. New Relic’s bet is that developers won’t adopt a separate quality dashboard — they need quality signals in the same view as latency, error rates, and database queries.
The Setup Is Lighter Than You’d Expect
AI Evaluation is built on OpenTelemetry. Auto-instrumentation covers LangChain, LangGraph, AutoGen, MCP servers, and AWS Bedrock. The minimal setup is two environment variables and a GenAI instrumentation package — no new agent installation required. New Relic’s OTel AI monitoring has been generally available since August 26.
Prompt and completion content capture is off by default, which is a sensible privacy default for production systems. If you want it, you opt in and configure drop filters or attribute-level obfuscation. LLM calls then appear as spans inside your existing traces.
Before You Ship: Prompt Playground and Regression Datasets
AI Evaluation also adds pre-deployment tooling. The Prompt Playground lets you run side-by-side model comparisons before shipping — useful for A/B testing prompts or evaluating a model swap without pushing to production first.
Reusable datasets let you version “golden datasets” sourced from real distributed traces or synthetic data, then run regression tests whenever you change a prompt or switch models. If you’ve been doing prompt engineering by feel, this gives you a more structured baseline.
APM Is Absorbing AI Observability
The release sits inside a consolidation trend accelerating through 2026. Dynatrace acquired Arize — an AI observability specialist — for $915 million in October. ClickHouse acquired Langfuse in January. The APM incumbents are betting that AI quality signals belong in the same platform as the rest of your telemetry.
That’s a reasonable position if you’re already on New Relic. It’s less obvious if you rely on a purpose-built platform like Braintrust or Arize for features those tools have had for years. The LLM-as-a-judge pattern is solid — research puts its correlation with human reviewers at around 81%, which is workable at production scale — but it’s not infallible. Agents with long-horizon tool use are particularly hard to judge correctly because a plausible-looking output isn’t the same as a correct one.
Availability
AI Agent Monitoring — real-time workflow visualization and tool-call tracking — is generally available now. AI Evaluation, including the scoring and prompt playground features, enters public preview in November 2026.
If you’re already running New Relic and shipping AI agents, this is worth enabling when the preview opens. If you’re starting fresh, the question is whether you want AI quality signals integrated into your existing APM or a dedicated evaluation platform. The consolidation bet is compelling, but purpose-built tools still have a depth advantage worth weighing.
