By ByteBot
79% of engineering leaders say AI agents help their teams write code faster but haven’t made release cycles any shorter. The bottleneck isn’t the code — it’s finding the failures. Specifically, finding the traces where the agent confidently reported success and was wrong. LangChain shipped AI Search in LangSmith this month, and it’s the first observability feature designed for exactly that problem: type a plain-English description of a failure behavior, get back the matching traces.
The Failure Mode Nobody Talks About
Standard monitoring is built for binary outcomes. Did the request succeed or fail? Did the service return 200 or 500? That model breaks down with AI agents because most agent failures return 200. The agent executes, returns a result, and every metric shows green. What’s invisible: the agent retrieved the wrong document, called a tool that didn’t exist, or told the user a task was complete when it wasn’t.
Debugging that in LangSmith until now meant writing metadata filters, running keyword searches across trace content, and scrolling manually through thousands of runs hoping something surfaced. Agent trace spans average 50KB each, and a single production session can generate over 10GB of trace data. Finding behavioral failures at that scale — not errors, not timeouts, but wrong answers — was effectively a manual job.
What AI Search Does
AI Search lets you describe a behavior in natural language and find the traces where it happened. From LangChain’s announcement, example queries include:
- “Find traces where the agent said it finished a task but didn’t complete it”
- “Show me conversations where the user seemed frustrated or pushed back”
- “Find requests where PII was exposed in the response”
- “Show me cases where the agent was asked to cancel a subscription”
These aren’t keyword searches. The feature uses a decision model that evaluates trace content against your description — not text similarity, but behavioral match. You can combine natural-language queries with existing filters (error tags, time ranges, custom metadata) for more precise targeting.
How It Works Under the Hood
LangSmith’s backend is SmithDB, a purpose-built database written in Rust using Apache DataFusion and the Vortex columnar format. As traces arrive, SmithDB converts them into “trajectories” — flat, ordered sequences of messages, tool calls, and tool results that show the path the agent took from start to finish. Because every supported SDK maps to the same trajectory format, one AI Search query works across agents built on different LangChain versions.
When you submit a query, structured filters narrow the candidate set first. Then the decision model evaluates trajectories in parallel batches, with an 8,000-token budget per trajectory that prioritizes user messages and agent responses over system messages and tool results. The SmithDB architecture is designed specifically for agent workloads — not a general-purpose database with observability bolted on.
Who Should Use It Now
AI Search is strongest for teams running LangChain or LangGraph agents in production. LangChain appears in over 1.6 million GitHub repositories, so that’s not a small audience. If you’re using a custom orchestration layer or a non-LangChain framework, the trajectory format still ingests your traces, but you’ll see less value from the SDK-level integration.
The other constraint is tier. AI Search is available on LangSmith Plus ($39/seat/month) and Enterprise. The free Developer tier covers 5,000 traces per month with 14-day retention — good for local development, not for production debugging at scale. Most teams shipping production agents are already on Plus or Enterprise for the evaluation and dashboard features anyway.
No other major observability platform currently offers behavioral natural-language trace search. Arize Phoenix, Langfuse, and Braintrust are all solid tools in this space, but none have shipped this specific capability. LangSmith has a head start here, and the SmithDB trajectory architecture gives it a structural advantage for LangChain-native stacks.
The Bigger Shift
AI Search is a small feature with a larger implication. Traditional observability answers syntactic questions: did it run, did it error, how long did it take? Agent observability needs to answer semantic questions: did it do what we intended? For two years, the industry tried to answer that question with the same tools built for microservices. It mostly didn’t work.
According to the BairesDev Q3 2026 Dev Barometer, 42% of developers now say AI writes at least half their code, up from 12% a year ago. The agents running on that code are in production, handling real user interactions, and failing in ways that existing monitoring doesn’t catch. AI Search is the right abstraction: describe the failure, find the traces, fix the behavior. The question is whether teams with agents in production are using it yet.
