Langfuse v4: LLM Eval Gates That Block Bad Deploys

LLM evals have always been a post-mortem sport. Something breaks in production, you dig through traces, find the prompt regression, and wonder why you did not catch it before shipping. Langfuse just changed that. Launch Week 5 (May 25–29) shipped a GitHub Actions integration that turns eval results into merge blockers — and stacked on top of a v4 architecture rebuild that makes the whole platform significantly faster, it is the testing infrastructure AI teams have been pretending they did not need.

What Changed in v4

Before the feature drops, the architecture: Langfuse v4 moves to an observations-centric data model. Everything — LLM calls, tool executions, agent steps — lands in a single wide ClickHouse table. No joins. No deduplication at read time. The result is dashboard load times 10x faster on large projects, and full-text search that goes from 18 seconds scanning 494 GB to under half a second reading less than one gigabyte.

The SDK changes are minimal but worth noting. In Python, update_current_trace() is now propagate_attributes(). In JS, same swap. Old SDKs continue to work, but expect up to a 10-minute lag on new observations in the v4 UI. The v4 migration docs have the full details.

CI/CD Eval Gates: The Main Event

The headline feature is langfuse/experiment-action, a GitHub Action that runs your evaluation script against a Langfuse dataset on every pull request. If your scores drop below the threshold you set, the workflow fails. The PR gets a comment with the detailed result. The reviewer sees exactly what regressed before merging.

- uses: langfuse/experiment-action@v1
  with:
    experiment_path: ./evals/my_experiment.py
    dataset_name: golden_set_v2
    dataset_version: "3"
    langfuse_secret_key: ${{ secrets.LANGFUSE_SECRET_KEY }}
    langfuse_public_key: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
    github_token: ${{ secrets.GITHUB_TOKEN }}

In your experiment script, you raise a RegressionError when a score misses the threshold. The action catches it, attaches the scores to the PR comment, and fails the job. Every run is tracked in Langfuse for later analysis. This requires Python SDK v4.6.0+ or JS SDK v5.3.0+.

This matters because prompt changes fail silently. A developer adjusts a system prompt to fix a customer complaint and unknowingly breaks three edge cases that worked fine last week. Traditional CI has no way to catch that. Langfuse threads eval results directly into the GitHub PR workflow — the same place the team is already reviewing changes.

Code Evaluators: Deterministic Checks, Zero Token Cost

Shipped alongside the CI/CD integration: code evaluators. You write a Python or TypeScript evaluate() function directly in the Langfuse UI, attach it to live observations or a dataset experiment, and Langfuse runs it. The result lands as a native score. No LLM calls, no token cost, fully deterministic.

The use cases are the checks you were already writing manually: Is the output valid JSON? Does it include required tool arguments? Does the action field contain only approved values? These are objective checks where an LLM judge adds cost and nondeterminism without adding value. Code evaluators handle the deterministic layer; LLM-as-judge handles the semantic layer.

One constraint: standard library only — no network egress. That keeps evaluators fast and reproducible. Scores can be numeric, categorical, boolean, or text, feeding into the same scoring system as all other Langfuse evaluations.

Agent Skills: Agents That Instrument Themselves

The third major drop: a Langfuse Agent Skill for Claude Code, Cursor, and Codex. It follows the open Agent Skills standard and ships as a focused bundle of instructions that teaches your coding agent how to use Langfuse — instrument an app, query production traces, manage prompts, and configure evaluators.

There are three ways to wire it up: load the skill playbook in your agent, use the CLI for full REST API coverage from the terminal, or connect via the MCP server. The MCP server got a significant expansion — it now covers 15 tool categories (observations, metrics, scores, datasets, annotation queues, and more), up from prompt management only.

The practical implication: instead of manually adding Langfuse instrumentation to a new project, you hand your agent the skill and let it handle setup. That closes the loop Langfuse has been building toward — observability that does not require constant human attention to maintain.

What to Do Now

If you are already running Langfuse, upgrading to Python SDK v4 or JS SDK v5 unlocks the new data model and real-time v4 UI. The CI/CD integration requires those minimum versions. Langfuse is holding a Town Hall on June 11 (9am PT) to cover V4, releases, and roadmap.

If you are not on Langfuse yet: the experiment-action is the feature that pushes it from a debugging tool to a production requirement for teams shipping LLM features. Start with the experiment-action GitHub repo. Self-hosted v4 support is still in progress, but Cloud users get it immediately.

LLM teams are roughly where web teams were in 2005 with testing — everyone agrees quality matters, but the tooling for automated validation has been missing. That gap is closing. Langfuse CI/CD eval gates are the kind of obvious-in-retrospect feature the whole space needed.