Gate AI Agent Deployments with Bedrock AgentCore and GitHub Actions

Developers are shipping AI agents to production without tests. Not because they are reckless — because the tooling was not there. That changes now. AWS published a reference implementation that wires Amazon Bedrock AgentCore Evaluations into GitHub Actions, turning every pull request that touches an agent into a gated quality check. If your agent’s goal success rate drops, the merge is blocked. Same as your unit tests. It took the software industry a decade to build CI/CD culture around code; AI agents are about to get the same treatment.

What AWS Shipped

Amazon Bedrock AgentCore Evaluations went generally available in March 2026. AWS’s new reference implementation extends it into pull request workflows: when a PR modifies agent code, a system prompt, a model selection, or a tool configuration, GitHub Actions automatically deploys the agent to a dev environment, invokes it with a test dataset, retrieves the resulting OpenTelemetry traces from CloudWatch, scores them using AgentCore’s built-in evaluators, and blocks the merge if any score falls below a configured threshold. The full reference implementation is available on GitHub.

The Four Evaluators That Gate Your PRs

The reference implementation uses four evaluators from AgentCore’s catalog of 20+:

• GoalSuccessRate — Did the agent actually complete the task the user gave it?
• Correctness — Is the response accurate and appropriate given the prompt?
• ToolSelectionAccuracy — Did the agent pick the right tool from its available set?
• ToolParameterAccuracy — Did it derive the right parameters from context before calling the tool?

This matters because most AI monitoring only checks the final output. These evaluators examine the reasoning path — whether the agent reached a good answer through a correct process or stumbled onto it by accident. The broader catalog also includes safety evaluators (Harmfulness, Stereotyping, Refusal) and trajectory checks for validating multi-step tool sequences.

The Variance Problem Nobody Talks About

Here is the implementation detail that will save you a frustrating afternoon: LLM-as-judge evaluation is non-deterministic. Run the same agent against the same prompts twice, and you might see scores of 0.84 and 0.79 with zero code changes. Set a hard threshold at 0.80 and you will generate random red builds that erode team confidence in the pipeline.

The fix is straightforward: set thresholds with margin below your target reliability, and consider averaging across multiple evaluation runs before applying a pass/fail decision. AWS explicitly warns about this in their implementation guide. The broader industry recommendation is to gate on a regression — block the PR only when scores drop more than 3% from baseline — rather than on an absolute number. Block on signal, not on noise.

Trajectory Checks: Cheaper and More Deterministic

Before reaching for LLM-as-judge for every check, consider trajectory evaluation. AgentCore includes three trajectory evaluators: ExactOrderMatch, InOrderMatch, and AnyOrderMatch. These validate that an agent called tool_X before tool_Y, or that a specific sequence of tools was used in the expected order. They are deterministic, add no extra model invocation costs, and catch a large class of multi-step agent regressions. Use trajectory checks as your primary gate; reserve LLM scoring for the subjective quality questions that trajectory checks cannot answer. DeepEval’s LLM-as-judge analysis covers the broader variance problem in depth if you want more context on when to use each approach.

The Honest Trade-offs

This pipeline is not free. A few things to plan around:

• 10 minutes per PR. CDK deployment, 30-second runtime startup, 30-90 seconds for CloudWatch trace propagation — it adds up. This is a PR check, not a pre-commit hook.
• Cost scales with evaluators. Four evaluators across five test prompts means 20 LLM judge model calls per pull request. At moderate PR volume that is manageable; at high volume, trajectory checks reduce that number significantly.
• ARM64 cross-compilation. The MCP server runtime requires ARM64 container images, but GitHub-hosted runners are x86_64. You will need Docker Buildx with QEMU cross-compilation. The reference implementation handles this, but be aware if you are customizing the Dockerfile.
• M2M tokens bypass role checks. By design, machine-to-machine tokens used by the CI pipeline get access to all tools, skipping the role enforcement applied to real users. Flag this with your security team before deploying to a sensitive environment.

Getting Started

The minimal path to running agent evaluation in GitHub Actions, using the agent-evaluation library:

name: Agent Evaluation
on:
  pull_request:
    branches: [main]
env:
  AWS_REGION: us-east-1
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ env.AWS_REGION }}
      - run: pip install agent-evaluation
      - run: agenteval run
      - name: Post Results
        run: cat agenteval_summary.md >> $GITHUB_STEP_SUMMARY

For the full stack — CDK infrastructure, Cognito setup, OIDC federation, and multi-evaluator scoring — follow AWS’s detailed implementation guide linked above. The complete reference implementation takes roughly a day to wire up the first time.

AWS Is Not the Only Option

If you are not in the AWS ecosystem, the same pattern exists elsewhere. DeepEval offers a pytest-native interface with 50+ metrics that runs framework-agnostically. LangSmith integrates tightly with LangChain and LangGraph workloads. Langfuse provides a self-hosted open-source option for teams that need data residency control. The AWS implementation stands out for its managed infrastructure and native integration with IAM, CloudWatch, and CDK — if your agents already live in AWS, that coherence is worth something.

The broader point: every major cloud now offers agent evaluation as a managed service. The tooling excuse for skipping agent testing is gone. Your agents deserve the same quality gates your APIs have had for a decade.