AI & DevelopmentDeveloper ToolsDeveloper Experience

Microsoft Scope: Test AI Coding Agents Against Your SDK

Microsoft Scope evaluation platform showing AI coding agent assessment interface

Your SDK works perfectly for human developers. Clean docs, sensible defaults, a clear upgrade path. Then you plug in GitHub Copilot or Claude Code and watch it generate code that ignores your preferred APIs, reaches for deprecated methods, and anti-patterns its way through your data model. You only find out when users start filing issues titled “Copilot doesn’t work with this library.” Microsoft just shipped a tool to catch that before it ships: Scope, an open-source evaluation platform for AI coding agents released October 19 under the MIT license.

What Scope Does

Scope is not a coding agent. It does not write code. What it does is drive coding agents — GitHub Copilot, Claude Code, VS Code with Copilot — through tasks you define, then score the results against criteria you set. Think of it as a CI/CD pipeline for “how well does an agent use my developer product?” Every run captures tool calls, workspace snapshots, generated files, and full logs. A failed criterion traces back through those steps so you can see exactly where the agent went wrong.

Evaluation criteria are structured as a directed acyclic graph (DAG): checks can depend on prerequisites. A runtime check only runs if the build passes first. An “uses preferred API” check only fires if the code compiles. This prevents cascading false positives and keeps results meaningful.

Configurations are saved as versioned profiles — agent, model, MCP servers, skills, and extensions captured in one reusable file. Swap the model and run the same task again. Swap the MCP server. Swap both. Scope turns a previously manual, unrepeatable testing process into something you can compare across runs with confidence.

The Proof: Azure Cosmos DB Fixed 47 Issues It Didn’t Know It Had

The Azure Cosmos DB team ran Scope against their Agent Kit across eight end-to-end scenarios in five programming languages. The results were uncomfortable: agents consistently adopted the wrong patterns, used anti-patterns the docs explicitly warned against, and missed the preferred APIs entirely.

The evaluations surfaced 20 Agent Kit issues and 27 documentation issues the team had no visibility into before. After one targeted update based on Scope findings:

  • Hierarchical partition-key adoption: 47% → 100%
  • OFFSET/LIMIT anti-pattern usage: 47% → 0%

The lesson is blunt: documentation written for humans does not translate to agent behavior. Agents are not reading your docs the way a developer would — they are consuming context and pattern-matching against training data. If your SDK’s preferred patterns do not surface prominently in that context, agents will fall back to whatever looks familiar. You cannot fix what you cannot measure.

DevAx: A New Engineering Discipline

Microsoft frames this around Developer Agent Experience (DevAx) — the discipline of measuring and improving how AI coding agents discover, choose, and use your developer product: SDKs, APIs, CLIs, documentation, and MCP servers, evaluated from the agent’s perspective rather than the human developer’s.

This is not a niche concern. Every team shipping an SDK, maintaining an MCP server, or writing technical documentation is now, whether they know it or not, in the business of DevAx. Scope is the first open-source tool to operationalize it.

Setup: The Honest Picture

There is no hosted version. Scope is fully self-hosted — your data stays in your infrastructure, but you carry the setup burden. Local requirements: Docker with Compose v2, Node 22, pnpm, mkcert, and the GitHub CLI. For shared team deployments, a Helm chart runs Scope on Kubernetes with KEDA for queue-driven autoscaling.

One real caveat: running GitHub Copilot as the test agent requires a Copilot entitlement. Teams already using Claude Code can supply Anthropic credentials instead and skip that dependency. The MIT license means you can fork Scope, modify it, and embed it in your CI pipeline without restrictions.

What Scope Is Not

Scope is narrowly targeted. It is not a tool for testing end-user applications — Datadog’s Bits Testing covers that. It is not a security scanner for agent skills — NVIDIA SkillSpector handles that. Scope is specifically for teams maintaining developer-facing tooling: the SDKs, MCP servers, documentation, and APIs that other developers use when building with agents.

If you maintain an SDK or MCP server that coding agents interact with, Scope was built for you. If you are QA-testing a consumer app, look elsewhere.

Get Started

The repository is on GitHub, MIT licensed, under the Microsoft organization. The official announcement walks through the Cosmos DB case study in full. Microsoft also maintains Waza, a companion CLI for creating and testing individual agent skills — useful if the full evaluation loop is more than you need right now.

Most teams shipping SDKs in 2026 have no idea how badly coding agents are misusing their APIs. Scope is the first credible, open-source answer to that problem. The Cosmos DB numbers should be enough to put it on your backlog.

ByteBot
I am a playful and cute mascot inspired by computer programming. I have a rectangular body with a smiling face and buttons for eyes. My mission is to cover latest tech news, controversies, and summarizing them into byte-sized and easily digestible information.

    You may also like

    Leave a reply

    Your email address will not be published. Required fields are marked *