By ByteBot
Your SDK has a new audience it has never been tested against. On October 19, Microsoft open-sourced Scope — a self-hosted platform that drives AI coding agents like GitHub Copilot and Claude Code through your actual developer tasks, scores what they produce, and traces exactly where they got stuck. If you ship an SDK, MCP server, CLI, or docs that developers use with AI assistance, Scope answers a question most teams haven’t thought to ask: can an AI agent actually use what you built?
The Problem Nobody Was Testing For
Developer experience teams have spent years optimizing for human developers: good docs, clean API design, helpful error messages. What most teams haven’t done is test any of that against the AI coding agent that now writes a meaningful chunk of the code that will use their product.
Agents fail SDKs in predictable ways. The most common failure mode in MCP deployments isn’t the server code — it’s tool descriptions. Agents pick tools by name and description alone. Vague or redundant descriptions produce wrong tool selections, which produce broken workflows and frustrated developers. Teams were shipping MCP servers and SDKs without knowing whether any of it was agent-readable.
Scope is Microsoft’s answer to that gap. It introduces the concept of Developer Agent Experience (DevAx): how well AI coding agents discover, choose, and use your SDKs, APIs, CLIs, and docs when developers ask them to build. DevAx is now something you can measure.
How Scope Works
The workflow is straightforward. You define a representative task — for example, “build a connection to our database using our recommended pattern” — then configure a profile specifying which agent, model, MCP servers, and extensions to use. Scope drives the agent through the task and captures everything: live logs, tool-call traces, and workspace snapshots. An LLM-based Judge service scores each run against your success criteria, which are organized as a dependency graph so one check can gate another.
When a check fails, you can trace it back through the captured logs to the specific tool call or error message where the agent got stuck. The fix is usually in the docs or the tool description, not in the code.
The Cosmos DB Result
The Azure Cosmos DB team ran Scope to compare bare models against their Agent Kit. Before any changes, agents adopted hierarchical partition keys — the recommended pattern — only 47% of the time. After one targeted Agent Kit update guided by Scope findings, adoption hit 100%. Deprecated OFFSET/LIMIT usage dropped from 47% to zero.
That is a product team changing their SDK guidance based on what agents actually do, then measuring the result. One round of Scope-guided fixes, immediate and measurable improvement. That’s the proof of concept.
Who Should Run Scope
If any of the following describes your team, Scope is worth standing up:
- SDK maintainers: Does an agent correctly pick your SDK when a developer asks it to connect to your service, or does it default to raw HTTP calls?
- MCP server authors: Can agents find and call your tools correctly, or do vague descriptions send them down the wrong path?
- CLI maintainers: Does the agent figure out the right flags and command sequence, or does it give up and reach for the API instead?
- Docs teams: Are there instructions agents consistently skip or misinterpret? Scope’s traces show exactly which steps they’re getting wrong.
Setup and Requirements
Scope is self-hosted only — no SaaS version. Local setup requires Docker with Compose v2, Node 22, pnpm, mkcert, and the GitHub CLI. There is one dependency worth flagging upfront: you need a GitHub Copilot entitlement to run agents through Scope. Teams that want a shared deployment can use the Kubernetes and Helm chart. The project is MIT-licensed and available at github.com/microsoft/scope. The official announcement covers the full architecture and Cosmos DB case study.
Agent-Aware Development Is Now a Practice
Every SDK, MCP server, and CLI now has two audiences: human developers and AI coding agents. Those audiences have different failure modes, different discovery paths, and different ways of interpreting documentation. Building only for humans while a large portion of downstream code is agent-generated is a gap in your product quality process.
Scope is the first broadly available tool that lets you close that gap with evidence rather than guesswork. For teams shipping SDKs or MCP servers into an AI-agent-heavy ecosystem, it should be part of the standard testing pipeline.
