The agent skills ecosystem hit one million published skills this year. Studies have already found that 26% of them carry security vulnerabilities, and the average quality score across 47,000 analyzed skills sits at 6.2 out of 12. Nobody has been testing them. NVIDIA’s answer is SkillEvaluator — a three-tier, open source quality gate for Agent Skills that ships CI templates, an offline security scanner, and a sandboxed agent runner that proves whether a skill actually helps your agent do anything useful. Version 0.9.0 landed November 13, 2026.
The Problem Nobody Wanted to Name
The skills.sh registry reached one million published skills seven months after launch — faster than npm, the App Store, or any package registry in history. That growth is impressive and alarming in roughly equal measure. A technical report analyzing 31,132 public skills found that 26.1% contain at least one security vulnerability and 5.2% show patterns suggesting malicious intent. Separate research found that only 9.4% of things calling themselves “skills” in public registries are actually installable, reusable artifacts.
The ecosystem scaled past its quality infrastructure. That’s the npm story replaying. npm solved distribution before anyone solved trust, and developers spent years cleaning up the mess. The agent skills world is at the same inflection point — except this time, a skill running in your CI pipeline has access to your production credentials, signing keys, and deployment authority. The stakes are higher.
Tier 1: Start With the Offline Check
The best property of SkillEvaluator’s first tier is that it costs nothing to try. No API key, no account, no cloud call. Install it and run it:
pip install skillevaluator
uv run --python 3.13 skillevaluator quality-checkTier 1 runs deterministic checks across nine dimensions: schema validity, content quality, security (via the embedded SkillSpector scanner), PII detection, license compliance, secrets scanning, code integrity, Unicode safety, and script execution. The exit code communicates pass or fail — which means a plain run: step in your CI config is already a gate. No output parsing required.
SkillSpector, the security scanner, is integrated into Tier 1 rather than treated as a separate pre-check. That’s a sensible design choice: security shouldn’t be optional or forgettable.
Block Bad Skills on Every Pull Request
Once you’re past the basics, one command gets you CI coverage:
skillevaluator ci initThat generates ready-made GitHub Actions and GitLab CI templates and wires them into your repository. From that point, any pull request that changes a skill triggers an automatic Tier 1 check. The min-score flag sets your quality threshold; required checks — security and schema — stay as hard gates regardless of score. If a skill fails those, the PR doesn’t merge.
This is the configuration most skills catalog maintainers should run. If you’re building an enterprise registry of internal skills, you’re shipping untested code to every agent in your organization without it.
Tier 3: Does the Skill Actually Help?
Tier 1 and Tier 2 answer safety and deduplication questions. Tier 3 answers the harder one: does this skill make the agent better?
The evaluation design is clean. For each test case, the agent runs twice inside a sandboxed environment — once with the skill installed, once without. Same model, same prompt, same task inputs, same grading criteria. The only variable is whether the skill is present. The delta is your answer.
Harbor 0.25.0, the open-source agent sandbox framework NVIDIA also maintains, powers the sandboxed runs. Version 0.9.0 of SkillEvaluator adds grounded evaluation-dataset generation, skill-version comparison with reusable baselines, and token and cost budgets — which matters if you’re running evals at scale across a large catalog.
The Skill That Evaluates Skills
SkillEvaluator installs as an agent skill itself. Add it to Claude Code, Codex, or OpenCode, then ask your agent to evaluate a skill directly. It’s a good use of the format: a skill that improves the quality of other skills, running inside the same environment that will eventually execute them.
The full documentation, including CI integration guides and the NVIDIA Developer Blog announcement, is at docs.nvidia.com/skills/skillevaluator.
The Bottom Line
The agent skills ecosystem is at the point where “anyone can publish” is no longer a feature — it’s a liability. SkillEvaluator doesn’t fix the supply chain problem overnight, but it gives developers a credible, zero-cost starting point. Run Tier 1. Gate your CI. The skills catalog that ships without this in 2026 is the catalog that explains to its users why their agent leaked credentials or spent three times the token budget doing nothing useful.
