NVIDIA has never been one to sit still when it comes to innovation, and this time the move targets a pain point that a lot of people working with AI agents know all too well but rarely manage to solve in an organized way.
You have probably noticed that AI agents can be incredibly capable yet still stumble on the simplest tasks when they do not have the right context at hand.
And that is exactly the starting point for all of this: without the right instructions, an agent can waste tokens for nothing, wander down the wrong paths, and take far longer than it should to get anywhere.
To tackle this problem head-on, NVIDIA created SkillEvaluator, an open-source tool designed to rigorously and transparently measure how much so-called skills actually make a difference in AI agent performance.
The idea is simple but powerful: if you are going to trust an agent to handle complex tasks, you need to know whether the context you are feeding it is actually working.
And now, for the first time, that answer comes backed by hard numbers drawn from more than 300 verified skills across over 30 NVIDIA products 👇
Receive the best innovation content in your email.
All the news, tips, trends, and resources you're looking for, delivered to your inbox.
By subscribing to the newsletter, you agree to receive communications from Método Viral. We are committed to always protecting and respecting your privacy.
What skills are and why they matter so much
Before understanding what SkillEvaluator does, it is worth stepping back and understanding the concept of a skill in the world of artificial intelligence agents. A skill, in this context, is basically a block of structured context you hand to the agent before it executes anything. It could be an instruction on how to use a tool, a set of rules to follow in a specific situation, or even examples of how to respond to certain types of requests. Its purpose is straightforward: prepare the agent to act better, with more precision and less waste of computational resources throughout the process.
In NVIDIA’s specific case, verified skills are packaged and signed capability descriptors that tell the agent exactly what a company product does, when it should be triggered, and how it should be called. The verified part is precisely the measurement confirming that a given skill is ready for use and not just a promise on paper. It is this quality seal that separates a generic instruction from something that has already gone through a serious evaluation process.
The problem is that, until now, measuring whether a skill was actually working was a pretty subjective process. Development teams would test, tweak, test again, and cross their fingers hoping the results improved. There was no standardized way to compare different versions of a skill or to know for certain whether the change you made to the agent’s context brought any real performance gain. It was a lot of gut feeling, very few metrics. And when you are dealing with systems that need to make decisions at scale, gut feeling is not enough.
That is where NVIDIA stepped in with a concrete proposal. By developing SkillEvaluator, the company created a framework capable of isolating the impact of each skill individually, testing it under controlled conditions and generating measurable data on what changed, how much it changed, and whether the change was positive or negative for the agent. This impact is measured through a metric called Skill Lift, which represents exactly the score difference between running a task with the skill installed and without it.
How SkillEvaluator works in practice
SkillEvaluator operates with a three-layer evaluation logic, and each layer answers a different question about the skill. The first layer handles safety and structure, running static schema checks, quality scoring, prompt injection scanning, data leakage detection, sensitive information identification, and license verification. The second layer focuses on distinction, using embedding similarity to spot duplicate guidance within a single skill or overlaps between different skills in the catalog.
The third layer is where the real magic happens. It runs a live evaluation with an agent executing automatically generated tasks, once with the skill installed and once without, all inside an isolated and controlled environment. To do this, the tool relies on Harbor, an open-source framework built for running agent evaluations in repeatable, sandboxed environments. SkillEvaluator handles all the setup, turning test cases into tasks, running the agents in sandboxes, and calculating the skill’s final impact.
The brilliant part here is the rigor of the comparison. For each evaluated case, the agent runs twice with the exact same prompt, the same model, the same inputs, and the same grading criteria. The only variable that changes between the two runs is the presence or absence of the skill. This ensures that any observed difference in the result is genuinely attributable to the skill and not some random factor. And because this comparison is repeated across two different agent environments, the results gain even more robustness.
Being open source, SkillEvaluator was also built with flexibility in mind. Development teams can adapt the evaluation criteria to their own use cases, plug different artificial intelligence models into the testing process, and integrate the tool into their existing pipelines. NVIDIA clearly does not want this to be just a technology showcase but rather something real teams can use day to day to make better decisions about how they are building their agents.
What the results show so far
The initial data released alongside the SkillEvaluator launch is quite revealing about the real impact that well-built skills can have on AI agent performance. The tests were conducted in two popular environments, Codex and Claude Code, and evaluated each skill across five different dimensions: correctness, discovery, effectiveness, efficiency, and safety.
Without any skill installed, the agents performed fairly average across most dimensions. Baseline scores fell between 39 and 46 points out of 100 in correctness, discovery, effectiveness, and efficiency, already signaling a huge room for improvement. The only exception was safety, which started with a very high score of 97 points, since the primary goal in that dimension was to make sure installing a skill did not introduce any new problems.
The gains with verified skills
When verified skills entered the picture, the numbers jumped impressively. Correctness climbed from 46 to 87 points, a gain of 41 points. Discovery leaped from 42 to 82, an advance of 40 points. Effectiveness went from 39 to 78, adding 39 points, and efficiency grew from 43 to 78, a gain of 35 points. Safety, already high, rose just one point, going from 97 to 98. On the overall average excluding safety, the gain landed at 39 points, which is a remarkable result.
These numbers do not represent probability of success but rather a higher average performance on the specialized tasks evaluated. It is worth highlighting that the discovery and efficiency dimensions serve as indicators that the skill is being activated and used correctly when present. This matters a lot because each skill in an environment competes for the agent’s attention, and a skill that loads when it is not relevant can actually hurt overall performance.
The environment matters too
An interesting detail is that Skill Lift varied between the two tested environments. Claude Code showed slightly larger gains, with a Skill Lift of 42 points excluding safety, compared to 36 points for OpenAI Codex. This difference is expected since each environment has its own system prompts, ways of handling context, and tool-calling implementations. Even so, both showed consistent and significant gains, reinforcing that the skill provides a structured foundation that neither environment can produce on its own.
Why this changes the game for anyone working with AI agents
If you work in artificial intelligence-based system development, you have probably been in the situation of tweaking an agent’s prompt multiple times without being sure whether the changes were helping or hurting. Most teams still operate on a fairly manual trial-and-error cycle where decisions about what to include or remove from the agent’s context depend heavily on the intuition of whoever is developing it. SkillEvaluator proposes a break from that pattern by introducing a systematic and reproducible evaluation process that any team member can run and interpret.
One of the most valuable takeaways the tests revealed is that token savings are not automatic. In one example, the Jetson memory optimization skill slashed token consumption from over 617,000 to just over 142,000, a drop of nearly 77 percent, while also cutting execution time in half. On the other hand, an installation skill for another product ended up increasing token consumption by more than 120 percent, revealing a clear optimization opportunity. In other words, the tool does not just show what works — it also pinpoints where there is still work to be done.
Another point that became evident is that the product matters more than the agent. Skill Lift varied far more across different products than between the two test environments. While Claude Code and Codex differ by about five points on average, Skill Lift by product ranged from roughly 2 all the way up to 46 points. This shows that task domain, evaluation design, and test suite quality weigh far more than the choice of environment itself.
NVIDIA has also already put SkillEvaluator into practice with partners. OpenClaw is piloting the tool for official organizations on ClawHub, displaying results with and without skills directly in the interface. Meanwhile, Nous Research tested the solution on Hermes Agent, with an optional security sweep that checks for sensitive data, licensing issues, and other concerns before installation — all in about one and a half seconds per skill.
SkillEvaluator is available as an open-source project and represents another step by NVIDIA toward more transparent, measurable, and efficient artificial intelligence development for all types of teams. 🚀
