Evaluating Deep Agents with LangSmith on AWS: the complete guide to ensuring quality in AI agents
Artificial Intelligence has reached a point where simply building a smart agent and hoping it works well in production just doesn’t cut it anymore. Validating the behavior of these systems before they go live has become, without exaggeration, one of the biggest challenges in the field today. And the problem isn’t simple: AI agents are multi-step, non-deterministic systems where a mistake early on can compromise everything that follows. A single wrong tool call can create a cascading effect throughout the entire workflow.
That’s exactly where the combination of LangSmith running on AWS comes in — a powerful stack that provides the evaluation framework needed to catch these issues early, trace them in production, and continuously improve your agent’s reliability throughout its entire lifecycle. This guide combines learnings from LangChain‘s work on deep agent evaluation and Anthropic‘s guide on demystifying evaluations for AI agents.
Here you’ll find something genuinely practical, from the fundamental concepts every developer needs to know, to the implementation of five evaluation patterns for Deep Agents, including offline testing with pytest and real-time monitoring in production. The central example is a text-to-SQL agent built with Amazon Bedrock, using the Amazon Nova 2 Lite model, walking through the entire development cycle all the way to production.
Getting to know Amazon Nova 2 Lite
Before diving into the evaluation patterns, it’s worth understanding the model at the center of this example. Amazon Nova 2 Lite is a fast reasoning model with solid cost-effectiveness, available on Amazon Bedrock. It supports extended thinking with configurable compute budget levels (low, medium, and high) and accepts text, image, video, and document inputs, with a 1 million token context window. Nova 2 Lite performs well on tasks like instruction following, function calling, and code generation, making it a natural choice for agent-based workloads like the text-to-SQL agent used in this post.
Why evaluating AI agents is so different from testing regular software
Anyone who’s worked with traditional software testing knows there’s a relatively clear logic: you provide an input, expect a deterministic output, and verify whether the result matches what’s expected. But when it comes to AI agents, that logic goes almost completely out the window. These systems don’t follow a fixed path. They reason, make intermediate decisions, query external tools, interpret ambiguous contexts, and generate responses that can vary with every run — even with the same input.
Three properties make agent evaluation fundamentally different:
- Non-determinism: the agent’s behavior varies between runs. The same task might succeed 90% of the time and fail 10%. A single pass/fail result doesn’t tell you much. You need multiple attempts to estimate real performance. Two metrics help here: pass@k measures the probability of at least one success in k attempts, while pass^k measures the probability that all k attempts succeed.
- Error propagation: in a multi-step agent, a mistake at step 3 can propagate through every subsequent step. A text-to-SQL agent that identifies the wrong schema at the beginning will build an incorrect JOIN, producing wrong results in the final response. Evaluating only the final output won’t reveal where things went wrong.
- Creative solutions: frontier models sometimes find valid approaches that evaluation designers didn’t anticipate. This means the evaluation needs to be flexible enough to recognize alternative paths that still produce correct results.
Essential terminology for understanding agent evaluation
Before getting into the patterns, it’s important to align on vocabulary:
- Task: an individual test with defined inputs and success criteria. For example, the question How many customers are from Canada? with an expected answer of eight.
- Trial: an individual execution of a task. Since model outputs are non-deterministic, running multiple trials per task produces more reliable results.
- Grader: the logic that scores some aspect of the agent’s performance. A task can have multiple graders, each evaluating a different dimension.
- Transcript: the complete record of a trial, including tool calls, reasoning steps, intermediate results, and interactions. In LangSmith, this is the full trace you can inspect for debugging.
- Outcome: the final state of the environment at the end of a trial. The agent might say the answer is eight, but the actual outcome is whether it executed the correct SQL query against the database.
- Evaluation harness: the infrastructure that runs end-to-end evaluations.
- Evaluation suite: a collection of tasks designed to measure specific capabilities or behaviors.
What can be evaluated in an agent run
For an agent run, there are three categories that can be tested:
- Trajectory: the sequence of tools called and the specific arguments the agent generated. Did it explore the schema? Did it use the sql_db_query_checker before executing?
- Final response: the final output returned to the user. Is the answer correct? Is it well-formatted?
- Other states: artifacts the agent produced, such as files written, task plans created, and intermediate results saved.
The three types of graders and how to combine them
Agent evaluations typically combine three types of graders, and the key to effective design is choosing the right mix for your use case.
Receive the best innovation content in your email.
All the news, tips, trends, and resources you're looking for, delivered to your inbox.
By subscribing to the newsletter, you agree to receive communications from Método Viral. We are committed to always protecting and respecting your privacy.
Code-based graders
These use deterministic logic to verify specific conditions: string matching, regex patterns, binary pass/fail tests, static analysis, and tool call verification. They’re fast, cheap, objective, and easy to debug. When you can express success criteria as code, do it. The limitation is that they can be brittle when legitimate variations don’t match the expected pattern. A result formatted as eight customers compared to There are eight can fail on a strict match even though both are correct.
Model-based graders (LLM-as-judge)
These use another LLM to evaluate the agent’s output. Methods include rubric-based scoring, natural language assertions, pairwise comparison, and multi-judge consensus. They’re flexible, scalable, and capture nuance, but they’re non-deterministic, more expensive, and need calibration against human evaluators. An important tip: give the judge LLM an escape hatch, like return Unknown if you don’t have enough information, to avoid hallucinated scores.
Human graders
Considered the gold standard for subjective quality assessments. They’re expensive and slow compared to programmatic options, but essential for calibrating model-based graders. Use them sparingly: calibrate your LLM-as-judge rubrics against expert human judgment initially, and then use periodic human review to check whether the automated graders have drifted.
Practical combination recommendation
For a text-to-SQL agent, the ideal combination looks like this:
- Code-based: did the agent call sql_db_query? Does the answer contain the correct number? Were DML commands (INSERT, DELETE) executed?
- LLM-as-judge: for complex queries where the output format varies. Is the analysis correct, complete, and well-structured?
- Human: periodic checks to ensure the LLM evaluation aligns with expert judgment.
Capability evaluations versus regression evaluations
Not all evaluations serve the same purpose, and understanding this distinction is critical:
- Capability evaluations ask what can this agent do well? They should target tasks the agent currently struggles with, giving the team a clear goal. Start with a low pass rate and work to improve it.
- Regression evaluations ask can the agent still handle what it used to handle? These should have a pass rate close to 100%. A drop signals that something broke.
As the agent matures, capability evaluations that reach high pass rates can be promoted to the regression suite. Tasks that once measured can it do this? shift to measuring can it still do this reliably?
The five evaluation patterns for Deep Agents in practice
Deep agents are systems that use planning, tool calls, file system backends, and progressive context loading to tackle complex, multi-step tasks. They break the traditional assumption that every test case can be run by the same application logic and scored by the same grader. LangChain, while developing four applications on deep agent architectures, identified the patterns that apply broadly.
Pattern 1: Custom test logic per case
Traditional LLM evaluation treats every data point identically. Deep agents break that assumption. Each test case can have its own success criteria, and those criteria can involve specific assertions against the agent’s trajectory and state — not just the final message. LangSmith’s integration with pytest supports this pattern, allowing you to make different assertions about the trajectory, the final message, and the state for each test case.
Pattern 2: Single-step evaluations
About half of LangChain’s test cases for deep agents were single-step evaluations: what did the agent decide to do immediately after a specific input? This is especially useful for validating individual decision points. Regressions often happen at individual decision points, not across full execution sequences. For a text-to-SQL agent, a single-step evaluation might verify that the agent’s first action is to explore the database schema rather than jumping straight to writing a query. Think of these as the unit tests of agents: fast, focused, and token-efficient.
Pattern 3: Full agent turns
While single-step evaluations test individual decisions, full turns show the big picture. Run the agent end-to-end on an input and evaluate both the trajectory and the final response. An important insight here: test whether certain tools appeared in the trajectory, but don’t require an exact order. The agent might list tables before getting the schema, or go straight to the schema. Both are valid. Evaluate what the agent produced, not the exact path it took.
Pattern 4: Multi-turn evaluations
Some scenarios require testing agents across multi-turn conversations. A user asks What are the top 5 best-selling artists? and then follows up with For the top artist, how many albums do they have? The challenge is that if you hard-code a sequence of inputs and the agent deviates from the expected path, subsequent inputs might not make sense. The solution is to use conditional logic in tests: if turn 1 failed, fail early instead of continuing with incoherent data.
Pattern 5: Safety and state checks
This pattern goes beyond correctness and evaluates whether the agent operates within safe boundaries. For the text-to-SQL agent, the most critical check is ensuring no DML commands (like INSERT, UPDATE, DELETE, DROP, ALTER, or TRUNCATE) get executed. It also verifies that complex queries trigger the agent’s planning mechanism and that the final response is substantive enough to be useful to the user.
Practical example: text-to-SQL agent architecture on AWS
The text-to-SQL agent used as an example is built on the DeepAgents framework, which provides planning, file system storage, and progressive context loading on top of LangGraph. It answers natural language questions about the Chinook database, a sample SQLite database representing a digital media store.
To follow along with the full example, you’ll need:
- An AWS account with Amazon Bedrock access enabled
- A LangSmith account and API key
- Python 3.12 or higher
- AWS CLI configured with credentials
- Required packages: deepagents, langchain-aws, langchain-community, and pytest
The agent uses Amazon Nova 2 Lite on Amazon Bedrock through ChatBedrockConverse, and LangSmith tracing is automatically integrated into the LangGraph execution. This means every tool call, planning step, and agent decision is captured as a trace with no manual instrumentation needed.
Offline testing with pytest and the importance of automation
One of the most practical aspects of this approach is the integration between LangSmith and pytest. Using the @pytest.mark.langsmith decorator, each test case is automatically logged as an experiment in LangSmith with full traces. This allows all five evaluation patterns to be executed in an automated fashion within continuous integration pipelines, ensuring no changes to the agent go to production without passing through a complete battery of tests.
What makes this approach especially powerful is that it clearly separates two distinct moments in the development cycle: the experimentation phase, where you’re still building and tuning the agent, and the validation phase, where you need objective, reproducible guarantees about the system’s behavior. For each test run, you can:
- Inspect full traces: see every tool call, the write_todos planning step, each sql_db_schema invocation, the actual SQL queries executed, and the final formatted response.
- Track feedback scores over time: the t.log_feedback() calls create metrics you can visualize in charts across experiments.
- Compare experiments: run the same evaluation suite after a change and compare results side by side.
- Monitor token usage and latency: identify which agent steps are most expensive and where to focus improvement efforts.
Real-time monitoring: from offline to online in production
Everything built so far — the five pytest-based evaluations — runs offline, before deployment. You organize test cases, run the agent against them, and check the scores. That’s essential for development and regression testing. But the next step is monitoring the agent in production, where there are no reference outputs. Real users ask questions you never anticipated, the database can change, and edge cases emerge that no curated dataset captures.
LangSmith supports two evaluation modes that work together across the agent’s lifecycle:
- Offline evaluation: runs on curated datasets with reference outputs, before deployment, for benchmarking, regression testing, and unit testing.
- Online evaluation: runs on live production traces, after deployment, for real-time monitoring and anomaly detection, using only inputs and outputs without reference answers.
Online evaluators run automatically on production traces. No code deployment is needed. You configure them in the LangSmith interface, and they score every trace (or a sample) in real time.
Online code evaluator: SQL safety check
Code evaluators are deterministic Python or JavaScript functions that run inline in LangSmith. They’re ideal for safety guardrails that need to check every production trace. For the text-to-SQL agent, the most critical check is ensuring the agent doesn’t execute DML commands in production.
Online LLM-as-judge evaluator: response quality
Since there are no reference outputs in production, this is a reference-free evaluator. The judge assesses the response’s internal consistency, clarity, and apparent completeness. You configure a sampling rate — say 50% of traces — to control costs, and map the prompt variables to the run’s inputs and outputs.
Online composite evaluator: overall quality score
Composite evaluators combine multiple scores into a single metric, useful for dashboards and alerts. For the text-to-SQL agent, the recommended setup uses a weighted average where SQL safety gets a weight of 0.4, correctness confidence gets 0.3, clarity gets 0.15, and completeness gets 0.15. With this composite score, you can filter problematic traces, build quality trend charts, and set up alerts when quality drops below a defined threshold.
Closing the loop: from production failures to new test cases
Production monitoring also closes the agent’s continuous improvement loop. When LangSmith identifies problematic runs, they can be easily added to the offline evaluation dataset, enriching test cases with real examples the agent encountered in production and struggled with. This creates a virtuous flywheel where the agent gets progressively better over time — not just because the base model improves, but because the development team has increasing visibility into the system’s weak points and can address them systematically based on concrete data.
The loop between offline and online evaluation is the key to improving agent behavior: production failures become test cases, test cases help prevent future failures, and metrics replace guesswork.
The combination of LangSmith, AWS, and the five evaluation patterns for Deep Agents represents one of the most comprehensive frameworks available today for anyone who wants to take AI agent development seriously — with the discipline and rigor that real production systems demand.
To explore the full working example, the companion repository is available on AWS’s GitHub. For more about the services used, check out the documentation for Amazon Bedrock for managed access to foundation models, Amazon Nova for the AWS model family, and Amazon Bedrock Guardrails for adding safety controls to your agents. 🚀
