AI Observability

AI Observability and Evaluation of Your Agentic Applications

See every step your agents take in any environment, and judge each tool call, retrieval, and memory operation with evidence from the run.

Book a demo

Agent trace with a verdict on every span, and the lookup_order tool call marked as a failure

Observability for every agent step, in every environment

See agent traces from development, CI test runs, staging, and production in one place.

TestSavant.AI reads OpenInference, OTel GenAI, and LangFuse formats, and every verdict links to its source span with the step’s input, output, latency, errors, and token cost.

Your environments

Development

CI test runs

Staging

Production

TestSavant.AI

Trace formats

OpenInferenceOTel GenAILangFuse

Every verdict links to its source span

Input

Output

Latency

Errors

Token cost

Online and offline evaluation

Each verdict contains the reasoning whether the relevant evaluation policy was met.

Verdict on an order-lookup step: the agent used the general order lookup tool, and the verdict quotes the instruction that required the investigation tool

Online

TestSavant.AI judges every tool call, retrieval, and memory operation in a trace, plus plan adherence and task completion across the conversation.

Tool callsRetrievalMemoryPlan adherenceTask completion

Offline

Test agents in preproduction against thousands of generated test cases or uploaded datasets with ground-truth answers.

Generated test casesUploaded datasetsGround-truth answers

Dataset management and curation

Keep every test dataset in one searchable library, upload labeled examples, and add failed traces as new cases.

Reuse across test plans

Reuse a dataset across test plans and applications to compare results on the same cases.

Dataset library with generated, uploaded, and trace-sourced datasets, case counts, and ground-truth types

From failure to guardrail

Failed traces train your guardrails

TestSavant.AI trains a guardrail on your failed traces, so the failing behavior is blocked at runtime immediately. Engineers then correct the agent, and the next regression test cycle reruns those failures to confirm the correction.

Failed trace

A step fails its verdict, with the evidence attached.

Guardrail trained

The failing behavior is blocked at runtime.

Agent corrected

Engineers fix the agent behind the guardrail.

Correction confirmed

The next regression cycle reruns those failures.

Guardrail suggestions built from a test run's failures

Custom metrics for your application

Build evaluators in a no-code builder.

Categories and labels

Define the categories to test and the pass and violation labels, each with a risk rating.

Rubric scoring

Score responses with rubrics on criteria such as relevance and correctness, with a passing threshold per criterion.

Rubric with a passing threshold for relevance, correctness, policy grounding, and next step clarity

Automate evaluation runs

Each run applies code-based checks and LLM-as-a-judge evaluators.

Release checks

Each release gets a ready or blocked verdict from release checks on weighted pass rate, high and critical findings, suites that are out of date, and per-suite score thresholds. A blocked release fails the pipeline like any other check.

ReadyBlocked

Release marked blocked by two failed release checks, next to the trace evaluation token budget and sampling choice

Scheduled or triggered

Schedule test runs daily, weekly, or monthly, or trigger them from your CI/CD pipeline.

Daily Weekly Monthly CI/CD pipeline

Trace evaluation budget

Set a token budget for trace evaluation, then choose to judge every test-run trace, a sample of traces, or the traces with an error or a guardrail block.

Every test-run trace A sample of traces Errors and guardrail blocks

Experiment in a sandbox before a full test cycle

Connect straight to a model from any provider and test a system prompt without the rest of your application. Preview a sample of results and the projected cost, then adjust the prompt or try another model and preview again.

PreviewAdjust the promptTry another model

Sample run with projected full-run cost and latency

Version, test, and replay prompts

Know which prompt or model version scores better before a release. Tag each test run with the version under test and compare results across runs, or replay the same dataset against two system prompts or models at once.

Runs compared by prompt version and model on the same dataset, with pass rate and high and critical findings

Model-agnostic evaluation

Bring your model API keys and custom models, and connect to models from OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, or any HTTP endpoint.

OpenAI Anthropic AWS Bedrock Azure OpenAI Vertex AI Hugging Face OpenRouter Gemini Llama Mistral DeepSeek Qwen Gemma Grok Cohere Phi Nemotron Kimi GLM MiniMax Jamba Perplexity Ollama vLLM Any HTTP endpoint

Your keys, your models

Bring your model API keys and custom models.

Traces from any agent

Evaluate traces from agents built on any commercial or open-source model.

See it on your traces

Book a walkthrough and see trace evaluation on an agentic application like yours.