The AI Assurance Operating Manual
Traditional QA methods fall short on AI testing. This manual helps organizations build an AI testing program covering agentic, RAG, and chatbot applications.
What you get inside
Templates and tools your team can apply today.
- A QA readiness checklist to benchmark your current program.
- An intake template for onboarding any new AI system.
- 25 discovery questions to assess your AI testing readiness.
- A sample release scorecard to measure your release.
Who this is for
Built for organizations tasked with testing AI
- QA leaders who need to test the AI features on their roadmap.
- Teams testing agentic, RAG, and chatbot AI applications.
- Practitioners who need a repeatable way to catch AI regressions before release.
- Teams already testing AI who want to operationalize the practice.
How this informs your testing program
Regression on every release
AI output is not deterministic. The model that passed QA last sprint may produce different answers this sprint. Run structured regression on every release and catch drift before users do.
- Known failure replay to catch regressions on previously found issues
- Instruction adherence and refusal behavior after any prompt or model revision
- Retrieval precision and citation accuracy after any KB change
- Tool-use reliability, permissions, and error handling
Every AI application type
Agentic workflows, RAG systems, and chatbots each carry distinct failure modes, test categories, and release blockers.
A test plan built for AI
Cover the dimensions that decide whether your AI is ready, from task success and grounding to safety, privacy, and tool use.
A defensible release decision
Defend the decision to release or hold back a release, with the reasoning behind every call.
The playbook for testing AI, from coverage to regression to release readiness
