If your AI system denies a loan, flags an intake, or blocks an agentic action, could you produce a clear, human-readable explanation that stands up to a regulator, a judge, and the person impacted—without revealing trade secrets—today? If the answer is “not reliably,” you don’t have a model problem. You have a governance-to-runtime gap.
I’m sharing notes after reading a strong explainability guide from the Norton Rose Fulbright team (EU/UK/AU focus). It crisply lays out what the law already expects and what’s coming next. Rather than summarize the whole piece, I’ll translate the legal obligations into operational practices I see working in production—especially for agentic AI, RAG copilots, decision support tools, and safety/guardrail layers.
Hat tip to the authors of the guide; link at the end. This post is about the “how.”
The short version
- The duty to explain is here. Under the GDPR (notably Article 22) and the EU AI Act’s “right to an explanation,” deployers (the entities using AI) owe affected people meaningful information about the role AI played and the reasons for outcomes—regardless of who built the tech.
- “Human in the loop” is not a cheat code. Oversight must be meaningful: a human with competence, authority, and time to review the specifics—not rubber-stamping.
- Black box is context-dependent, not forbidden. But “trust us, it works” is over. If you can’t produce a clear rationale for specific decisions and a process narrative for governance, you’re exposed.
- Procurement and runtime need to meet. Most organizations have policies and model cards. Fewer have per-decision artifacts at runtime that legal/compliance can actually use.
What “meaningful information” really means (in practice)
The law splits explanation into two buckets:
- Outcome-based explanations (the “why” for this decision).
What inputs mattered, how they were used, and the rationale for the outcome. For decisions that significantly affect people—credit, employment, healthcare—you need something intelligible to the recipient and auditable by regulators. - Process-based explanations (the “how” for your governance).
What data you collected, how you prepared and validated it, which model(s) you chose and why, guardrails, testing, human oversight, logging, contestation procedures, and how you monitor drift and incidents over time.
You need both—per decision and across the lifecycle. A tidy policy document without runtime artifacts is theater. A pretty per-decision tooltip without governance plumbing is a liability.
The runtime artifact your program actually needs
Organizations that are doing this well create a Decision Audit Object for each consequential outcome. Think of it as a compact, rights-sized dossier you can hand to legal/compliance and—if appropriate—summarize for the individual.
A useful Decision Audit Object tends to include:
- Context: use case, system name/version, risk tier, decision timestamp, request ID.
- Input summary: categories of inputs used (with links to records if permissible), data sources, and any redaction/normalization.
- Model/agent participation: which components contributed (RAG retriever, policy engine, LLM, classifier, tool-use chain), including versions.
- Guardrail rationale: why the guardrail allowed/blocked/flagged. For content attacks (e.g., prompt injection), this can include token-level or feature-level contribution so reviewers see what tipped the decision.
- Confidence and sensitivity: uncertainty indicators and notes on any special category data.
- Human oversight: who reviewed, what they changed (if anything), and the basis for override/affirmation.
- Contestability hooks: how a recipient can challenge the decision, and how your team logs and handles disputes.
- Chain-of-custody: cryptographic or tamper-evident logging so you can prove nothing was altered after the fact.
Kept small, this object can live inline with your case management, risk, or CRM tools. For highly regulated decisions, it anchors your legal obligations in something concrete.
“Meaningful human involvement” that actually is
To clear the bar regulators describe, your human oversight needs three ingredients:
- Competence: reviewers trained on the domain and on how the AI/guardrail works. Not a help-desk script.
- Authority: power to overturn the decision and trigger either remediation or escalation.
- Time + signals: the decision record must be legible—not a model dump. Present attributions, reasons, and comparable precedents so the human can act like a human, not a checkbox.
This is where runtime explainability earns its keep: if reviewers get a one-line label (“blocked: injection risk”), they can’t exercise judgment. If they see the rationale (“blocked due to cross-domain goal hijack; strongest contributors: {‘ignore previous instructions’: 0.31, ‘new system prompt’: 0.22, ‘exfiltrate’: 0.18}”), they can audit, teach, and defend the decision.
Don’t confuse model interpretability with deployer explainability
Interpretable models are great where feasible. But deployer obligations don’t require you to open the model’s internals; they require giving people reasons they can use. In many cases, guardrail-level explanations and policy-level rationales are the right layer:
- “Your resume was not advanced because key qualifications A/B were missing and the role requires them by policy.”
- “Your loan was declined because debt-to-income exceeded X% and the risk score crossed threshold Y, which triggers manual review under our credit policy.”
- “Your message was blocked because it attempted to override system instructions (top contributing phrases listed).”
The point is to make your decision policy and guardrail logic visible and usable—without dumping proprietary gradients or vendor source.
What about agentic AI?
Agent frameworks amplify the need for explanations because actions cascade. You’ll want:
- Intent traces: what the agent intended to do (goal, sub-goals) vs. what it did.
- Tool-use ledger: which tools were called, with parameters and outputs.
- Guardrail breakpoints: where the policy engine intervened and why.
- Roll-back semantics: what was reverted, when, and how you proved it.
Again, the deployer obligation is the north star: if an action significantly affects someone, you need to explain the role the AI played and the main elements of the decision. Agent logs should make that trivial.
Procurement is governance
If you buy instead of build, your Request for Information (RFI) should anticipate your legal duties. Ask vendors for a transparency package that covers:
- Per-decision explanations: examples of rationale objects for your use case.
- Guardrail introspection: how the system explains allows/blocks/flags; whether it supports token- or feature-level attributions.
- Human-in-the-loop workflows: how you can override, log reasons, and notify impacted parties.
- Data handling and confidentiality: how you’ll get explainability data without exposing trade secrets or third-party PII.
- Testing artifacts: adversarial testing reports, coverage metrics, and incident postmortems.
- Change control: how explanations keep working across model upgrades.
If a vendor claims a “low-risk derogation” under the AI Act to avoid providing transparency, you’ll still need enough information to meet your own deployer obligations. That’s a governance call, not a marketing one.
Common failure modes (and how to avoid them)
- Human-in-the-loop theater: adding a reviewer who sees nothing but a binary label. Fix: ship the Decision Audit Object with real signals.
- Explainability washing: model cards with no runtime evidence. Fix: bind per-decision artifacts to your cases, not to a slide deck.
- Over-disclosure or gibberish: drowning recipients in math or internal code. Fix: present reasons in plain language for the audience, then link deeper for auditors.
- Policy/procedure drift: great day-one SOPs, then entropy. Fix: treat explanations like SLAs and monitor them.
- One-off exceptions: “We’ll explain if someone asks.” Fix: build explanations by default; redact or escalate when sensitive.
Metrics that matter
If you don’t measure explanation quality, it will decay. A practical dashboard includes:
- Time-to-Explain (TTE): median time to assemble a compliant rationale.
- Coverage: % of consequential decisions with a Decision Audit Object attached.
- Contest rate & overturn rate: and the top reasons for overturn. These are gold for both fairness and security.
- Reviewer burden: minutes per review; goal is low enough to be used, high enough to be thoughtful.
- Drift in rationale patterns: sudden shifts can reveal model or data issues before KPIs move.
- Guardrail precision/recall: measured on your adversarial test sets; not just “blocked a lot.”
A concrete runtime example (attack prevention + explainability)
Say an employee pastes a third-party prompt into an internal copilot that silently tries to override system instructions and exfiltrate sensitive tables.
A mature runtime will:
- Detect and attribute: flag instruction-hijack and exfiltration intent. Provide token-level contribution (e.g., “ignore previous instructions,” “system prompt,” “export,” “customers.csv”).
- Rationalize: produce a human-readable reason: “Blocked: attempted instruction override + data exfiltration. Top contributing phrases listed. Policy R-12 requires block and alert.”
- Record oversight: route to a reviewer with authority. They see reasons, examples, and policy references; they can affirm or override.
- Preserve evidence: write a tamper-evident log including inputs (appropriately redacted), rationale, reviewer decision, and notifications.
- Close the loop: if reviewers routinely override a pattern, update the policy/guardrail; if they affirm, expand test coverage.
That single flow satisfies security, explainability, and contestability without exposing proprietary model internals.
Where TestSavant fits
My team works the runtime side of this problem. We continuously adversarial-test LLMs/agents and attach per-decision rationales to guardrail outcomes—down to token-level attributions for things like prompt injection, goal hijack, and data exfiltration attempts. The aim isn’t “sales.” It’s to make it trivial for legal, compliance, and risk to show what the system saw, why it acted, and how a human could contest or override—in language a regulator can use.
Different tools can do this; the point is the pattern: runtime evidence as a first-class artifact.
A practical 10-step plan for deployers
- Risk-tier your use cases. Decide which decisions are “consequential” and require explanations by default.
- Define your Decision Audit Object schema. Start small; expand as regulators and stakeholders demand.
- Bind explanations to cases. Store them where humans already work (CRM, case, claims, HRIS), not in a silo.
- Operationalize contestability. Clear paths to challenge, with clocks and owners.
- Instrument guardrails. Ensure they produce human-readable rationales and (where appropriate) token/feature attributions.
- Train reviewers. Teach both the domain and how to interpret the rationale and attributions.
- Set explainability SLAs. Time-to-Explain, Coverage, and Overturn metrics on a real dashboard.
- Adversarial test continuously. Use up-to-date attack sets and measure guardrail precision/recall on them.
- Upgrade procurement. Demand transparency packages; contract for the data you need to meet your obligations.
- Audit trails you can prove. Tamper-evident logs with chain-of-custody so evidence holds up under scrutiny.
Credit, context, and next steps
The legal analysis that sparked this post—covering GDPR, the EU AI Act’s right to an explanation, the UK’s in-flight changes, regulator guidance, and case law—is excellent work by the Norton Rose Fulbright team. It underscores a simple reality: explainability isn’t a slide. It’s a per-decision artifact and a habit.
I’ll link their full piece here for those who want the doctrinal view. My goal was to translate it into a runtime blueprint you can adopt this quarter.
If your org is building this capability—whether through internal tooling or a vendor—I’m happy to swap notes and share examples of Decision Audit Objects and review workflows. The destination is the same for all of us: decisions people can understand, contest, and trust.
