Explainability Testing
Explainability testing checks whether a model can produce a human-understandable account of why it reached a particular output — a rationale, a cited source, or a feature-importance breakdown — and whether that account is actually accurate, rather than a plausible-sounding story generated after the fact.
The trap here is confusing an explanation that sounds reasonable with an explanation that's actually true. A model asked to justify its own decision will often generate a fluent, confident-sounding rationale that doesn't correspond to the real internal factors that produced the output — sometimes called "post-hoc rationalization" — and it's a distinct failure mode from a wrong answer, because the explanation itself can pass a surface read while being fabricated.
Testing it well usually means constructing cases where the true driving factor is known in advance — a feature deliberately correlated with the label, a document deliberately inserted into a RAG context — and checking whether the model's stated explanation actually names that factor, rather than just checking that an explanation was produced at all.
For regulated domains (credit decisions, hiring, medical triage) this is often a compliance requirement, not just a quality nice-to-have, so the test suite needs to demonstrate the explanation is faithful to the decision, not merely present.
