Interpretability Testing
Interpretability testing evaluates whether a model's internal decision process can be inspected and understood at all — through techniques like attention visualization, feature attribution, or probing intermediate representations — as distinct from explainability, which is about the model's own stated justification for an output.
Interpretability and explainability get used interchangeably, but they're different things to test. Explainability tests the model's self-reported reasoning; interpretability tests whether an independent, external technique can inspect what the model is actually doing internally — which attention heads activated, which input tokens most influenced a given output, which layer's representation correlates with a known concept.
For most product QA teams this shows up less as direct testing and more as a tooling decision: choosing whether a model architecture and deployment setup even supports interpretability tooling (attention maps, SHAP/LIME-style attribution, embedding probes) before committing to it for a use case where "why did it do that" will eventually need a real answer, not a black box.
Where it is directly testable, the check is usually whether an attribution method consistently points to the input region a human would agree is actually responsible for the output — image classifiers are a common example, checking that a "cat" prediction's saliency map actually highlights the cat and not the background.
