What is interpretability?
Interpretability is the degree to which a person can follow how a model reaches its outputs by reading the model itself. A logistic regression with a dozen inputs is interpretable, because a reviewer can see exactly how much each input, such as a late payment, moves the score. A shallow decision tree is interpretable because every prediction follows a short path of yes-or-no rules. Points-based scorecards, rule lists, and generalized additive models belong to the same family.
Interpretability is a matter of degree. A logistic regression with four hundred engineered features is readable in principle and opaque in practice. Deep neural networks and large language models sit at the far end, with billions of parameters that carry no individual meaning. Researchers in mechanistic interpretability study what the internal components of these networks represent, which helps model builders but does not give a business a decision-by-decision account that an auditor could read.
Interpretability compared with explainability
The two words are often used interchangeably, and the difference matters when reviewing a vendor’s claims. Interpretability is a property of the model: its own structure serves as the explanation. Explainable AI (XAI) is a set of methods, such as Shapley values and counterfactual explanations, applied after training to a model that cannot be read directly. Those methods approximate the model’s reasoning, while an interpretable model leaves no gap between what it does and what the explanation says.
Interpretability in practice
A county building department ranks properties for inspection using complaint history, permit records, and building age. Its analysts test a gradient-boosted model and a points-based scorecard against past inspection results. The scorecard comes close to the complex model, so the department adopts it. A prior code violation adds a set number of points and an open complaint adds more, so when an owner asks why a building was chosen, the answer is on the scorecard. On structured data like this, a well-built interpretable model often performs close to a complex one, while the gap is usually wider for images, audio, and free text.
What to watch for
- Ask a vendor proposing a complex model to show how a simple interpretable model performs on the same data, so the trade-off is measured.
- Count the features and interactions, because interpretability fades as a model grows even when its form stays simple.
- Check readable models for unfair inputs, such as ZIP code acting as a proxy for a protected characteristic, which their transparency makes easier to catch.
- Keep research claims about the internals of large language models separate from the decision records an auditor or regulator expects.