What is information leakage?
Information leakage, also called data leakage, is the inclusion in training data of information that would not exist at the moment a prediction is actually made. The model learns to rely on that information, scores very well in testing because the test data contains it too, and then fails in production because the information is missing there. It is one of the most common reasons a model that looked excellent never delivers.
Leakage takes two main forms. Target leakage is a feature that is a consequence of the outcome rather than a cause. A field such as “collections agency assigned” predicts loan default very well because it is only populated after the default. Train-test contamination is the second form: the same customer, document, or day appears in both training and test data, or a preprocessing step such as normalization was fitted on the full dataset before the split.
Why information leakage matters
Leakage produces the most convincing kind of wrong answer. The test score is real, the method was followed, and nothing fails until the model is live. By then, staff have been trained on the new process and the budget is spent. Leakage is also hard to see from outside, which makes it a question to raise during model validation and in any vendor’s proof of concept on the company’s data.
Information leakage in practice
A collections agency builds a model to predict which accounts will pay within 90 days. The training data is a snapshot of the account system taken today, and it includes a “last payment date” field. For accounts that paid, that field is the proof of payment. The model learns to read it and reports near-perfect results. In production, scoring accounts on the day they are placed, the field is empty for everyone. The fix is to rebuild every feature as it would have looked on the placement date, which usually requires data lineage good enough to reconstruct historical states.
In Kirk Tech Solutions engagements, every feature in a candidate model is reviewed for the point in the business process at which it becomes available, before the model is accepted.
What to watch for
- Treat any feature that predicts the outcome almost perfectly as a leak until proven otherwise.
- Build each training record from data as it stood at the prediction time, using point-in-time snapshots.
- Split data by time, and by customer or entity, before any preprocessing is fitted.
- Ask a vendor to list every feature in the model and to state when each one becomes available.