What is active learning?
Active learning is a way of training a model in which the model chooses which examples get labeled next. Instead of paying people to label a random sample, the team trains a first model on a small labeled set, runs it over the unlabeled pool, and sends the examples with the lowest confidence score to a human. Those new labels are added, the model is retrained, and the cycle repeats.
The selection rule can be uncertainty (the model’s top two answers are nearly tied), disagreement among several models, or the expected effect on the model’s performance. Uncertainty sampling is the most common because it is simple to implement.
Active learning in practice
A law firm is training a model to flag indemnification clauses in supplier contracts. Labeling requires an associate’s time, so random labeling of fifty thousand clauses is out of the question. With active learning, the associate reviews a few hundred clauses the model finds ambiguous, and the model reaches the target accuracy with a fraction of the labeling effort that random sampling would have taken.
What to watch for
- The resulting labeled set is biased toward hard cases, so measure accuracy on a separately drawn random test set.
- Rare classes the first model never predicts may never be selected, so seed them by hand.
- Retraining after every small batch of labels adds engineering work, which is only worth it when labels are expensive.