What is anomaly detection?
Anomaly detection finds data that does not fit the normal pattern. The anomaly can be a single extreme reading, such as a pressure spike. It can be a combination of values that are each ordinary on their own, such as a routine-sized payment sent to a new account at 3 a.m. It can also be a change over time, such as a meter that stops varying because its sensor has frozen.
Most methods learn what normal looks like and then score how far each new record departs from it. Anomalies are rare and often unlike anything seen before, so labeled examples are scarce. For that reason, most anomaly detection relies on unsupervised learning.
Common anomaly detection techniques
- Statistical thresholds flag values more than a set number of standard deviations from a rolling average, and they are the right starting point for a single measurement.
- Isolation forests build random decision trees and score how quickly each record is isolated, since unusual records separate in fewer splits.
- Distance and clustering methods flag records that sit far from any cluster or from their nearest neighbors.
- Autoencoders learn to compress and rebuild normal data, so a large reconstruction error marks something unfamiliar.
- Forecast residuals compare each new value in a time series with its forecast and flag gaps larger than the usual forecast error.
Anomaly detection in practice
A food processor monitors vibration and temperature on the refrigeration compressors for its cold storage. A model trained on months of normal operation scores each compressor every minute. When a bearing starts to wear, the vibration pattern drifts before any single reading crosses an alarm limit. The maintenance team gets a work order days before a failure would spoil inventory, which is the basis of predictive maintenance. Banks apply the same approach to fraud detection on card payments. Accounts payable teams use it to catch duplicate invoices, and managed service providers use it to spot unusual logins.
In Kirk Tech Solutions engagements, each alert lands in a review queue with its supporting data attached, and a person decides whether to hold a payment or stop a line.
What to watch for
- Set alert thresholds with the people who respond, because a flood of false positives teaches staff to ignore alerts.
- Normal behavior changes with seasons, product mix, and new equipment, so refresh the baseline on a schedule.
- Record which alerts led to action and feed those outcomes back into tuning.
- Keep a set of confirmed past incidents to test the detector, even when training is unsupervised.