What is data mining?
Data mining is the process of finding patterns, relationships, and anomalies in large existing datasets using statistical and machine learning techniques. Typical methods include clustering to find natural segments, association rules to find items that occur together, decision trees to find the conditions behind an outcome, and anomaly detection to find records that do not fit. The emphasis is on discovery in data a company already holds, which distinguishes it from building a model that scores new cases every day.
The term dates from the era of data warehouses and has largely been absorbed into data science, but the activity is unchanged. The classic process runs from understanding the business question, through preparing the data, to modeling, evaluation, and acting on the findings.
Data mining in practice
A regional grocery chain mines two years of loyalty-card transactions. Clustering reveals a group of shoppers who visit twice a week for fresh items and never buy from the center aisles. Association analysis shows which products are bought together on those trips. Store managers move high-margin complements next to the fresh departments and target a promotion at that segment. No live model is deployed. The output is a set of findings that changed decisions.
What to watch for
- With enough variables, some correlations will appear by chance, so confirm findings on a second sample before acting.
- Prefer patterns the business can act on over patterns that are merely interesting.
- Mining personal data is still processing under privacy law, even when no individual is targeted.