What is dimensionality reduction?
Dimensionality reduction is the process of describing data with fewer variables than it started with. Each variable is a dimension, so a sensor log with 400 readings per record has 400 dimensions. Many of those readings move together or carry little signal. Dimensionality reduction keeps the information that matters and drops the rest, leaving a smaller table that models and people can work with.
There are two broad approaches. Feature selection keeps a subset of the original columns. Feature extraction builds a few new variables, each a combination of the originals, which together form a low-dimensional representation of each record. Both counter the curse of dimensionality. As variables pile up, records spread thin and their distances to one another become nearly equal, so models overfit and distance-based methods such as clustering lose accuracy.
Common dimensionality reduction techniques
- Principal component analysis (PCA) finds the directions in which the data varies most and re-expresses each record along a few of them, called principal components. It is linear, fast, and the usual first step.
- t-SNE and UMAP (t-distributed stochastic neighbor embedding and uniform manifold approximation and projection) place records on a two-dimensional map for visual exploration, keeping similar records near each other.
- Autoencoders are neural networks that learn to compress records and rebuild them, which captures nonlinear patterns that PCA misses.
- Embedding models reduce text, images, and other unstructured data to vectors of a few hundred numbers.
Dimensionality reduction in practice
An injection molding plant records about 400 temperature, pressure, and timing values for every shot, far too many for anyone to watch. PCA, fitted on months of good production, reduces them to a handful of components that capture most normal variation. Each new shot then gets two numbers: how far it sits from normal within those components, and how much of it the components fail to explain. When either number crosses its control limit, an alert fires. The component weights point to the sensors responsible, such as a barrel heater zone that is drifting.
What to watch for
- Extracted variables are blends of the originals, so models built on them are harder to explain to an auditor or a plant manager.
- Standardize variables before PCA, or those measured in the largest units will dominate the components.
- Fit the reduction on training data only and apply the same transformation to new data, or test results will look better than they should.
- Cluster sizes and the gaps between clusters on a t-SNE or UMAP map carry little meaning, so use those maps for exploration rather than measurement.