What is edge AI?
Edge AI is machine learning inference that runs on a local device, at the point where data is captured, rather than on a server in a data center or cloud. The device may be a smart camera, an IoT gateway, an industrial PC on a production line, a handheld scanner, or a phone. The model is trained elsewhere, usually on GPUs, then packaged and installed on the device, where it processes sensor data, images, or audio and returns a result in milliseconds.
Edge AI is the AI-specific form of edge computing. Most edge AI is computer vision: detecting defects, reading labels, counting people, or spotting safety violations. Audio models for keyword detection and small language models for offline assistants are also common.
How edge AI works
Deploying a model to a device is a distinct engineering task. The trained model is first converted to a portable format such as ONNX, then quantized to 8-bit or lower so it fits the device’s memory and runs on its accelerator. A device-specific runtime compiles it for the chip, whether that is a small GPU, an NPU, or a VPU. The model is packaged with its preprocessing code, often in a container, and pushed to devices over the air through a fleet management service that tracks which version each device runs. Devices report inference counts, confidence distributions, and a sample of inputs so the team can detect drift and schedule retraining. A new version goes to a few devices first and to the rest once it proves out.
Edge AI in practice
A packaging manufacturer runs a visual inspection system on a carton line that produces several hundred cartons a minute. A camera above the line feeds an industrial PC with a small GPU, where an object detection model checks print alignment and flap sealing on every carton and triggers a reject arm in a fraction of a second. Rejected images and a random sample of passed ones are uploaded nightly to a central server, where engineers review borderline cases and retrain the model each quarter. The line keeps running during network outages because nothing in the decision path leaves the plant floor.
What to watch for
- Test the quantized, device-compiled model rather than the original, because conversion can change accuracy in ways that only show on real inputs.
- Budget for fleet management from the start, since updating hundreds of devices by hand does not scale.
- Decide what data devices send back and how it is protected, since plant or store images can contain people and proprietary processes.
- Match the model to the device’s accelerator before buying hardware, because an unsupported operation falls back to a slow CPU path.