What is an AI accelerator (NPU, VPU)?
An AI accelerator is a chip, or a dedicated block inside a larger chip, designed to run neural network inference with far less power than a CPU or GPU would use. A neural processing unit (NPU) is the accelerator built into current laptop and phone processors, and it runs tasks such as speech recognition, background blur, and small language models without draining the battery. A vision processing unit (VPU) is an accelerator tuned for image and video work, found in smart cameras, drones, and industrial inspection systems.
Accelerators trade flexibility for efficiency. They run inference only, support a limited set of operations and number formats, and need models converted into their own format, usually after quantization to 8-bit or lower. They cannot train models or hold large language models that need tens of gigabytes of memory.
AI accelerator in practice
A grocery chain installs cameras at self-checkout lanes to flag incorrectly scanned items. Each camera contains a VPU running a small object detection model, so the decision is made at the lane in a fraction of a second and no video leaves the store. Only flagged events, as short clips and metadata, go to the central system.
What to watch for
- Each accelerator family has its own toolchain, and a model must be converted and tested for it.
- Check the supported operations before choosing a model, because an unsupported layer falls back to the slow CPU path.
- Accelerators handle small models well, and anything larger still belongs on a GPU or a server.