What is a modality?
A modality is a distinct type of data that an AI model can accept as input or generate as output. Text, images, audio, and video are the familiar ones, and sensor time series, tables, and 3D scans count as well. Each has its own structure. Text is a sequence of tokens, an image is a grid of pixels, audio is a waveform sampled thousands of times per second, and video adds time to images. A model needs a matching encoder to turn each modality into numbers it can process.
Models are described by their modalities. A text-only language model takes and returns text. A vision-language model accepts images and text and returns text. Speech recognition turns audio into text, and text-to-speech does the reverse. Multimodal AI combines several modalities in one model or system.
Why modality matters
Listing the modalities involved is an early step in choosing a model for a use case. A machine shop’s quality team works with inspection photos, spindle vibration readings, and operators’ shift notes. A text-only model forces the photos and readings through a conversion step first. Each conversion loses something: a caption omits detail an inspector would notice, and a text summary of vibration data flattens the pattern over time. Optical character recognition (OCR) loses a form’s layout in the same way, and a call transcript loses tone of voice. Checking which modalities a model handles natively, for input and for output, shows how much of the original evidence reaches it.