What is BERT?
BERT, short for Bidirectional Encoder Representations from Transformers, is a language model from Google researchers that reads a passage in both directions at once to build a representation of each word in its context. It is pre-trained on large amounts of text by hiding words and learning to predict them, and is then fine-tuned with a small added layer for a specific task. BERT is an encoder-only transformer, and unlike GPT-style models it does not generate text.
Its release changed natural language processing by showing that one pre-trained model could be adapted to many tasks with little labeled data. Variants followed, including smaller distilled versions and domain-specific versions for biomedical and financial text. The sentence embedding models used in semantic search descend from the same design.
Why BERT matters
BERT-class models remain the practical choice for narrow, high-volume text tasks. An insurer routes tens of thousands of inbound emails a day to the right department with a fine-tuned BERT model that runs on ordinary CPUs and answers in milliseconds. For classification, entity extraction, and ranking, a fine-tuned small encoder often matches a large language model’s accuracy at a small fraction of the compute and is far easier to run privately.
What to watch for
- BERT needs labeled examples for fine-tuning, typically a few hundred to a few thousand per task.
- Its input length is limited to a few hundred tokens, so long documents must be split.
- Choose a domain or multilingual variant when the text differs from general web English.