What is a transformer?
A transformer is a neural network architecture for sequences that replaces step-by-step recurrence with attention. Rather than reading a sentence one word at a time, a transformer takes in the whole sequence and lets every position look at every other position to decide what matters. It was introduced in the research paper “Attention Is All You Need” for machine translation and quickly became the dominant architecture for language, then for images, audio, and protein structures.
Every large language model in use today, including GPT and Claude, is a transformer. So are BERT-style models that drive search and classification, and the vision transformers used in some image systems. When a vendor says “LLM,” the architecture underneath is a transformer.
How a transformer works
Text is first split into tokens, and each token is converted into an embedding, a list of numbers representing its meaning. Because the model sees all tokens at once, a positional encoding is added so it knows the order. The sequence then passes through a stack of identical blocks. Each block has two parts: a self-attention layer, in which every token computes how much to attend to every other token and blends in their information, and a feedforward layer that processes each token’s updated representation. Multiple attention heads run in parallel within a layer, each free to learn a different kind of relationship, such as grammar, coreference, or topic.
In a decoder-only model such as GPT, the final layer outputs a probability for every possible next token, and generation proceeds one token at a time, feeding each choice back in. Training is parallel across the whole sequence, which is what let transformers scale to sizes that recurrent networks never reached.
Types of transformer
| Type | How it reads | Typical use | Example |
|---|---|---|---|
| Encoder-only | Whole input at once, in both directions | Classification, search embeddings, entity extraction | BERT |
| Decoder-only | Left to right, predicting the next token | Chat, text generation, coding agents | GPT |
| Encoder-decoder | Encodes the input, then generates the output | Translation, summarization | Machine translation systems |
Why the transformer matters
The transformer made it practical to pre-train one very large model on vast amounts of unlabeled text and then adapt it to many tasks. That is the foundation model pattern behind generative AI. For a business, it means the same architecture that drafts an email can also classify support tickets, extract fields from invoices, and answer questions over a document store, with the differences lying in training data and prompting rather than in the model’s design.
Transformers and private AI
Because the architecture is public and many strong models are released with open weights, a company can run a transformer on hardware it controls. The sizing questions follow from the architecture: the number of parameters determines GPU memory, the context window determines how much text fits in one request, and the attention computation grows with the square of the input length. Kirk Tech Solutions typically sizes private deployments by starting from the longest documents the business needs the model to read and working back to the GPU memory and quantization level required.
What to watch for
- Attention cost grows quickly with input length, so very long documents need chunking, retrieval, or a model built for long context.
- A transformer predicts plausible text, which is why hallucination exists, so ground it in retrieved documents when facts matter.
- Model size drives memory and cost. A smaller model fine-tuned for the task often beats a larger general one on both accuracy and latency.
- The same architecture serves many tasks, so evaluate on your task with your data rather than on published benchmarks.