What is an activation function?
An activation function is a simple mathematical rule applied to the output of each unit in a neural network after it sums its weighted inputs. Without it, every layer would be a linear transformation and stacking layers would gain nothing, because a chain of linear steps is itself linear. The activation function introduces a bend, and many bends stacked across layers let the network approximate irregular relationships.
A few functions cover most uses. ReLU (rectified linear unit) outputs zero for negative inputs and passes positive inputs unchanged; it is cheap to compute and the default in many architectures. GELU (Gaussian error linear unit) is a smoother variant used in transformer models. The sigmoid squashes any value into the range between zero and one, which makes it the natural choice for an output that represents a probability, as in logistic regression. Softmax generalizes that to several classes so the outputs sum to one.
Why activation functions matter
Activation functions are a design choice inside the model architecture, made by whoever designs the architecture rather than by the team that fine-tunes or buys the model. The one worth understanding is the activation on the output layer, because it tells you what the model produces. A sigmoid output is a probability for one class, a softmax output is a distribution over several, and a linear output is a plain number such as a forecast. That tells you whether a vendor’s scores can be read as probabilities or only as rankings.