What Is a Neural Network? The Math, Step by Step
Quick answer: A neural network is a machine-learning model that learns a mapping from inputs to outputs. In a basic feedforward network, each neuron multiplies its inputs by weights, adds a bias and applies an activation function such as ReLU or the sigmoid; a layer of neurons is a matrix multiplication followed by an activation. Training adjusts the weights and biases to reduce prediction error, and multiple layers let the network model complex patterns, such as handwritten digits.
“Neural network” sounds like biology, but the math is short: multiply, add, and apply a simple function, many times over. Here is everything you need, with an example you can compute in a minute.
One Neuron
A neuron takes inputs \(x_1, x_2, \dots\), multiplies each by a weight, adds a bias \(b\), and applies an activation function \(\phi\):
\[ a = \phi(w_1 x_1 + w_2 x_2 + \dots + b). \]Worked example. Inputs \(x = (1, 2)\), weights \(w = (0.5, -0.25)\), bias \(b = 0.5\):
\[ z = 0.5(1) + (-0.25)(2) + 0.5 = 0.5. \]With the sigmoid activation \(\sigma(z) = \frac{1}{1 + e^{-z}}\), the output is \(\sigma(0.5) \approx 0.622\). With ReLU, \(\max(0, z)\), it is 0.5. (A sigmoid output is just a number between 0 and 1; it is not automatically a probability of being correct.)
Activation Functions
| Function | Formula | Output range |
|---|---|---|
| Sigmoid | \(\frac{1}{1 + e^{-z}}\) | Between 0 and 1 |
| ReLU | \(\max(0, z)\) | 0 or more |
| Tanh | \(\tanh z\) | Between −1 and 1 |
Without these nonlinear functions, stacked layers would collapse into a single affine function (a linear map plus a shift), and the network could only draw straight-line boundaries.
Layers: Many Neurons at Once
A fully connected layer is computed all at once with a matrix: \(a = \phi(Wx + b)\), where \(W\) is \(m \times n\) for \(n\) inputs and \(m\) neurons, and each row holds one neuron’s weights (see linear algebra for AI). A network has an input layer, one or more hidden layers and an output layer.
Example: a classic digit-recognition network flattens a 28×28 grayscale image (from the MNIST dataset) into 784 inputs, passes them through hidden layers, and outputs 10 scores, one per digit 0–9; softmax turns them into probabilities, and the largest is the network’s guess.
Why Hidden Layers Matter: XOR
A single threshold neuron cannot compute XOR (“one or the other, but not both”), because no single straight line separates the points (0,1) and (1,0) from (0,0) and (1,1). A nonlinear hidden layer solves it. With one hidden layer of two neurons, a network can. This limitation of single-layer perceptrons, highlighted by Minsky and Papert in 1969, is why depth matters. In fact, networks with one hidden layer can approximate any continuous function on a bounded region, given enough neurons (the universal approximation theorem).
How a Network Learns
- Forward pass: compute the output from the input.
- Loss: measure how far the output is from the right answer.
- Backpropagation: use the chain rule to calculate how each weight affects the loss.
- Update: an optimizer such as gradient descent adjusts each weight and bias a little (see gradient descent explained).
One learning step by hand. Take our ReLU neuron (output 0.5) with target 1 and loss \(L = \tfrac{1}{2}(a – 1)^2 = 0.125\). The slope of the loss with respect to the output is \(a – 1 = -0.5\), so the gradients are \(-0.5 \times (1, 2) = (-0.5, -1)\) for the weights and \(-0.5\) for the bias. With learning rate 0.1, the new weights are \((0.55, -0.15)\) and the new bias is 0.55. The new output is \(0.55 – 0.30 + 0.55 = 0.8\) and the loss drops to 0.02. Repeat over many examples, and the network improves. A good network is then checked on new data it never trained on, to make sure it has not just memorized (overfitting).
A Short History
- 1943: Warren McCulloch and Walter Pitts describe artificial neurons.
- 1958: Frank Rosenblatt’s perceptron learns simple patterns.
- 1986: Rumelhart, Hinton and Williams popularize backpropagation.
- 1969: Marvin Minsky and Seymour Papert show the limits of single-layer perceptrons.
- 2012: AlexNet, a deep network, wins the ImageNet image-recognition competition by a wide margin.
- 2017–today: the transformer, introduced in 2017, powers chatbots (see the math behind ChatGPT).
Common Misconceptions
“Neural networks work like brains.” They were loosely inspired by neurons, but the math is far simpler than biology.
“Each neuron stores a fact.” Knowledge is spread across many weights.
“More layers is always better.” Deeper networks need more data and care to train.
Watch: Neural Networks, Visualized
If you learn better by watching, these videos are worth your time.
Check Your Understanding
Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.
Answer each question and press Submit to see your result right away.
Frequently Asked Questions
What is a neural network in simple terms?
A function made of many simple units called neurons. Each multiplies its inputs by weights, adds a bias and applies an activation function; layers of neurons are trained with gradient descent to match examples.
What math do neural networks use?
Linear algebra (weighted sums and matrix multiplication), simple nonlinear functions (activations), calculus (the chain rule and gradient descent) and probability for outputs.
What is an activation function?
A nonlinear function, such as ReLU or the sigmoid, applied to each neuron’s weighted sum. It lets networks learn curved, complex patterns.
What are weights and biases?
Weights are the numbers that multiply each input; the bias is a number added to the total. Training adjusts both.
Who invented neural networks?
Warren McCulloch and Walter Pitts described artificial neurons in 1943, Frank Rosenblatt built the perceptron in 1958, and backpropagation was popularized in 1986 by Rumelhart, Hinton and Williams.
Keep Exploring
- Linear Algebra for AI: Vectors, Matrices and Examples
- Gradient Descent Explained Simply: How AI Learns
- The Math Behind ChatGPT: Explained With Examples
Sources and Further Reading
- Rosenblatt, The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain (Psychological Review, 1958)
- Rumelhart, Hinton and Williams, Learning representations by back-propagating errors (Nature, 1986)
- Goodfellow, Bengio and Courville, Deep Learning, Chapter 6: Deep Feedforward Networks (MIT Press)
Related to This Article
More math articles
- FHA Loan Calculator: Monthly Payment, MIP and Down Payment
- Chapter 19: Freedom of Religion
- TDEE Calculator for Women
- What is the Highest ASVAB Score?
- Earmarks
- 4th Grade KAP Math Worksheets: FREE & Printable
- How is the FTCE General Knowledge Test Scored?
- The Ultimate 7th Grade MCAS Math Course (+FREE Worksheets)
- Unlocking the Secrets of Curves: Higher Order Derivatives in Graph Analysis
- Navy Body Fat Calculator


What people say about "What Is a Neural Network? The Math, Step by Step - Effortless Math"?
No one replied yet.