What Is a Neural Network? The Math, Step by Step

Quick answer: A neural network is a machine-learning model that learns a mapping from inputs to outputs. In a basic feedforward network, each neuron multiplies its inputs by weights, adds a bias and applies an activation function such as ReLU or the sigmoid; a layer of neurons is a matrix multiplication followed by an activation. Training adjusts the weights and biases to reduce prediction error, and multiple layers let the network model complex patterns, such as handwritten digits.

“Neural network” sounds like biology, but the math is short: multiply, add, and apply a simple function, many times over. Here is everything you need, with an example you can compute in a minute.

One Neuron

A neuron takes inputs \(x_1, x_2, \dots\), multiplies each by a weight, adds a bias \(b\), and applies an activation function \(\phi\):

\[ a = \phi(w_1 x_1 + w_2 x_2 + \dots + b). \]

Worked example. Inputs \(x = (1, 2)\), weights \(w = (0.5, -0.25)\), bias \(b = 0.5\):

\[ z = 0.5(1) + (-0.25)(2) + 0.5 = 0.5. \]

With the sigmoid activation \(\sigma(z) = \frac{1}{1 + e^{-z}}\), the output is \(\sigma(0.5) \approx 0.622\). With ReLU, \(\max(0, z)\), it is 0.5. (A sigmoid output is just a number between 0 and 1; it is not automatically a probability of being correct.)

Activation Functions

FunctionFormulaOutput range
Sigmoid\(\frac{1}{1 + e^{-z}}\)Between 0 and 1
ReLU\(\max(0, z)\)0 or more
Tanh\(\tanh z\)Between −1 and 1

Without these nonlinear functions, stacked layers would collapse into a single affine function (a linear map plus a shift), and the network could only draw straight-line boundaries.

Layers: Many Neurons at Once

A fully connected layer is computed all at once with a matrix: \(a = \phi(Wx + b)\), where \(W\) is \(m \times n\) for \(n\) inputs and \(m\) neurons, and each row holds one neuron’s weights (see linear algebra for AI). A network has an input layer, one or more hidden layers and an output layer.

Example: a classic digit-recognition network flattens a 28×28 grayscale image (from the MNIST dataset) into 784 inputs, passes them through hidden layers, and outputs 10 scores, one per digit 0–9; softmax turns them into probabilities, and the largest is the network’s guess.

Why Hidden Layers Matter: XOR

A single threshold neuron cannot compute XOR (“one or the other, but not both”), because no single straight line separates the points (0,1) and (1,0) from (0,0) and (1,1). A nonlinear hidden layer solves it. With one hidden layer of two neurons, a network can. This limitation of single-layer perceptrons, highlighted by Minsky and Papert in 1969, is why depth matters. In fact, networks with one hidden layer can approximate any continuous function on a bounded region, given enough neurons (the universal approximation theorem).

How a Network Learns

  1. Forward pass: compute the output from the input.
  2. Loss: measure how far the output is from the right answer.
  3. Backpropagation: use the chain rule to calculate how each weight affects the loss.
  4. Update: an optimizer such as gradient descent adjusts each weight and bias a little (see gradient descent explained).

One learning step by hand. Take our ReLU neuron (output 0.5) with target 1 and loss \(L = \tfrac{1}{2}(a – 1)^2 = 0.125\). The slope of the loss with respect to the output is \(a – 1 = -0.5\), so the gradients are \(-0.5 \times (1, 2) = (-0.5, -1)\) for the weights and \(-0.5\) for the bias. With learning rate 0.1, the new weights are \((0.55, -0.15)\) and the new bias is 0.55. The new output is \(0.55 – 0.30 + 0.55 = 0.8\) and the loss drops to 0.02. Repeat over many examples, and the network improves. A good network is then checked on new data it never trained on, to make sure it has not just memorized (overfitting).

A Short History

  • 1943: Warren McCulloch and Walter Pitts describe artificial neurons.
  • 1958: Frank Rosenblatt’s perceptron learns simple patterns.
  • 1986: Rumelhart, Hinton and Williams popularize backpropagation.
  • 1969: Marvin Minsky and Seymour Papert show the limits of single-layer perceptrons.
  • 2012: AlexNet, a deep network, wins the ImageNet image-recognition competition by a wide margin.
  • 2017–today: the transformer, introduced in 2017, powers chatbots (see the math behind ChatGPT).

Common Misconceptions

“Neural networks work like brains.” They were loosely inspired by neurons, but the math is far simpler than biology.

“Each neuron stores a fact.” Knowledge is spread across many weights.

“More layers is always better.” Deeper networks need more data and care to train.

Watch: Neural Networks, Visualized

If you learn better by watching, these videos are worth your time.

But what is a neural network? (3Blue1Brown). The classic visual introduction, using handwritten digits.
Gradient descent, how neural networks learn (3Blue1Brown). How the weights get trained.

Check Your Understanding

Five quick questions to see how much of this has stuck. Pick an answer and press Submit; you will see a short explanation either way.

Answer each question and press Submit to see your result right away.

1. Inputs (1, 2), weights (0.5, −0.25), bias 0.5. What is the weighted sum z?
Found a bug? Let us know!
2. What is ReLU(−3)?
Found a bug? Let us know!
3. After one gradient step in the worked example, what is the new output?
Found a bug? Let us know!
4. Which simple function can a single neuron NOT compute?
Found a bug? Let us know!
5. How many inputs does a 28×28 image network have?
Found a bug? Let us know!

Frequently Asked Questions

What is a neural network in simple terms?

A function made of many simple units called neurons. Each multiplies its inputs by weights, adds a bias and applies an activation function; layers of neurons are trained with gradient descent to match examples.

What math do neural networks use?

Linear algebra (weighted sums and matrix multiplication), simple nonlinear functions (activations), calculus (the chain rule and gradient descent) and probability for outputs.

What is an activation function?

A nonlinear function, such as ReLU or the sigmoid, applied to each neuron’s weighted sum. It lets networks learn curved, complex patterns.

What are weights and biases?

Weights are the numbers that multiply each input; the bias is a number added to the total. Training adjusts both.

Who invented neural networks?

Warren McCulloch and Walter Pitts described artificial neurons in 1943, Frank Rosenblatt built the perceptron in 1958, and backpropagation was popularized in 1986 by Rumelhart, Hinton and Williams.

Keep Exploring

Sources and Further Reading

Related to This Article

What people say about "What Is a Neural Network? The Math, Step by Step - Effortless Math"?

No one replied yet.

Leave a Reply