1. Biological Inspiration

The human brain contains approximately 86 billion neurons, each connected to thousands of others. A neuron receives electrical signals through dendrites, processes them in the cell body, and — if the combined signal exceeds a threshold — fires an output signal through its axon to other neurons.

Artificial neural networks are a mathematical abstraction of this process. An artificial neuron receives numerical inputs, multiplies each by a weight (strength of connection), sums them up, adds a bias, and passes the result through an activation function to produce an output. Learning occurs by adjusting the weights — just as the brain strengthens synaptic connections through repeated use.

Important: Modern neural networks are engineering tools, not accurate models of the brain. The biological analogy is motivational, not literal.

2. The Perceptron — Single Neuron

The perceptron is the simplest neural network — a single artificial neuron. Given inputs x₁, x₂, …, xₙ:

A single perceptron can only learn linearly separable patterns — the same limitation as logistic regression. To model non-linear relationships, multiple layers are needed.

3. Multi-Layer Neural Network (MLP)

A Multi-Layer Perceptron (MLP) — also called a fully connected neural network or deep neural network — adds one or more hidden layers between the input and output:

• Input Layer: One neuron per input feature. No computation — just passes values forward.
• Hidden Layers: One or more layers of neurons that learn intermediate representations. Each neuron connects to every neuron in the previous layer (fully connected). The non-linear activation functions here are what give neural networks their power.
• Output Layer: Produces the final prediction. Number of neurons = number of output classes (for classification) or 1 (for regression).

The Universal Approximation Theorem states that a neural network with a single hidden layer containing a sufficient number of neurons can approximate any continuous function to arbitrary accuracy. This is why neural networks are so powerful — they are universal function approximators.

4. Activation Functions

Activation functions introduce non-linearity into the network. Without them, stacking multiple layers would be equivalent to a single linear transformation — no matter how many layers, the network could only learn linear functions.

Why ReLU is the Default Choice

ReLU is preferred because: it does not suffer from the vanishing gradient problem for positive inputs (gradient = 1 for z > 0); it is computationally efficient (simple max operation); it creates sparse activations (many neurons output 0), reducing computation; it converges faster than sigmoid/tanh in practice.

The Vanishing Gradient Problem

Sigmoid and tanh saturate at their extremes — their gradients approach zero for large or small input values. During backpropagation, these near-zero gradients get multiplied together across layers, causing gradients in early layers to become vanishingly small. This makes deep networks very slow to train with sigmoid/tanh. ReLU avoids this for positive values.

5. Forward Propagation

Forward propagation is the process of computing the network’s output from an input. For each layer l:

Where W⁽ˡ⁾ is the weight matrix for layer l, A⁽ˡ⁻¹⁾ is the output of the previous layer (A⁽⁰⁾ = X, the input), b⁽ˡ⁾ is the bias vector, and A⁽ˡ⁾ is the activated output — which becomes the input to the next layer.

The final layer’s output Aˢ is the network’s prediction. For binary classification, Aˢ = sigmoid(Zˢ); for multi-class, Aˢ = softmax(Zˢ).

6. Loss Functions

7. Training — The Four-Step Loop

Neural network training repeats this loop for each batch of data, for many epochs:

1. Forward Propagation: Pass the input batch through all layers to compute predictions.
2. Compute Loss: Calculate the loss function value — how wrong the predictions are.
3. Backpropagation: Compute the gradient of the loss with respect to every weight and bias in the network using the chain rule. This tells us which direction to adjust each parameter to reduce loss.
4. Gradient Descent Update: Adjust all weights and biases in the direction that reduces loss: W := W − α × ∂L/∂W

One epoch = one complete pass through the entire training dataset. An iteration = one gradient update using one batch. If you have 1,000 examples and batch size 100, one epoch = 10 iterations.

8. Key Hyperparameters

9. Python Code

import numpy as np
import tensorflow as tf
from tensorflow import keras
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, to_categorical

# Load and prepare data
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features (essential for neural networks)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test  = scaler.transform(X_test)

# One-hot encode labels for multi-class
y_train_cat = to_categorical(y_train, num_classes=3)
y_test_cat  = to_categorical(y_test, num_classes=3)

# Build the neural network
model = keras.Sequential([
    keras.layers.Input(shape=(4,)),               # Input layer: 4 features
    keras.layers.Dense(64, activation='relu'),     # Hidden layer 1
    keras.layers.Dropout(0.3),                     # Regularisation
    keras.layers.Dense(32, activation='relu'),     # Hidden layer 2
    keras.layers.Dense(3, activation='softmax')    # Output: 3 classes
])

# Compile
model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy']
)

model.summary()

# Train with early stopping
early_stop = keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True)
history = model.fit(
    X_train, y_train_cat,
    epochs=100,
    batch_size=16,
    validation_split=0.2,
    callbacks=[early_stop],
    verbose=0
)

# Evaluate
loss, accuracy = model.evaluate(X_test, y_test_cat, verbose=0)
print(f"Test Accuracy: {accuracy:.3f}")
print(f"Training stopped at epoch: {len(history.history['loss'])}")
    

10. Common Mistakes Students Make

• Not scaling input features: Neural networks are extremely sensitive to input scale. Unscaled features cause very slow convergence or training failure. Always use StandardScaler or normalise to [0,1] before training.
• Using sigmoid in hidden layers: Sigmoid saturates and causes vanishing gradients in deep networks. Use ReLU (or Leaky ReLU) in all hidden layers. Only use sigmoid in the output layer for binary classification.
• Training for a fixed number of epochs without early stopping: Training too long leads to overfitting. Always use EarlyStopping with a validation set — stop training when validation loss stops improving.
• Not using a validation set: Without a validation set, you cannot monitor overfitting during training. Always split data into train/validation/test, or use validation_split in Keras.
• Starting with a very complex architecture: Start simple — one or two hidden layers with a moderate number of neurons. Add complexity only if underfitting. A simple model that works is always better than a complex one that does not.

11. Frequently Asked Questions