Deep Neural Networks: How They Work, Types, Training

A deep neural network (DNN) is an artificial neural network with more than one hidden layer between the input and the output. Each layer of neurons computes a weighted sum of its inputs, adds a bias and passes the result through a non-linear activation function. Stacking those layers lets the network build features on top of features: early layers pick up edges or word fragments, later layers pick up shapes, objects or meaning. The network learns by comparing its prediction with the correct answer and pushing the error backwards through the layers to correct every weight.

What makes a neural network “deep”?

Depth is counted in hidden layers, not in neurons. A network with an input layer, one hidden layer and an output layer is a shallow network. Add a second hidden layer and it qualifies as deep. In practice, working models are far deeper than two: an image classifier like ResNet-50 has 50 weight layers, and the transformer blocks inside a modern language model are stacked dozens of times.

Why not just widen one layer instead? The universal approximation theorem says a single hidden layer with enough neurons can approximate any continuous function. The catch is “enough” can mean an impractical number. Depth is cheaper. Each extra layer reuses the features the layer below already built, so a deep network reaches the same accuracy with far fewer parameters than a very wide shallow one.

Depth has a cost. Error signals shrink as they travel back through many layers, which is the vanishing gradient problem. Three fixes made very deep networks trainable: ReLU-type activations that do not saturate, normalisation layers, and skip (residual) connections that let the signal jump over a block of layers.

The maths inside one neuron

A neuron is a small formula, nothing more. For inputs x₁…x_n:

z = w₁x₁ + w₂x₂ + … + w_nx_n + b, then a = f(z)

Weights (w) say how much each input matters. They are learned.
Bias (b) shifts the threshold so the neuron can fire even when all inputs are zero. It is learned too.
z is the pre-activation, often written as the dot product z = w·x + b.
f is the activation function, and it is the only non-linear step.

That non-linearity is not decoration. Without it, a stack of layers collapses: a linear function of a linear function is still linear, so a 50-layer network without activations would have exactly the power of a single layer.

For a whole layer, the same thing is written once in matrix form: a = f(Wx + b), where W holds one row of weights per neuron. That is why training runs on a GPU, which multiplies large matrices in parallel.

How many parameters does a layer have?

A dense layer with n inputs and m neurons has n × m weights plus m biases. A layer taking 784 pixel values into 128 neurons holds 784 × 128 + 128 = 100,480 parameters. Counting parameters is the quickest way to judge whether a model will fit on your hardware.

Activation functions and when to use each

Activation	Formula	Output range	Typical use
ReLU	max(0, z)	0 to ∞	Default for hidden layers in dense networks and CNNs. Cheap and does not saturate for positive z.
Leaky ReLU	max(0.01z, z)	−∞ to ∞	When neurons “die” and stop updating because their output is stuck at 0.
GELU / SiLU (Swish)	z·Φ(z) / z·σ(z)	slightly below 0 to ∞	Hidden layers in transformer blocks. Smooth, so gradients behave better than plain ReLU.
Sigmoid	1 / (1 + e^−z)	0 to 1	Output neuron for binary classification, or multi-label outputs. Avoid in hidden layers.
Tanh	(e^z − e^−z) / (e^z + e^−z)	−1 to 1	Hidden state in classic RNNs and LSTMs. Zero-centred, but saturates at both ends.
Softmax	e^z_i / Σ e^z_j	0 to 1, sums to 1	Output layer for multi-class classification, since it turns scores into probabilities.

Student rule of thumb: ReLU everywhere in the hidden layers unless you are copying a transformer design, sigmoid for one yes/no output, softmax for one-of-many output, and nothing at all on the output of a regression model.

Forward pass: a worked example with real numbers

Take a tiny network with 2 inputs, one hidden layer of 2 ReLU neurons and 1 sigmoid output. That is 2×2 + 2 = 6 parameters in the hidden layer and 2 + 1 = 3 in the output layer, so 9 in total.

Inputs: x₁ = 2, x₂ = 3.

Hidden neuron h₁ with w = [0.5, −0.2], b = 0.1:
z = (0.5 × 2) + (−0.2 × 3) + 0.1 = 1.0 − 0.6 + 0.1 = 0.5
ReLU(0.5) = 0.5

Hidden neuron h₂ with w = [−0.4, 0.3], b = 0.2:
z = (−0.4 × 2) + (0.3 × 3) + 0.2 = −0.8 + 0.9 + 0.2 = 0.3
ReLU(0.3) = 0.3

Output neuron with w = [1.0, −2.0], b = 0.05:
z = (1.0 × 0.5) + (−2.0 × 0.3) + 0.05 = 0.5 − 0.6 + 0.05 = −0.05
sigmoid(−0.05) = 1 / (1 + e^0.05) = 1 / 2.0513 = 0.488

The network predicts 0.488, so roughly a coin flip. If the true label is 1, the binary cross-entropy loss is L = −ln(0.488) = 0.718. That single number is what training tries to reduce.

Backpropagation: how the network learns

Backpropagation is the chain rule of calculus applied layer by layer. It answers one question for every parameter: if I nudge this weight up slightly, does the loss go up or down, and by how much? The answer is the partial derivative ∂L/∂w, and gradient descent then moves the weight the other way.

One training step has four parts:

Forward pass. Feed a batch of examples through the layers and get predictions.
Loss. Compare predictions with the true labels to get a single number.
Backward pass. Starting at the output, compute the gradient of the loss for each layer’s weights, reusing the gradient from the layer above. This reuse is why training a 100-layer network costs only about twice a forward pass, not 100 times.
Update. w_new = w_old − η × ∂L/∂w, where η is the learning rate.

Continue the worked example. For a sigmoid output with cross-entropy loss, the gradient at the output collapses to a clean expression: ∂L/∂z = ŷ − y = 0.488 − 1 = −0.512. The gradient for the first output weight is that value times the activation feeding it: ∂L/∂w₁ = −0.512 × 0.5 = −0.256. With a learning rate of 0.1:

w₁ = 1.0 − 0.1 × (−0.256) = 1.0256

The weight went up, which pushes the next prediction closer to 1. The same arithmetic runs for all nine parameters, then for the next batch, and it repeats for thousands of batches. A pass over the whole training set is one epoch.

Types of deep neural networks compared

Architecture	Core idea	Data it suits	Strength	Main limitation
Feedforward / MLP	Fully connected layers, information flows one way	Fixed-length tabular data	Simple baseline, easy to train	Ignores spatial or time order; parameter count explodes on images
CNN (convolutional)	Small filters slide over the input and share weights	Images, video frames, spectrograms, some 1D signals	Few parameters, detects a pattern anywhere in the frame	Limited view of long-range context
RNN / LSTM	A hidden state carries information from one time step to the next; the LSTM adds a memory cell with learned controls that decide what to keep, drop and output	Time series, sensor streams, speech	Handles variable-length sequences with a small memory footprint	Must process steps in order, so training is slow and long dependencies still fade
Transformer	Self-attention lets every token look at every other token in one step	Text, code, and increasingly images and audio	Trains in parallel, captures long-range context, scales well	Attention cost grows with the square of sequence length; needs large data

As of 2026 the transformer is the default for language and is widely used for vision as well, while CNNs stay popular where compute is tight, such as on-device cameras and embedded boards. Recurrent models still earn their place on low-power streaming data.

Training essentials: loss, optimiser, learning rate

Loss function. Pick it from the task, not from taste. Mean squared error for regression, binary cross-entropy for one yes/no output, categorical cross-entropy with softmax for multi-class.

Optimiser. Plain SGD updates every weight by the same learning rate. SGD with momentum (typically 0.9) smooths the path. Adam and AdamW keep a per-parameter estimate of the gradient scale and are the usual default; a starting learning rate of 0.001 works for most Adam runs, against 0.01 to 0.1 for SGD.

Learning rate. This is the setting most worth tuning. Too high and the loss jumps around or turns into NaN; too low and training crawls. A common recipe is a short warm-up followed by cosine decay to near zero.

Batch size and epochs. A batch is how many examples are averaged before one update. Larger batches give smoother gradients and better GPU use; smaller batches add useful noise. Track validation loss each epoch and stop when it stops improving.

Overfitting and how to fix it

Overfitting is when training loss keeps falling while validation loss rises. The model is memorising the training set instead of learning the pattern. Remedies, roughly in order of effect:

More data, or data augmentation such as random crops, flips and colour shifts for images.
Dropout: randomly switch off 20% to 50% of neurons during training so the network cannot lean on any one path.
Weight decay (L2): add a penalty on large weights, typically 1e-4 to 1e-2.
Early stopping: keep the checkpoint with the best validation score.
A smaller model, or transfer learning from a pre-trained model when your dataset is small.

The opposite failure, underfitting, shows as high loss on both training and validation data. Then you need a bigger model, better features or longer training, not more regularisation.

Where deep neural networks are used

Vision: defect detection on production lines, medical image screening, number-plate reading, crop disease detection from phone photos.
Language: translation, search ranking, summarisation, chat assistants, and clinical text mining in health records.
Speech and audio: dictation, voice assistants in Indian languages, industrial fault detection from machine sound.
Forecasting: electricity load, equipment failure from vibration sensors, demand planning.
Robotics and vehicles: perception and control stacks, including humanoid robot foundation models.
Edge devices: compressed networks running inside cameras, drones and IoT sensor nodes, usually quantised to 8-bit integers to fit the memory budget.

Student tip: build the tiny network above in NumPy by hand before touching PyTorch or TensorFlow. Once the nine-parameter version trains correctly, the framework version is only a change of notation.

References

Artificial neural network fundamentals, Springer.
Natural language processing in clinical text, JAMIA.
NVIDIA Project GR00T, robot foundation models.

FAQs

What is a deep neural network in simple words?

It is a neural network with more than one hidden layer between input and output. Each layer transforms the data a little, so the stack can learn complicated patterns such as faces in photos or meaning in sentences directly from examples.

How is a deep neural network different from a normal neural network?

A shallow network has a single hidden layer. A deep network has two or more, often dozens. The extra layers build features on top of features, which reaches the same accuracy with far fewer parameters than one very wide layer would need.

What is the difference between forward propagation and backpropagation?

Forward propagation runs data through the layers to produce a prediction. Backpropagation runs the error backwards using the chain rule to find how much each weight contributed, so gradient descent can correct it.

How many layers does a network need to be called deep?

Two hidden layers is the textbook threshold. In practice, useful models range from about 5 layers for small tabular tasks to 50 or more for image models and stacked transformer blocks for language models.

Which activation function should I use in hidden layers?

Start with ReLU. Switch to Leaky ReLU if neurons stop updating, and to GELU or SiLU if you are building transformer blocks. Keep sigmoid and softmax for the output layer only.

Deep Neural Networks: How They Work, Architectures and Training