Weight Init Lab

An interactive guide to neural network weight initialization. Before a neural network learns anything, every weight gets a random number. Pick those numbers badly and the network can't learn at all. This page lets you see why, and how Xavier, He and related schemes fix it.

Why the first numbers matter

A deep network is a long chain of matrix multiplications with a squashing function between each one. Each layer multiplies its input by a weight matrix, so whatever scale the weights have gets multiplied in again and again. A scale slightly too small and the signal shrinks toward zero by the tenth layer. Slightly too large and it explodes, or slams every tanh unit into its flat ±1 region where nothing can change.

The same thing happens in reverse during backpropagation. Gradients travel back through the same weights, so they vanish or explode the same way. When gradients vanish, the early layers stop learning. The goal of a good initialization is simple to state: keep the size of the signal roughly constant from layer to layer, in both directions.

Watch a signal travel through the network

Below, 128 random inputs pass through a deep, fully connected network that has only been initialized, never trained. Each panel shows the distribution of one layer's outputs. The chart underneath tracks the spread of the forward signal and of the backward gradient at every layer.

Initialization Activation Depth 10 Width

Try:

Forward signal spread (std of activations) Backward gradient spread (std of ∂L/∂activation)

Where the formulas come from

The derivation behind Xavier initialization is short enough to follow in four steps. Take one layer that computes z = Wx, where each output sums over n_in inputs.

Assume the weights and inputs are independent, with mean zero. The variance of a product of independent zero-mean variables is the product of their variances, and variances add over a sum. So each output's variance is:

Var(z) = n_in · Var(w) · Var(x)
For the signal to keep its size going forward, we need Var(z) = Var(x), which means:

Var(w) = 1 / n_in LeCun initialization, from 1998
Run the same argument backward. The gradient reaching each input is a sum over n_out outputs, so keeping gradients steady requires Var(w) = 1 / n_out. Unless the layer is square you can't satisfy both, so Glorot and Bengio (2010) took the average:

Var(w) = 2 / (n_in + n_out) Xavier / Glorot. For a uniform U(−a, a), a = √(6 / (n_in + n_out))
This analysis treats the activation as roughly linear near zero, which holds for tanh. ReLU breaks it: it zeroes out half of its inputs, so it throws away half the signal's energy at every layer. He et al. (2015) doubled the variance to compensate:

Var(w) = 2 / n_in He / Kaiming initialization, for ReLU

Sigmoid is the awkward one. Its outputs are centered on 0.5, not zero, and its slope is at most 0.25, so every layer shrinks the gradient by at least a factor of four no matter how the weights are scaled. That is a large part of why deep networks moved away from it.

A field guide to the schemes

The last column shows the actual weight standard deviation for a square layer at the width selected in the lab above, so you can compare the scales directly.

Scheme	Weight variance	Built for	What goes wrong without it	Std at width 128

The training race

Showing the signal at initialization is one thing. This is the payoff: four identical networks, differing only in how their weights start, trained side by side on the same data with the same mini-batches. Each lane shows the decision boundary as it forms, and a strip of bars for the gradient size at each layer, from the input layer on the left to the output on the right.

Dataset Activation Hidden layers 6 Learning rate 0.1

Step 0 of 4000

Press start. Every network gets the same data, the same mini-batches and the same underlying random draws; only the scale of the starting weights differs. The dashed line marks a loss of ln 2 ≈ 0.693, which is what a network scores when it just guesses 50/50.

What to take away

Initialization is about variance, not about the particular random values. Keep each layer's output variance close to its input variance, forward and backward, and a deep network starts in a place where gradients can reach every layer.

Match the scheme to the activation: Xavier for tanh and other functions that look linear near zero, He for ReLU and its relatives. Modern tools like batch normalization, layer normalization and residual connections make networks far more forgiving, but they all lean on the same idea, and a bad starting scale can still waste hours of training.