A network computes a = w1*x, h = relu(a), o = w2*h, with squared loss L = (o - t)^2/2. Given x = 2, w1 = 0.5, w2 = -3, t = 1, compute dL/dw1 and dL/dw2 by backpropagation.

A network computes a = w1*x, h = relu(a), o = w2*h, with squared loss L = (o - t)^2/2. Given x = 2, w1 = 0.5, w2 = -3, t = 1, compute dL/dw1 and dL/dw2 by backpropagation.

Approach: Run the forward pass to get a, h and o, then propagate the output error backwards, remembering the ReLU derivative is an indicator on the sign of the pre-activation.

dL/dw2 = -4 and dL/dw1 = 24. The forward pass gives a = 0.5*2 = 1, h = relu(1) = 1, o = -3*1 = -3, so the error is o - t = -4 and L = 8. The chain rule is applied backwards from the output, so backpropagation starts with dL/do = o - t = -4. For the second weight, do/dw2 = h = 1, so dL/dw2 = -4*1 = -4. Propagating further back, dL/dh = w2 * dL/do = -3 * -4 = 12, and the ReLU derivative at a = 1 is 1 because the pre-activation is positive, so dL/da = 12. Finally da/dw1 = x = 2, giving dL/dw1 = 12*2 = 24. Had a been negative the ReLU derivative would be 0 and dL/dw1 would be exactly 0, the dead unit case, where no gradient reaches w1 and the unit cannot recover from that input alone.

Follow-up: With learning rate 0.01 and one gradient step on both weights, does the loss decrease, and at what learning rate does the ReLU unit die on this example?

Key concepts: forward pass, backpropagation, ReLU derivative, chain rule.