Show that the negative log-likelihood of logistic regression is convex in the weights. State the gradient and the Hessian, and say what convexity does and does not guarantee about the fitted solution.

Show that the negative log-likelihood of logistic regression is convex in the weights. State the gradient and the Hessian, and say what convexity does and does not guarantee about the fitted solution.

Approach: Write the log-likelihood with p = sigmoid(x^T b), differentiate twice, and check the sign of the quadratic form of the Hessian.

The gradient is sum_i (p_i - y_i) x_i and the Hessian is X^T W X with W = diag(p_i (1 - p_i)), which is positive semidefinite, so the loss is convex and every local minimum is global. Writing the loss as sum_i [-y_i log p_i - (1 - y_i) log(1 - p_i)] with p_i = 1/(1 + e^{-x_i^T b}), the derivative of p with respect to the linear score is p(1 - p), and the two log terms collapse to the residual form (p_i - y_i) x_i. For the Hessian, v^T X^T W X v = sum_i w_i (x_i^T v)^2 >= 0 because every w_i = p_i(1 - p_i) lies in (0, 1/4]. Convexity guarantees no spurious local minima and that any stationary point found by a descent method is the optimum. It does not guarantee the optimum is finite: on separable data the likelihood is maximised by pushing ||b|| to infinity, the weights w_i degenerate to zero, and the fit diverges. A ridge penalty makes the objective strictly convex and restores a finite unique solution.

Follow-up: With an L2 penalty added, what is the condition number of the Hessian near the optimum and how does it bound the convergence rate of gradient descent?

Key concepts: log-likelihood, convexity, Hessian, separable data.