Vitor Sousa

Search

Try a topic

↑ ↓ Navigate Enter OpenEsc Close

Notebook

Maximum Likelihood = Minimum Cross-Entropy

Derivation ML theory

arg⁡max⁡θ∑ilog⁡pθ(yi∣xi)  =  arg⁡min⁡θ  −1N∑ilog⁡pθ(yi∣xi)\arg\max_\theta \sum_i \log p_\theta(y_i \mid x_i) \;=\; \arg\min_\theta \;-\frac{1}{N}\sum_i \log p_\theta(y_i \mid x_i)

The RHS is cross-entropy between the empirical distribution and the model.

Why it matters: “Why cross-entropy loss?” has one answer: it’s MLE for a categorical distribution. Same identity gives you MSE for Gaussians and binary cross-entropy for Bernoullis.