← NotebookBias-Variance Decomposition April 21, 2026
Derivation ML theory
E [ ( y − f ^ ( x ) ) 2 ] = ( E [ f ^ ( x ) ] − f ( x ) ) 2 ⏟ bias 2 + Var ( f ^ ( x ) ) ⏟ variance + σ 2 ⏟ noise \mathbb{E}[(y - \hat{f}(x))^2] = \underbrace{(\mathbb{E}[\hat{f}(x)] - f(x))^2}_{\text{bias}^2} + \underbrace{\text{Var}(\hat{f}(x))}_{\text{variance}} + \underbrace{\sigma^2}_{\text{noise}} E [( y − f ^ ( x ) ) 2 ] = bias 2 ( E [ f ^ ( x )] − f ( x ) ) 2 + variance Var ( f ^ ( x )) + noise σ 2
Bias: model too simple to capture f
Variance: model too sensitive to training sample
Noise: irreducible — the floor
Why it matters: Every regularization choice (dropout, weight decay, early stopping) trades bias for variance. Knowing which side you’re on tells you which knob to turn.