Jensen's Inequality

For convex f: $\mathbb{E}[f(X)] \geq f(\mathbb{E}[X])$. For concave f (e.g., log): $\mathbb{E}[f(X)] \leq f(\mathbb{E}[X])$.

Why it matters: Whole derivation of ELBO / variational inference starts here. Also why $\log \mathbb{E}[\cdot]$ and $\mathbb{E}[\log \cdot]$ aren’t interchangeable — they differ by the KL to the variational posterior.