Softmax + Temperature
- τ → 0: argmax (sharp)
- τ = 1: standard softmax
- τ → ∞: uniform
Why it matters: Sampling temperature in LLMs is this τ. Also appears in knowledge distillation (soft targets) and contrastive learning (InfoNCE).
Why it matters: Sampling temperature in LLMs is this τ. Also appears in knowledge distillation (soft targets) and contrastive learning (InfoNCE).