Short technical notes: derivations, code observations, and debugging lessons from working through machine-learning systems.https://www.vitorsousa.com/https://www.vitorsousa.com/notebook/benchmark-your-benchmark/https://www.vitorsousa.com/notebook/benchmark-your-benchmark/A number without variance isn't a result — and a benchmark without a correctness gate isn't a benchmark.Fri, 24 Jul 2026 00:00:00 GMTDebuggingbenchmarkingmeasurementtestinghttps://www.vitorsousa.com/notebook/gelu-swiglu/https://www.vitorsousa.com/notebook/gelu-swiglu/Why it matters: The gate is a content-dependent multiplier on each hidden unit — the FFN learns which features to let through, not just how to mix them.Sat, 02 May 2026 00:00:00 GMTCode notetransformersffnswiglugelufrom-scratchhttps://www.vitorsousa.com/notebook/pre-norm-vs-post-norm/https://www.vitorsousa.com/notebook/pre-norm-vs-post-norm/Why it matters: In pre-norm the residual stream is never normalized, so every block has a clean identity gradient path back to the input — deep stacks (≥12 layers) train without…Sat, 02 May 2026 00:00:00 GMTCode notetransformersnormalizationarchitecturefrom-scratchhttps://www.vitorsousa.com/notebook/rmsnorm-vs-layernorm/https://www.vitorsousa.com/notebook/rmsnorm-vs-layernorm/Why it matters: RMSNorm drops the mean-centering step and the bias, normalizing by the root-mean-square instead of the standard deviation — same gain term, fewer ops, ~7–15% low…Sat, 02 May 2026 00:00:00 GMTCode notetransformersnormalizationrmsnormlayernormfrom-scratchhttps://www.vitorsousa.com/notebook/scaled-dot-product-attention/https://www.vitorsousa.com/notebook/scaled-dot-product-attention/Why it matters: Without the √dk divisor, dot products grow with dimension and push softmax into its saturated tail — gradients vanish and training stalls past dk ≥ 64.Sat, 02 May 2026 00:00:00 GMTCode notetransformersattentionfrom-scratchhttps://www.vitorsousa.com/notebook/rope-15-lines/https://www.vitorsousa.com/notebook/rope-15-lines/Rotate Q and K by a position-dependent angle so attention scores depend only on relative distance — the real-valued cos/sin + rotate_half form used in Llama and in rlvr-from-scratch.Thu, 23 Apr 2026 00:00:00 GMTCode noteropepositional-encodingpytorchtransformerhttps://www.vitorsousa.com/notebook/bias-variance-decomposition/https://www.vitorsousa.com/notebook/bias-variance-decomposition/Expected squared error splits into bias², variance, and irreducible noise.Tue, 21 Apr 2026 00:00:00 GMTDerivationtheoryregularizationgeneralizationhttps://www.vitorsousa.com/notebook/causal-mask-additive/https://www.vitorsousa.com/notebook/causal-mask-additive/Additive 0.0/-inf masks compose; multiplicative masks don't.Tue, 21 Apr 2026 00:00:00 GMTCode noteattentionmaskingpytorchhttps://www.vitorsousa.com/notebook/chain-rule-vectors/https://www.vitorsousa.com/notebook/chain-rule-vectors/Backprop is the vector chain rule applied right-to-left via VJPs.Tue, 21 Apr 2026 00:00:00 GMTDerivationcalculusbackpropjacobianhttps://www.vitorsousa.com/notebook/eigendecomposition/https://www.vitorsousa.com/notebook/eigendecomposition/A = QΛQ⁻¹ diagonalizes repeated matrix action; spectral radius governs stability.Tue, 21 Apr 2026 00:00:00 GMTDerivationlinear-algebraeigenvaluesdynamicshttps://www.vitorsousa.com/notebook/einops-rearrange/https://www.vitorsousa.com/notebook/einops-rearrange/Named dimensions at the call site — shape bugs drop to near zero.Tue, 21 Apr 2026 00:00:00 GMTCode noteeinopspytorchtensorshttps://www.vitorsousa.com/notebook/he-initialization/https://www.vitorsousa.com/notebook/he-initialization/Variance-preserving init for ReLU-family activations; Xavier is for tanh/sigmoid.Tue, 21 Apr 2026 00:00:00 GMTCode noteinitializationrelupytorchhttps://www.vitorsousa.com/notebook/jensens-inequality/https://www.vitorsousa.com/notebook/jensens-inequality/For concave f, E[f(X)] ≤ f(E[X]); the gap is exploited by ELBO.Tue, 21 Apr 2026 00:00:00 GMTDerivationtheoryelbovariational-inferencehttps://www.vitorsousa.com/notebook/kv-cache/https://www.vitorsousa.com/notebook/kv-cache/Concatenate previous K/V to new K/V; per-step cost drops from O(T²) to O(T).Tue, 21 Apr 2026 00:00:00 GMTCode noteinferenceattentionoptimizationhttps://www.vitorsousa.com/notebook/mle-equals-cross-entropy/https://www.vitorsousa.com/notebook/mle-equals-cross-entropy/Minimizing cross-entropy is MLE for a categorical distribution.Tue, 21 Apr 2026 00:00:00 GMTDerivationtheorycross-entropymlehttps://www.vitorsousa.com/notebook/rope-at-dk-not-dmodel/https://www.vitorsousa.com/notebook/rope-at-dk-not-dmodel/Rotate Q and K per-head, after the head split, to keep the relative-position property.Tue, 21 Apr 2026 00:00:00 GMTCode noteropepositional-encodingattentionhttps://www.vitorsousa.com/notebook/seed-everything/https://www.vitorsousa.com/notebook/seed-everything/Same seed + same code → same loss curves; non-negotiable for debugging.Tue, 21 Apr 2026 00:00:00 GMTCode notereproducibilitypytorchdebugginghttps://www.vitorsousa.com/notebook/sinusoidal-vs-learned-vs-rotary/https://www.vitorsousa.com/notebook/sinusoidal-vs-learned-vs-rotary/RoPE wins because it encodes relative position as a first-class operation.Tue, 21 Apr 2026 00:00:00 GMTCode notepositional-encodingropetransformerhttps://www.vitorsousa.com/notebook/softmax-temperature/https://www.vitorsousa.com/notebook/softmax-temperature/τ → 0 is argmax, τ = 1 is softmax, τ → ∞ is uniform.Tue, 21 Apr 2026 00:00:00 GMTDerivationsoftmaxsamplingdistillationhttps://www.vitorsousa.com/notebook/standardize-vs-normalize/https://www.vitorsousa.com/notebook/standardize-vs-normalize/Standardize for Gaussian-ish; normalize for bounded ranges; robust for outliers.Tue, 21 Apr 2026 00:00:00 GMTCode notepreprocessingscalingstatisticshttps://www.vitorsousa.com/notebook/stratified-split/https://www.vitorsousa.com/notebook/stratified-split/Preserve class ratios across train/test under imbalance with stratify=y.Tue, 21 Apr 2026 00:00:00 GMTCode notesklearnevaluationclass-imbalancehttps://www.vitorsousa.com/notebook/svd/https://www.vitorsousa.com/notebook/svd/Every matrix is rotate-scale-rotate; truncation gives best low-rank approximation.Tue, 21 Apr 2026 00:00:00 GMTDerivationlinear-algebrasvdpcahttps://www.vitorsousa.com/notebook/uv-package-manager/https://www.vitorsousa.com/notebook/uv-package-manager/10–100× faster than pip, lockfile built-in, replaces pip + venv + pip-tools.Tue, 21 Apr 2026 00:00:00 GMTCode notepythonuvpackaginghttps://www.vitorsousa.com/notebook/contiguous-after-transpose/https://www.vitorsousa.com/notebook/contiguous-after-transpose/view() needs contiguous memory; transpose() returns a non-contiguous view.Tue, 21 Apr 2026 00:00:00 GMTDebuggingpytorchtensorsdebugginghttps://www.vitorsousa.com/notebook/bessels-correction-variance/https://www.vitorsousa.com/notebook/bessels-correction-variance/Use N-1 degrees of freedom for unbiased variance on samples instead of the population formula.Sun, 15 Dec 2024 00:00:00 GMTCode notestatisticsvariancepython-trickshttps://www.vitorsousa.com/notebook/average-precision-manual/https://www.vitorsousa.com/notebook/average-precision-manual/Interpolate precision and integrate the PR curve yourself when library helpers are unavailable.Sun, 15 Dec 2024 00:00:00 GMTCode noteevaluationprecision-recallpython-trickshttps://www.vitorsousa.com/notebook/centroid-kmeans-rag/https://www.vitorsousa.com/notebook/centroid-kmeans-rag/Cluster embeddings and surface diverse centroids instead of near-duplicates for long-context prompts.Sun, 15 Dec 2024 00:00:00 GMTCode noteragretrievalclusteringhttps://www.vitorsousa.com/notebook/log-sum-exp-softmax/https://www.vitorsousa.com/notebook/log-sum-exp-softmax/Compute softmax in log-space to avoid overflow while keeping gradients stable.Sun, 15 Dec 2024 00:00:00 GMTCode notenumerical-stabilitylog-spacepython-trickshttps://www.vitorsousa.com/notebook/vectorized-covariance/https://www.vitorsousa.com/notebook/vectorized-covariance/Leverage BLAS-backed matrix multiplication for fast covariance estimates on wide datasets.Sun, 15 Dec 2024 00:00:00 GMTCode notenumpyperformancefeature-engineering