Calculus, AI, and linear algebra: a compact field guide

Most ML code is just calculus and linear algebra in disguise. Here is a concise refresher with runnable snippets.

Gradients in plain sight

A gradient is the vector of partial derivatives. For a scalar function $f(x, y)$:

$$\nabla f = \left[\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right]$$

Example: $f(x, y) = x^2 + xy + 3y^2$ yields $\nabla f = [2x + y,\ x + 6y]$.

import numpy as np

def f(xy):
    x, y = xy
    return x**2 + x*y + 3*y**2

# analytic gradient
def grad(xy):
    x, y = xy
    return np.array([2*x + y, x + 6*y])

pt = np.array([2.0, -1.0])
print("f:", f(pt))
print("grad:", grad(pt))

Finite differences are a quick sanity check:

def finite_diff(fn, pt, eps=1e-5):
    g = np.zeros_like(pt)
    for i in range(len(pt)):
        step = np.zeros_like(pt)
        step[i] = eps
        g[i] = (fn(pt + step) - fn(pt - step)) / (2 * eps)
    return g

print("finite diff:", finite_diff(f, pt))

Jacobians: vector outputs

For $g: \mathbb{R}^n \to \mathbb{R}^m$, the Jacobian stacks the gradients of each output component. A simple two-output function:

$$g(x, y) = \begin{bmatrix} x^2 + y \\ xy \end{bmatrix}$$

Its Jacobian is:

$$J = \begin{bmatrix} 2x & 1 \\ y & x \end{bmatrix}$$

def g(xy):
    x, y = xy
    return np.array([x**2 + y, x*y])

def jacobian(xy):
    x, y = xy
    return np.array([[2*x, 1], [y, x]])

pt = np.array([1.5, 0.5])
print("g(pt):", g(pt))
print("J(pt):\n", jacobian(pt))

Linear algebra fuel: projections and SVD

Principal component analysis (PCA) is just the singular value decomposition (SVD): $X = U\Sigma V^T$. The top right singular vectors in $V$ are the principal directions.

rng = np.random.default_rng(7)
X = rng.normal(size=(6, 3))  # 6 samples, 3 features

# center
Xc = X - X.mean(axis=0, keepdims=True)

# SVD
U, S, Vt = np.linalg.svd(Xc, full_matrices=False)

print("singular values:", S)
print("first principal direction:", Vt[0])

# project to 2D
X2 = Xc @ Vt[:2].T
print("projected shape:", X2.shape)

Projection of a vector $v$ onto a direction $u$ is:

$$\text{proj}_u(v) = \frac{v \cdot u}{\lVert u \rVert^2} u$$

v = np.array([2.0, 1.0, -1.0])
u = Vt[0]  # principal direction
proj = (v @ u) / (u @ u) * u
print("projection:", proj)

graph LR;
    Data["High-dimensional data X"] --> Center["Center columns"];
    Center --> SVD["SVD: X = U Σ Vᵀ"];
    SVD --> PCs["Take top k rows of Vᵀ (principal directions)"];
    PCs --> Project["Project: X · V_kᵀ"];
    Project --> Embeddings["Lower-dimensional embeddings"];

Why this matters for AI

• Gradients drive optimizers (SGD, Adam); Jacobians underpin backprop.
• SVD/PCA reduces dimensionality and denoises embeddings.
• Projections help in retrieval and similarity search by isolating informative axes.

If you keep these primitives sharp, most model code becomes easier to reason about and debug.