Skip to main content

Chapter 38
Regression as Boundary Alignment

Probability and Statistics
Author: Mark Anthony Higgins
© Mark Anthony Higgins 11 October 2025
Revised Date: 15 June 2026
Ai Search
Audio Overview
Previous Chapter 37
Next Chapter 39
Flow Algebra 
flow "Regression as Boundary Alignment"

# ---------- CONCEPT ----------
# Regression aligns probabilistic boundaries — finding the line, plane, or manifold that minimises discrepancy
# between predicted and observed values. It is the geometry of inference: the act of aligning one coherence field (predictions)
# to another (data) through least-squares or likelihood optimisation.

# ---------- STATES ----------
state RegressionSystem(R):
  DataPoints:
    description: "Pairs (x_i, y_i) — observations of predictor(s) x and response y."
  ModelFunction:
    description: "ŷ = f(x; θ) — mapping from inputs to predicted outputs, parameterised by θ."
  ErrorField:
    description: "ε_i = y_i − ŷ_i — residuals; local deviations between observation and model boundary."
  Objective:
    description: "L(θ) = E[ε²] or Σ ε_i² — expected or empirical squared error to be minimised."
  LinearRegression:
    description: "ŷ = β₀ + βᵀx — linear boundary alignment; β defines slope (orientation) and intercept (position)."
  MaximumLikelihood:
    description: "Under Gaussian noise, minimising Σ ε² ⇔ maximising likelihood; regression as statistical projection."
  MultivariateForm:
    description: "Y = Xβ + ε, β̂ = (XᵀX)⁻¹XᵀY."

# ---------- PARAMETERS ----------
params:
  predictors: "x ∈ ℝᵈ"
  response: "y ∈ ℝ"
  noise: "ε ∼ 𝓝(0, σ²)"
  model: "linear or nonlinear f(x; θ)"
  loss_function: "MSE, cross-entropy, or general convex L(θ)"
  closure: true

# ---------- INVARIANTS ----------
invariants:
  - "Least-squares optimality: β̂ = argmin_β Σ (y_i − x_iᵀβ)²."
  - "Normal equations: XᵀX β̂ = XᵀY."
  - "Projection form: β̂ = (XᵀX)⁻¹XᵀY ⇒ ŷ = P Y, P = X(XᵀX)⁻¹Xᵀ."
  - "Orthogonality: residuals ⟂ fitted subspace ⇒ Xᵀε̂ = 0."
  - "E[ε] = 0, Cov(ε) = σ²I under classical assumptions."
  - "If ε ~ 𝓝(0, σ²I) ⇒ β̂ ~ 𝓝(β, σ²(XᵀX)⁻¹)."
  - "Var(ŷ) = σ² X(XᵀX)⁻¹Xᵀ ⇒ prediction uncertainty field."
  - "R² = 1 − Σ(y_i − ŷ_i)² / Σ(y_i − ȳ)² ⇒ coherence fraction explained by model."

# ---------- TRANSFORMS ----------
transforms:
  FitOLS(X,Y):
    rule: "β̂ = (XᵀX)⁻¹XᵀY; computes least-squares alignment of model to data."
  ComputeResiduals(X,Y,β̂):
    rule: "ε̂ = Y − Xβ̂."
  Predict(X*,β̂):
    rule: "ŷ* = X*β̂."
  Regularise(L2):
    rule: "β̂ = (XᵀX + λI)⁻¹XᵀY (ridge regression)."
  Regularise(L1):
    rule: "β̂ = argmin Σ (y_i − x_iᵀβ)² + λ‖β‖₁ (lasso)."
  WeightedRegression(W):
    rule: "β̂ = (XᵀWX)⁻¹XᵀWY; weights adjust coherence importance per sample."
  BayesianUpdate(β₀,Σ₀,X,Y,σ²):
    rule: "Posterior β|Y ~ 𝓝(μ*,Σ*), with Σ*=(Σ₀⁻¹+XᵀX/σ²)⁻¹, μ*=Σ*(Σ₀⁻¹β₀+XᵀY/σ²)."
  NonlinearApprox(f,θ):
    rule: "Optimise θ by minimising L(θ)=Σ(y_i − f(x_i;θ))² or −log P(Y|X,θ)."

# ---------- MONITORS ----------
monitors:
  ResidualMean:
    measure: "E[ε̂]"
    target: 0
  OrthogonalityCheck:
    measure: "Xᵀε̂"
    target: 0
  VarianceInflation:
    measure: "VIF_j = 1/(1−R_j²)"
    meaning: "High ⇒ predictor collinearity."
  FitQuality:
    measure: "R² = 1 − RSS/TSS"
    meaning: "Fraction of variance explained by model."
  NormalityTest:
    measure: "ε̂ ∼ 𝓝(0, σ²)?"
    meaning: "Check Gaussian assumption for classical inference."
  LeverageInfluence:
    measure: "h_ii = X_iᵀ(XᵀX)⁻¹X_i"
    meaning: "High values ⇒ influential observations."
  CrossValidation:
    rule: "Estimate predictive generalisation by out-of-sample testing."

# ---------- EQUATIONS ----------
equations:
  - "β̂ = (XᵀX)⁻¹XᵀY"
  - "ŷ = Xβ̂"
  - "ε̂ = Y − ŷ"
  - "L(β) = Σ_i (y_i − x_iᵀβ)²"
  - "R² = 1 − \\frac{Σ(y_i − ŷ_i)²}{Σ(y_i − ȳ)²}"
  - "Cov(β̂) = σ² (XᵀX)⁻¹"
  - "β̂ ~ 𝓝(β, σ²(XᵀX)⁻¹)"
  - "Var(ŷ) = σ² X(XᵀX)⁻¹Xᵀ"
  - "Projection matrix: P = X(XᵀX)⁻¹Xᵀ ⇒ P² = P, Pᵀ = P"
  - "Orthogonality: Xᵀε̂ = 0"

# ---------- NOTES ----------
notes:
  - "Regression aligns predicted boundaries with data boundaries — it is a geometric projection of observation onto model space."
  - "The fitted model is the point on the model manifold closest to the observed data in squared-error distance."
  - "Residuals measure orthogonal deviation — the unmodelled component of the data’s coherence."
  - "Least squares corresponds to maximum likelihood under Gaussian noise — the most probable alignment given the uncertainty field."
  - "Regularisation introduces controlled tension — preventing overfitting by adding curvature to the cost surface."
  - "In Bayesian form, regression becomes a dynamic balance between prior and data coherence — posterior mean as weighted alignment."
  - "In Flow Algebra, regression represents *boundary alignment*: mapping observed data flow into the nearest coherent subspace of the model, minimising tension and restoring structural equilibrium."
Explanatory Notes

Regression is the mathematics of alignment — the process of fitting one field of coherence (a model) to another (data).
It describes how structure, when confronted with evidence, shifts to its most balanced position — the configuration that minimises tension between expectation and observation.

In this way, regression is not simply a computational tool but a principle of equilibrium: the model moves until its boundary aligns most closely with reality.

In its simplest form, linear regression seeks parameters β\beta such that predicted values

\[ \hat{y} = X\beta \]

lie as close as possible to the observed outcomes yy.
The residuals

\[ \varepsilon = y - \hat{y} \]

represent the remaining discrepancy — the energy of misalignment between model and data.
The goal of least-squares regression is to minimise the total residual energy:

\[ L(\beta) = \sum_i (y_i - x_i^{T}\beta)^2 \]

Geometrically, this corresponds to projecting the observed data yy onto the subspace spanned by the columns of XX — the space of all possible model predictions.
The fitted model y^=Py\hat{y} = P y, where P=X(XTX)−1XTP = X(X^T X)^{-1} X^T, is therefore the orthogonal projection of reality onto the model’s plane of coherence.

At the point of best alignment, two conditions hold:

  • Orthogonality: the residuals are perpendicular to the model space (XTε=0X^T\varepsilon = 0); no further adjustment of the model can reduce error.
  • Equilibrium: the model boundary passes through the “centre of gravity” of the data — the configuration of least tension.

This geometric interpretation makes regression a boundary-matching process: the model adjusts its slope (orientation) and intercept (position) until the data boundary and prediction boundary align as closely as possible under a quadratic tension field.
In probabilistic terms, this same principle arises from the maximum likelihood assumption with Gaussian noise — minimising squared error is equivalent to maximising the likelihood of the observations under a normal uncertainty field.

When data exhibit multicollinearity or noise instability, regularisation introduces curvature into the optimisation landscape:

\[ \beta_{\text{min}} = \arg\min_{\beta} \big( \|y - X\beta\|^2 + \lambda \|\beta\|^2 \big) \]

This modifies the energy surface from flat to elastic, imposing an internal tension that prevents the model from bending too sharply toward random fluctuations.

In ridge regression, this represents controlled coherence — the model retains flexibility but resists overfitting.
Similarly, LASSO regression introduces a linear penalty on the absolute values of coefficients, encouraging simplicity and sparse alignment.

In the Bayesian view, regression becomes an act of belief alignment.
The prior p(β)p(\beta) encodes the model’s expected boundary; the data likelihood p(y∣β)p(y|\beta) encodes observed constraints; the posterior

\[ p(\beta \mid y) \propto p(y \mid \beta)\,p(\beta) \]

represents their combined equilibrium.
The posterior mean is thus a weighted alignment — the least-tension point between prior belief and evidence.
This interpretation unites geometry and inference: projection and probability are the same operation, expressed in different languages.

In multiple dimensions, the covariance structure of the fitted model,

\[ \mathrm{Cov}(\hat{y}) = \sigma^2\, X (X^{T}X)^{-1} X^{T} \]

describes the uncertainty field of the alignment — the degree to which each prediction depends on the configuration of the data.
Its trace determines total variance explained, while its eigenvectors describe the principal directions of predictive sensitivity.

The quality of alignment is summarised by the coefficient of determination,

\[ R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2} \]

which measures the fraction of total variance captured by the model’s boundary.
An R2R^2 of 1 indicates perfect alignment; an R2R^2 of 0 means the model contributes nothing beyond the mean.

In Flow Algebra, regression is interpreted as boundary alignment in a coherence field.

  • The data represent a flow of empirical boundaries.
  • The model represents an abstract boundary parameterised by structure.
  • The process of regression moves the model boundary until the residual flow (the unaligned component) is minimised

The fitted model is therefore the equilibrium projection of data onto the manifold of possible explanations.
Residuals represent the tension left behind; variance explained represents the degree of structural coherence achieved.

At its heart, regression is a conversation between prediction and observation

A continuous act of reconciliation where boundaries align,
coherence restores balance,
and mathematics becomes the geometry of understanding.

Previous Chapter 37
Next Chapter 39
Related Chapters
Privacy Policy
Site Map
Kireth Research
38 Thorndale Croft, Wetwang, Driffield
East Riding of Yorkshire YO25 9XZ
Telephone:
01377 236594
Designed By
Kireth Web Design
Audio Explainer RSS Feed