Chapter 35
Entropy as Information Dispersion

Flow Algebra

flow "Entropy as Information Dispersion"

# ---------- CONCEPT ----------
# Entropy measures the dispersion of information — the uncertainty or spread within a probability field.
# It quantifies how evenly coherence is distributed across possible outcomes.
# High entropy means diffuse, balanced uncertainty; low entropy means concentrated, ordered coherence.

# ---------- STATES ----------
state InformationSystem(S):
  ProbabilityDistribution:
    description: "P = {p_i} or density p(x) — defines the weighting of possible outcomes."
  DiscreteEntropy:
    description: "H(P) = −∑ p_i log p_i — average information content of discrete outcomes."
  DifferentialEntropy:
    description: "h(p) = −∫ p(x) log p(x) dx — continuous analogue, measuring spread of density."
  JointEntropy:
    description: "H(X,Y) = −∑ p(x,y) log p(x,y) — uncertainty of a combined system."
  ConditionalEntropy:
    description: "H(X|Y) = H(X,Y) − H(Y) — residual uncertainty of X after knowing Y."
  MutualInformation:
    description: "I(X;Y) = H(X) + H(Y) − H(X,Y) — shared information between X and Y."
  KL_Divergence:
    description: "D_KL(P‖Q) = ∑ p_i log(p_i/q_i) — relative entropy between distributions (information gain)."

# ---------- PARAMETERS ----------
params:
  base: "logarithm base defines units: log₂ ⇒ bits, log_e ⇒ nats"
  normalisation: "∑ p_i = 1 or ∫ p(x) dx = 1"
  nonnegativity: "H(P) ≥ 0, D_KL(P‖Q) ≥ 0"
  closure: true

# ---------- INVARIANTS ----------
invariants:
  - "H(P) = −∑ p_i log p_i (discrete), h(p) = −∫ p(x) log p(x) dx (continuous)."
  - "H(P) maximised for uniform distribution: H_max = log n (n outcomes)."
  - "H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y)."
  - "I(X;Y) = H(X) − H(X|Y) = H(Y) − H(Y|X)."
  - "D_KL(P‖Q) = ∑ p_i log(p_i/q_i) ≥ 0, equality iff P=Q."
  - "Chain rule: H(X_1,…,X_n) = ∑ H(X_i|X_1,…,X_{i−1})."
  - "For independent variables: H(X,Y)=H(X)+H(Y)."
  - "Conditional entropy nonnegative: H(X|Y) ≥ 0."
  - "Mutual information symmetric: I(X;Y)=I(Y;X)."

# ---------- TRANSFORMS ----------
transforms:
  ComputeDiscreteEntropy(P):
    rule: "H(P) = −∑ p_i log p_i."
  ComputeDifferentialEntropy(p):
    rule: "h(p) = −∫ p(x) log p(x) dx."
  JointEntropy(X,Y):
    rule: "H(X,Y) = −∑ p(x,y) log p(x,y)."
  ConditionalEntropy(X|Y):
    rule: "H(X|Y)=H(X,Y)−H(Y)."
  MutualInformation(X,Y):
    rule: "I(X;Y)=H(X)+H(Y)−H(X,Y)."
  RelativeEntropy(P,Q):
    rule: "D_KL(P‖Q)=∑ p_i log(p_i/q_i)."
  NormaliseEntropy(H):
    rule: "H_norm = H / H_max — fractional uncertainty."
  ContinuousToDiscrete(p,bins):
    rule: "Approximate H(P_bin) ≈ h(p) + log(Δx)."
  EntropyFlowOverTime(P_t):
    rule: "Track H(P_t) as system evolves; ΔH/Δt measures information production or loss."

# ---------- MONITORS ----------
monitors:
  NormalisationCheck:
    measure: "∑ p_i or ∫ p(x) dx"
    target: 1
  NonnegativityCheck:
    measure: "H(P), D_KL(P‖Q)"
    condition: "≥ 0"
  UniformityMeasure:
    meaning: "Compare H(P) to H_max = log n; ratio shows dispersion fraction."
  IndependenceTest:
    meaning: "If H(X,Y)=H(X)+H(Y), variables independent."
  InformationFlowRate:
    measure: "dH/dt"
    meaning: "Rate of entropy change (information diffusion speed)."
  MutualInformationSymmetry:
    measure: "I(X;Y)−I(Y;X)"
    meaning: "Should equal 0 (symmetric measure)."

# ---------- EQUATIONS ----------
equations:
  - "H(P) = −\\sum_i p_i \\log p_i"
  - "h(p) = −\\int p(x) \\log p(x)\\,dx"
  - "H(X,Y) = −\\sum_{x,y} p(x,y)\\log p(x,y)"
  - "H(X|Y) = H(X,Y) − H(Y)"
  - "I(X;Y) = H(X) + H(Y) − H(X,Y)"
  - "D_{KL}(P‖Q) = \\sum_i p_i \\log \\frac{p_i}{q_i}"
  - "H_{\\text{max}} = \\log n,\\; H_{\\text{norm}} = H / H_{\\text{max}}"
  - "\\text{If independent: } H(X,Y)=H(X)+H(Y)"
  - "\\text{If deterministic: } H(X)=0"
  - "\\frac{dH}{dt} = −\\int \\dot{p}(x,t)\\log p(x,t)\\,dx"

# ---------- NOTES ----------
notes:
  - "Entropy measures how evenly coherence is distributed — it is the uncertainty embedded in a probability field."
  - "Low entropy implies structured coherence (few dominant outcomes); high entropy implies diffuse uncertainty (balanced weighting)."
  - "Mutual information measures overlap of coherence between two systems — shared predictability or dependence."
  - "KL divergence quantifies the informational distance between distributions — how much one flow of probability must shift to become another."
  - "Entropy grows when coherence spreads (diffusion of uncertainty), and decreases when information concentrates (order or structure forms)."
  - "In thermodynamic analogy, entropy corresponds to the energy of disorder; in Flow Algebra, it quantifies the spatial dispersion of coherence across possible states."
  - "Entropy dynamics describe the geometry of learning and loss — how knowledge, uncertainty, and balance evolve in probabilistic flow."