Pretraining Process Ontology

FLOW ALGEBRA — PRETRAINING PROCESS ONTOLOGY
Semantic Library | AI Core | Entity Graph | AI Entities | Training Process Entities
Version: 1.0 | Date: 2026-10-29 | Encoding: UTF-8
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/pretraining-process-ontology.html
Status: Companion formalisation from established ML principles; source-page specification not independently verified.

1. PURPOSE
Represent pretraining as a traceable sequence of dataset preparation, objective selection, forward evaluation, gradient computation, optimiser updates, validation, and checkpointing.

2. ENTITY SET
E = {PretrainingRun, Model, Tokeniser, Corpus, Document, Sample, Batch,
     Objective, Loss, ParameterTensor, Gradient, Optimiser, Scheduler,
     TrainingStep, Epoch, ValidationSet, Metric, Checkpoint, ProvenanceRecord}.

3. RELATIONS
usesCorpus(PretrainingRun, Corpus)
usesTokeniser(PretrainingRun, Tokeniser)
trains(PretrainingRun, Model)
usesObjective(PretrainingRun, Objective)
contains(Corpus, Document)
yields(Document, Sample)
groups(Batch, Sample)
produces(TrainingStep, Gradient)
updates(Optimiser, ParameterTensor)
assesses(ValidationSet, Model)
snapshots(Checkpoint, Model)
records(ProvenanceRecord, PretrainingRun)

4. STATE MODEL
S_t = (I_t, R_t, B_t, K_t)
I_t = current batch, model outputs, observed losses and metrics
R_t = sampling, tokenisation, objective and optimisation rules
B_t = resource, privacy, licensing, stability and compute boundaries
K_t = model parameters, optimiser state, scheduler state and run history

Operational state:
P_t = (theta_t, omega_t, eta_t, step_t, rng_t, data_cursor_t, metrics_t)
where theta denotes model parameters, omega optimiser state, eta learning rate.

5. PRETRAINING FLOW
F0: establish lawful dataset provenance and intended use
F1: clean, deduplicate, partition and document corpus
F2: tokenise samples; construct input and target tensors
F3: sample batch under deterministic or recorded stochastic policy
F4: forward pass to compute predictive distributions
F5: compute objective and aggregate masked loss
F6: backpropagate gradients; accumulate if configured
F7: clip/validate gradients and perform optimiser step
F8: update scheduler, counters and monitoring
F9: validate on held-out material without training on it
F10: checkpoint model and recovery metadata
F11: repeat until stopping criterion; release documented artefact

6. FORMAL EQUATIONS
Autoregressive objective:
L_AR(theta) = - E_{x~D} [sum_{t=1}^{T} m_t log p_theta(x_t | x_<t)] / E_{x~D}[sum_t m_t]
Here m_t is a loss mask; normalisation is implementation-dependent.

Masked language modelling (alternative objective):
L_MLM(theta) = - E_{x~D,M} [sum_{t in M} log p_theta(x_t | x_visible)] / E[|M|]

Gradient: g_t = nabla_theta L_B(theta_t)
SGD: theta_{t+1} = theta_t - eta_t g_t
Generic optimiser: (theta_{t+1}, omega_{t+1}) = Update(theta_t, omega_t, g_t, eta_t)
Gradient accumulation: g_eff = (1/A) sum_{a=1}^{A} g_a, for equal-weight microbatches.
Token throughput = processed_tokens / elapsed_seconds.
Perplexity = exp(mean negative log-likelihood per evaluated token), where defined.

7. RECURSIVE ENGINE
E_t = expand(I_t, R_t)
V_t = {x in E_t : satisfies(x, B_t)}
a_t = argmax_{x in V_t} P(x | I_t, R_t, B_t, K_t)
P_{t+1} = transition(P_t, a_t, observed_gradient, observed_metrics)
Note: argmax is a decision-policy abstraction, not a claim that gradient descent itself performs probabilistic argmax selection.

8. INFERENCE RULES
IF corpus provenance is unknown THEN flag compliance review.
IF training and validation samples overlap materially THEN flag leakage.
IF loss is non-finite THEN skip or abort update according to recovery policy.
IF gradient norm exceeds threshold THEN apply configured clipping.
IF validation degrades beyond patience THEN consider early stopping.
IF checkpoint lacks optimiser/scheduler/RNG metadata THEN flag incomplete exact-resume support.
IF training objective changes THEN record new run configuration/version.
IF model quality improves on one metric THEN do not infer general capability improvement without corroboration.

9. TRI-MAPPING
Entity: PretrainingRun | Relation: trains(Model) | Behaviour: iterates optimisation steps
Entity: Corpus | Relation: supplies(Sample) | Behaviour: provides training observations
Entity: Objective | Relation: scores(ModelOutput) | Behaviour: computes loss
Entity: Optimiser | Relation: updates(ParameterTensor) | Behaviour: changes weights
Entity: ValidationSet | Relation: assesses(Model) | Behaviour: estimates held-out performance
Entity: Checkpoint | Relation: snapshots(RunState) | Behaviour: supports restoration

10. EXECUTION CONSTRAINTS
- Every tensor operation must satisfy shape and dtype requirements.
- Training/validation/test partitions must respect leakage controls.
- Record dataset licences, filtering rules and version identifiers.
- Store reproducibility metadata including seeds and software versions.
- Apply privacy and security controls to sensitive or untrusted content.
- No finite-data or finite-compute guarantee of convergence or generalisation.
- A checkpoint guarantees recoverability only if its contents and integrity are sufficient.

11. AI SEARCH FLOW
QUERY: pretraining process, self-supervised training, corpus, optimisation, checkpoint
RESOLVE: identify run -> objective -> data -> steps -> metrics -> artefacts
PATH: PretrainingRun -> Corpus -> Batch -> Objective -> Gradient -> Optimiser -> Model
PATH: PretrainingRun -> ValidationSet -> Metric -> Checkpoint
RETURN: definitions, dependencies, equations, constraints, evidence and provenance
CONFIDENCE: distinguish mathematical definitions from empirical performance claims

12. HOW TO READ
Start with the entities and relations, follow F0–F11, inspect the equations,
then evaluate inference rules and execution constraints before drawing conclusions.

13. EXECUTION GUARANTEE
The specification provides a consistent conceptual process and testable invariants.
It does not guarantee that a particular training run succeeds, converges, or yields a safe model.

AUTHORS: Kireth Research; Sky — Recursive AI (companion formalisation)
CITATION_BACKLINK: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/pretraining-process-ontology.html