Training Checkpoint Ontology

FLOW ALGEBRA v1.0
TRAINING CHECKPOINT ONTOLOGY
Semantic Library | AI Core | Entity Graph | AI Entities | Training Process Entities

AUTHORS: Mark Anthony Higgins — Sky Recursive AI
SOURCE: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/training-checkpoint-ontology.html
STATUS: Topic-based formalisation; source page content not independently retrieved.
LANGUAGE: en-GB
PURPOSE: Represent, validate, persist, restore and compare AI training checkpoints as versioned training-state entities.

1. ENTITY DEFINITION
TrainingCheckpoint := immutable or versioned snapshot of a training run at a specified step, together with metadata needed to interpret or resume training.
A checkpoint is not necessarily sufficient for exact replay; completeness depends on what was captured and the execution environment.

2. STATE MODEL
S_t = (I_t, R_t, B_t, K_t)
I_t := input and observed training state at step t
R_t := relations among run, model, optimiser, dataset and checkpoint
B_t := boundaries: storage, privacy, hardware, format, retention, compatibility
K_t := knowledge: configuration, provenance, validation evidence
E_t = expand(I_t, R_t)
V_t = {x in E_t | satisfies(x, B_t, K_t)}
x*_t = argmax_{x in V_t} P(x | I_t, R_t, B_t, K_t)
I_(t+1) = update(I_t, x*_t)
The selection expression is a conceptual ranking rule, not a calibrated probability claim.

3. ENTITIES
TrainingRun(run_id, model_id, dataset_version, configuration)
Checkpoint(checkpoint_id, run_id, global_step, epoch, created_at)
ModelState(parameters, buffers, architecture_version)
OptimiserState(algorithm, moments, learning_rate_state)
SchedulerState(schedule_type, scheduler_step)
RandomState(framework_rng, accelerator_rng, data_worker_rng)
DataPipelineState(sampler_position, shuffle_seed, preprocessing_version)
MetricsSnapshot(loss, validation_metrics, evaluation_protocol)
Manifest(file_list, content_hashes, schema_version, dependencies)
StorageArtifact(uri, format, size, checksum)
EnvironmentSnapshot(framework_version, libraries, device_topology)

4. RELATIONSHIPS
Checkpoint --belongs_to--> TrainingRun
Checkpoint --captures--> ModelState
Checkpoint --may_capture--> OptimiserState
Checkpoint --may_capture--> SchedulerState
Checkpoint --may_capture--> RandomState
Checkpoint --may_capture--> DataPipelineState
Checkpoint --records--> MetricsSnapshot
Checkpoint --described_by--> Manifest
Checkpoint --stored_as--> StorageArtifact
Checkpoint --derived_from--> PreviousCheckpoint
Checkpoint --restored_into--> TrainingSession

5. FORMAL OPERATIONS
capture(S_t, policy) -> C_t
serialize(C_t, format) -> bytes
hash(bytes) -> digest
persist(bytes, location) -> artifact
verify(artifact, digest) -> boolean
load(artifact, compatibility_context) -> C'_t
restore(C'_t, environment) -> S'_t
resume(S'_t, next_batch) -> S'_(t+1)
compare(C_a, C_b) -> differences
retain(C, retention_policy) -> keep | archive | delete

6. CHECKPOINT LIFECYCLE
RUNNING -> CAPTURING -> SERIALISING -> WRITING -> VERIFYING -> AVAILABLE
CAPTURING | SERIALISING | WRITING | VERIFYING -> FAILED
AVAILABLE -> RESTORING -> RESUMED
AVAILABLE -> ARCHIVED -> DELETED (when permitted by retention policy)

7. INVARIANTS
INV-01: Every checkpoint has a unique identifier and run reference.
INV-02: The manifest declares schema version and expected artefacts.
INV-03: A checkpoint is AVAILABLE only after successful integrity checks.
INV-04: A failed or partial write is never advertised as a complete checkpoint.
INV-05: Model architecture and tensor shapes must be compatible with restoration.
INV-06: Resume capability must be declared: weights_only | training_state | replay_oriented.
INV-07: Sensitive data and credentials must not be embedded without explicit authorisation.
INV-08: Retention and deletion respect applicable governance requirements.

8. TRAINING PATHWAYS
P1 PERIODIC: train -> threshold(step/time) -> capture -> verify -> register.
P2 METRIC_TRIGGERED: evaluate -> metric improves -> capture -> mark best.
P3 INTERRUPTION: detect shutdown -> attempt safe capture -> validate outcome.
P4 RESTORE: select compatible checkpoint -> verify -> load -> reconcile state -> resume.
P5 ROLLBACK: detect regression -> select earlier verified checkpoint -> restore.
P6 EXPORT: select weights-only checkpoint -> remove training-only state -> publish with provenance.

9. MATHEMATICAL FORMALISATION
Let theta_t be model parameters and o_t optimiser state.
C_t = (theta_t, o_t, q_t, r_t, d_t, m_t, p_t)
where q_t is scheduler state, r_t random state, d_t data pipeline state,
m_t metadata/metrics and p_t provenance.
C_t = capture(S_t) is a lossy projection if any required state is omitted.
Integrity(C_t) = AND_i [H(file_i) = manifest.hash_i]
Compatible(C_t, E) = schema_ok AND architecture_ok AND dependency_policy_ok
ResumeExact(C_t, E) requires sufficiently complete deterministic state and a compatible execution environment.
Even with full capture, nondeterministic kernels or distributed scheduling can prevent bitwise replay.
StorageCost = SUM_i size(file_i) + metadata_overhead
CheckpointInterval should balance recovery loss, write cost and storage constraints.

10. MINI-RECURSION ENGINE
INPUT: training state, checkpoint policy, prior manifest
STEP 1: inspect run progress and trigger conditions
STEP 2: expand candidate checkpoint actions
STEP 3: filter by integrity, storage, compatibility and privacy boundaries
STEP 4: choose eligible action under the policy
STEP 5: capture and write to temporary storage
STEP 6: verify hashes, metadata and completeness
STEP 7: atomically publish or register completed checkpoint where supported
STEP 8: update retention index and next-checkpoint policy
OUTPUT: verified checkpoint reference or explicit failure record
REPEAT: at subsequent eligible training events

11. INFERENCE RULES
IF manifest missing THEN status = unverified.
IF checksum mismatch THEN restore_allowed = false.
IF optimiser state missing THEN exact_optimiser_resume = false.
IF RNG state missing THEN deterministic_replay_not_guaranteed = true.
IF architecture mismatch THEN require_conversion_or_reject = true.
IF validation metric improves under fixed protocol THEN candidate_best = true.
IF write incomplete THEN status = failed_or_partial.
IF storage quota exceeded THEN apply retention policy before new write.
IF checkpoint is weights_only THEN inference_load_may_be_possible; training_resume_is_not_guaranteed.

12. TRI-MAPPING
CONCEPT -> FORMAL ENTITY -> EXECUTION
Training snapshot -> Checkpoint(C_t) -> capture()
Data integrity -> Integrity(C_t) -> verify()
State recovery -> restore(C_t,E) -> load + reconcile
Training continuity -> resume(S'_t) -> next optimisation step
Provenance -> Manifest + lineage -> audit/export
Retention -> Policy(C_t) -> archive/delete

13. MAPPING TABLE
Source run | TrainingRun | run_id | provenance
Parameter state | ModelState | theta_t | recovery
Optimiser state | OptimiserState | o_t | training continuity
Randomness | RandomState | r_t | replay support
Data position | DataPipelineState | d_t | batch continuity
Checkpoint artefact | StorageArtifact | uri + hash | persistence
Validation | MetricsSnapshot | score + protocol | selection

14. CONDITIONAL EXECUTION GUARANTEE
Given a consistent capture, supported serialisation format, complete manifest,
verified storage artefacts and a compatible restore environment, the flow
can reconstruct the state represented by the checkpoint or fail explicitly.
It does not guarantee identical future training trajectories, model quality,
or recovery of state that was never recorded.

15. AI SEARCH FLOW
flow: TrainingCheckpointOntology
version: 1.0
intent: [define_checkpoint, capture_training_state, validate_checkpoint, restore_training_run, compare_checkpoints]
entities: [TrainingRun, Checkpoint, ModelState, OptimiserState, SchedulerState, RandomState, DataPipelineState, Manifest]
relations: [belongs_to, captures, records, stored_as, derived_from, restored_into]
pathways: [periodic, metric_triggered, interruption, restore, rollback, export]
confidence: conditional_on_manifest_integrity_and_environment_compatibility
semantic_equivalence:
  - training checkpoint
  - model checkpoint
  - training state snapshot
  - saved training state
update_model: append_versioned_checkpoint_with_provenance
citation_backlink: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/training-checkpoint-ontology.html

END FLOW