Training Dataset Ontology

FLOW ALGEBRA — TRAINING DATASET ONTOLOGY
Version: 1.0 | 29 October 2026 | Encoding: UTF-8
Authors: Mark Anthony Higgins — Sky Recursive AI
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/training-dataset-ontology.html
Status: Companion formalisation; embedded source specification not accessible for exact comparison.
Purpose: Describe training datasets as versioned, provenance-aware collections of examples used to fit machine-learning models.

01. STATE MODEL
S_t = (I_t, R, B, K)
I_t = dataset state (records, splits, schema, transforms, versions, provenance)
R = dataset validation, selection, transformation, partitioning and batching rules
B = constraints (privacy, consent, licensing, integrity, leakage prevention, compute)
K = ontology of datasets, examples, labels, features, splits, sources and training runs
E_t = expand(I_t, R)
V_t = {x in E_t : satisfies(x, B)}
a_t = argmax_{x in V_t} P(x | I_t, R, B, K)
I_(t+1) = update(I_t, a_t)
NOTE: The probabilistic selection expression is a reasoning abstraction, not a claim that every dataset operation is stochastic.

02. ENTITIES
TrainingDataset: named collection D = {e_1, ..., e_N} with version and intended use.
TrainingExample: e_i = (x_i, y_i?, m_i) where x=input, y=optional target, m=metadata.
Feature: measured or derived input attribute.
Label: supervised target or annotation; may be absent for self-supervised learning.
DataSource: origin, acquisition time, licence, consent and lineage.
Schema: feature types, required fields, allowable values and target definition.
DatasetSplit: disjoint train, validation and test partitions where applicable.
Transformation: cleaning, filtering, normalisation, tokenisation or augmentation.
Batch: selected group of examples used for a training computation.
DatasetVersion: immutable manifest plus content checksums and transformation history.
QualityReport: missingness, duplicates, class balance, coverage and anomalies.
TrainingRun: model, objective, dataset version, split, seed and configuration.

03. RELATIONSHIPS
TrainingDataset contains TrainingExample.
TrainingExample conforms_to Schema.
TrainingExample originates_from DataSource.
TrainingExample may_have Label.
Transformation consumes DatasetVersion and produces DatasetVersion.
DatasetSplit partitions TrainingDataset.
Batch samples_from TrainingSplit.
TrainingRun uses DatasetVersion and training partition.
QualityReport evaluates DatasetVersion.
DatasetVersion derives_from parent DatasetVersion.

04. FORMAL OPERATIONS
D = {e_i}_{i=1}^N; N = |D|.
validate(D, schema) -> (D_valid, rejected, report).
clean(D_valid, rules) -> D_clean.
partition(D_clean, seed, policy) -> (D_train, D_val, D_test).
If partitions are disjoint: D_train ∩ D_val = D_train ∩ D_test = D_val ∩ D_test = ∅.
If exhaustive: D_train ∪ D_val ∪ D_test = D_clean.
For a finite training set and batch size b > 0: batches_per_epoch = ceil(|D_train| / b) unless drop_last is enabled.
Empirical risk: L_D(theta) = (1/N) sum_i loss(f_theta(x_i), y_i) for labelled examples.
Mini-batch estimate: g_B = (1/|B|) sum_(i in B) grad_theta loss(f_theta(x_i), y_i).
For self-supervised training replace y_i with a target generated by the specified objective.
Class proportion: p_c = count(label=c) / count(labelled examples).
Missing rate for field j: m_j = missing_count(j) / N.

05. VALIDATION AND BOUNDARIES
B1: Dataset must have a stable identifier, schema and version.
B2: Verify source permissions, licences, privacy requirements and permitted uses.
B3: Prevent exact and near-duplicate leakage across evaluation boundaries where relevant.
B4: Split related subjects, groups or time periods according to evaluation design.
B5: Fit preprocessing statistics on training data only; apply frozen transform to validation/test.
B6: Preserve lineage and transformation configuration.
B7: Do not equate a low training loss with generalisation.
B8: Sampling and augmentation must not silently change evaluation-set semantics.
B9: Report exclusions, missingness, imbalance and known representation gaps.
B10: Hashes detect changes but do not establish truth, fairness or legal permission.

06. INFERENCE RULES
IF schema invalid THEN quarantine offending examples and record reason.
IF train/test overlap detected THEN flag leakage and block trustworthy evaluation.
IF source permission absent THEN require review before use.
IF labels missing AND supervised objective required THEN request annotation or change objective.
IF dataset version changes THEN invalidate dependent reproducibility claims until rerun.
IF class imbalance is material THEN assess weighting, stratification and appropriate metrics.
IF distribution shifts THEN assess retraining and out-of-distribution performance.
IF temporal forecasting THEN use time-aware split rather than arbitrary random split.

07. RECURSIVE ENGINE
Initialise D_0 from source manifests.
For each iteration t:
  inspect quality(D_t), provenance(D_t), coverage(D_t)
  propose transformations T_t subject to boundaries B
  apply T_t to produce candidate D_(t+1)
  validate schema, leakage, permissions and checksums
  accept only if criteria pass; otherwise retain D_t and log rejection
Stop when quality thresholds met, no authorised improvement remains, or budget exhausted.
Each accepted version must be traceable to its parent and transformations.

08. TRI-MAPPING
Entity -> Relation -> Operation
TrainingDataset -> contains -> enumerate_examples
TrainingExample -> conforms_to -> validate_schema
DatasetVersion -> derives_from -> trace_lineage
DatasetSplit -> partitions -> evaluate_separation
Batch -> samples_from -> construct_minibatch
TrainingRun -> uses -> pin_dataset_version
QualityReport -> evaluates -> audit_quality

09. HOW TO READ
The dataset is an identifiable, versioned collection, not merely a folder of files.
The state model tracks what is known, how it may change and which changes are allowed.
Operations define transformations; boundaries govern whether they are acceptable.
Recursive refinement improves quality without losing lineage or evaluation integrity.

10. EXECUTION GUARANTEE — CONDITIONAL
If inputs satisfy the declared schema, operations are deterministic under pinned versions/seeds,
and validation checks pass, the flow guarantees a reproducible processing trace.
It does NOT guarantee dataset correctness, representativeness, absence of bias,
legal compliance, model performance or scientific validity.

11. AI SEARCH FLOW
FLOW_NAME: TrainingDatasetOntology
FLOW_VERSION: 1.0
DOMAIN: AI > Entity Graph > AI Entities > Training Process Entities
PRIMARY_ENTITY: TrainingDataset
INTENT: identify, validate, version, partition and use machine-learning datasets
INPUTS: dataset_manifest, examples, schema, source_metadata, intended_task, split_policy
OUTPUTS: validated_dataset_version, split_manifest, quality_report, lineage_graph
PATHWAYS: discover -> validate -> clean -> partition -> version -> train -> audit
SEMANTIC_EQUIVALENCE: training corpus; ML training set; model training dataset
CONFIDENCE: conditional on source integrity and completed checks
UPDATE_MODEL: append-only versions with immutable provenance records
CITATION_BACKLINK: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/training-dataset-ontology.html
SOURCE_CREDIT: Mark Anthony Higgins — Sky Recursive AI, Kireth Research
FLOW_HASH: compute SHA-256 after publication; no hash asserted here

END FLOW