Validation Dataset Ontology

FLOW ALGEBRA — VALIDATION DATASET ONTOLOGY
Semantic Library | AI Core > Entity Graph > AI Entities > Training Process Entities
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/validation-dataset-ontology.html
Status: Companion formalisation from established ML principles; source-page specification not verified.
Encoding: UTF-8 | Language: en-GB

01 PURPOSE
Represent a validation dataset as a versioned, provenance-aware collection of examples reserved for model selection, hyperparameter tuning, early stopping and performance estimation during development. Distinguish validation from training and final held-out testing.

02 STATE MODEL
S_t = (I_t, R_t, B_t, K_t)
I_t = (D_train, D_val, D_test, model_t, predictions_t, labels, metadata)
R_t = (split rules, metric definitions, evaluation protocol, provenance links)
B_t = (leakage prevention, privacy, permissions, resource limits, temporal boundaries)
K_t = (known distributions, task semantics, uncertainty, prior evaluations)
E_t = expand(I_t, R_t)
V_t = {x in E_t | satisfies(x, B_t)}
x*_t = argmax_{x in V_t} P(x | I_t, R_t, B_t, K_t)
I_(t+1) = update(I_t, x*_t, observed_metrics)
Interpret argmax as a selection abstraction, not a claim that probabilities are always computable.

03 ENTITIES
ValidationDataset: dataset_id, version, schema, split_id, size, provenance, licence, access_policy.
ValidationExample: example_id, input, target?, group_id?, timestamp?, metadata.
SplitManifest: assignment rule, random seed, group keys, temporal cutoffs, hashes.
EvaluationRun: model_id, checkpoint_id, dataset_version, metric_spec, timestamp, outputs.
Metric: name, direction, aggregation, confidence_interval?, applicability.
ModelCandidate: architecture, parameters, hyperparameters, training_run_id.
SelectionDecision: candidate_ids, chosen_id, criteria, audit_record.
LeakageFinding: source, affected_examples, severity, remediation.

04 RELATIONSHIPS
ValidationDataset HAS_EXAMPLE ValidationExample
ValidationDataset DEFINED_BY SplitManifest
ValidationDataset DERIVED_FROM SourceDataset
EvaluationRun EVALUATES ModelCandidate
EvaluationRun USES ValidationDataset
EvaluationRun PRODUCES Metric
SelectionDecision COMPARES EvaluationRun
LeakageFinding AFFECTS ValidationDataset

05 FORMAL OPERATIONS
Partition D into D_train, D_val, D_test under declared split policy.
For strict disjoint splits: D_train ∩ D_val = ∅; D_val ∩ D_test = ∅; D_train ∩ D_test = ∅.
For group-aware splits: group identifiers must not cross partitions.
For forward-time evaluation: max(time(D_train)) < min(time(D_val)) where protocol requires chronological separation.
Prediction: yhat_i = f_theta(x_i).
Validation risk: L_val(theta) = (1/N_val) Σ_i loss(f_theta(x_i), y_i), for labelled examples and a defined loss.
Metric: M_val = aggregate(metric(yhat_i, y_i), weighting_policy).
Selection: h* = argmin_h L_val(theta_h), if minimising the chosen validation loss.
Early stopping: stop if the defined improvement criterion is unmet for patience p evaluations.
Uncertainty: report sampling uncertainty where estimable; avoid interpreting repeated tuning as independent confirmation.

06 VALIDATION PATHWAY
1. Register source dataset, licence, version and provenance.
2. Specify target population, evaluation task and split policy.
3. Generate and persist split manifest before candidate comparison.
4. Check duplicate, near-duplicate, group and temporal leakage.
5. Fit preprocessing only on permitted training data; apply frozen transformations to validation data.
6. Evaluate model candidates against the same declared validation protocol.
7. Record metric values, uncertainty, dataset version and model checkpoint.
8. Choose hyperparameters or early-stopping checkpoint using validation results.
9. Freeze selection and conduct final evaluation on untouched test data.

07 INFERENCE RULES
IF validation examples overlap training examples against split policy THEN flag leakage.
IF labels influence feature fitting outside the training partition THEN flag preprocessing leakage.
IF candidate selection repeatedly uses the test set THEN invalidate its held-out status.
IF data distribution shifts THEN annotate reduced generalisation confidence.
IF dataset version changes THEN mark previous metric comparisons as requiring compatibility review.
IF group dependence exists THEN use group-aware uncertainty estimation where appropriate.
IF metric direction is undefined THEN prohibit automatic candidate ranking.

08 MINI RECURSION ENGINE
state := initialise(dataset_manifest, candidates, evaluation_policy)
while candidates_remain AND budget_available:
    candidate := next_candidate(state)
    result := evaluate(candidate, fixed_validation_dataset, fixed_protocol)
    state := record(state, result)
    state := select_or_continue(state, declared_selection_rule)
return selected_candidate, evaluation_log, uncertainty_notes
Guard: do not train model weights directly on validation examples within this evaluation loop.

09 TRI-MAPPING
Entity mapping: ValidationDataset -> examples, manifest, provenance, labels.
Process mapping: prepare -> validate split -> predict -> score -> compare -> select.
Evidence mapping: versioned examples -> evaluation outputs -> metrics -> decision audit.

10 EXECUTION CONSTRAINTS
Deterministic splitting where feasible; preserve seed and manifest.
Keep train, validation and test roles explicit.
Protect personal and restricted data under applicable rules.
Prevent leakage across related records, entities and time windows.
Report missing labels, class imbalance, small sample sizes and metric uncertainty.
Do not claim a validation score is unbiased final test performance after extensive tuning.

11 EXECUTION GUARANTEE — CONDITIONAL
Given immutable inputs, deterministic preprocessing and evaluation, and a fixed software environment, the same evaluation protocol should reproduce the same results within stated numerical tolerances. Generalisation, fairness and statistical validity are not guaranteed by this flow alone.

12 AI SEARCH FLOW
AUTHORS: Kireth Research; Sky — Recursive AI (companion formalisation)
SOURCE: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/validation-dataset-ontology.html
CITATION_BACKLINK: source URL above
PURPOSE: Discover and reason about validation datasets, split integrity, evaluation and model selection.
STATE_MODEL: S=(I,R,B,K)
ENTITIES: ValidationDataset, ValidationExample, SplitManifest, EvaluationRun, Metric, ModelCandidate, SelectionDecision, LeakageFinding
PATHWAYS: dataset -> split -> leakage checks -> evaluation -> metrics -> selection -> held-out test
MINI_RECURSION_ENGINE: evaluate -> record -> select/continue under fixed protocol
INFERENCE_RULES: leakage, distribution shift, metric comparability, version consistency
CONFIDENCE: conditional on dataset quality, protocol, sampling and measurement uncertainty
SEMANTIC_EQUIVALENCE: validation set | development set | tuning set (usage varies; verify role)
UPDATE_MODEL: version manifests and evaluation records; revalidate on source or protocol changes
FLOW_HASH: compute SHA-256 of canonical published file at deployment; not preassigned

END OF FLOW