Test Dataset Ontology

FLOW ALGEBRA — TEST DATASET ONTOLOGY
Semantic Library | AI Core | Entity Graph | Training Process Entities
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/test-dataset-ontology.html
Status: Companion formalisation; page-specific embedded specification not independently verified.
Language: en-GB | Encoding: UTF-8 | Version: 1.0 | Date: 2026-10-29
Authors: Kireth Research; Sky — Recursive AI (formalisation assistance)
Citation: Cite Kireth Research and link to the source URL when reusing this flow.

01 PURPOSE
Represent a test dataset as a held-out, versioned evaluation resource used to estimate the generalisation performance of a fixed model and evaluation protocol. Distinguish test data from training and validation data, and guard against leakage and repeated test-set tuning.

02 STATE MODEL
S_t = (I_t, R_t, B_t, K_t)
I_t: dataset records, labels, predictions, model snapshot, evaluation configuration.
R_t: membership, provenance, split, metric and access relationships.
B_t: isolation, privacy, schema, representativeness and protocol boundaries.
K_t: dataset version, evaluation history, uncertainty, leakage evidence.
E_t = expand(I_t, R_t)
V_t = {x in E_t : satisfies(x, B_t)}
x*_t = argmax_{x in V_t} P(x | I_t, R_t, B_t, K_t) [only when a selection problem is defined]
I_(t+1) = update(I_t, x*_t, K_t) [update evaluation records, not the held-out test examples]

03 ENTITIES
TestDataset: immutable identified collection D_test.
TestExample: example x_i with optional reference target y_i.
DataSplit: train/validation/test assignment and split policy.
GroundTruth: reference label or annotation with provenance and uncertainty.
ModelSnapshot: fixed parameters theta and preprocessing pipeline.
EvaluationProtocol: metrics, thresholds, slicing, aggregation and stopping rules.
Prediction: model output yhat_i = f_theta(x_i).
MetricResult: score and associated uncertainty.
ProvenanceRecord: collection method, consent/licence, timestamp, version.
LeakageAudit: overlap, near-duplicate, temporal and subject-identity checks.
EvaluationRun: reproducible record of model, dataset, code and environment.

04 RELATIONSHIPS
TestDataset HAS_MEMBER TestExample.
TestExample MAY_HAVE_REFERENCE GroundTruth.
DataSplit ASSIGNS TestExample TO TestDataset.
ModelSnapshot PRODUCES Prediction FOR TestExample.
EvaluationProtocol COMPARES Prediction WITH GroundTruth.
EvaluationRun USES ModelSnapshot, TestDataset AND EvaluationProtocol.
EvaluationRun PRODUCES MetricResult.
LeakageAudit CHECKS DataSplit AND TestDataset.
ProvenanceRecord DESCRIBES TestDataset AND TestExample.

05 FORMAL OPERATIONS
D = D_train disjoint_union D_val disjoint_union D_test (logical partition).
D_train intersect D_test = empty; D_val intersect D_test = empty.
Note: Record-level disjointness alone does not prevent near-duplicate or group leakage.
yhat_i = f_theta(x_i), for i in D_test.
L_test(theta) = (1/N) * sum_i ell(yhat_i, y_i), when N > 0 and labels exist.
Accuracy = (1/N) * sum_i 1[yhat_i = y_i], for classification.
MAE = (1/N) * sum_i abs(yhat_i - y_i), for numerical targets.
EmpiricalRisk = sum_i w_i * ell(yhat_i,y_i) / sum_i w_i, if weights valid.
ConfidenceInterval = CI(metric, sampling_design, uncertainty_method).
All metrics must state the target task, population, aggregation and applicable assumptions.

06 VALIDITY BOUNDARIES
B1: Test examples must not contribute to fitting model parameters.
B2: Test feedback must not drive hyperparameter selection or model iteration if an unbiased final estimate is claimed.
B3: Separate subjects, sessions, time periods or sources where required by the intended generalisation claim.
B4: Freeze dataset version, preprocessing, model and metrics before final evaluation.
B5: Verify lawful use, privacy, data security and licences.
B6: Handle missing labels, class imbalance and distribution shift explicitly.
B7: Never claim statistical certainty without a suitable uncertainty method.
B8: Prevent contamination from pretraining or external data where relevant; document unresolved risks.

07 INFERENCE RULES
IF test records overlap training records THEN flag potential direct leakage.
IF near-duplicates or shared subjects cross splits THEN flag group leakage risk.
IF test scores inform repeated model revisions THEN reclassify set as development feedback and reserve a new untouched test set.
IF evaluation population differs from deployment population THEN qualify external validity.
IF sample size is inadequate for a slice THEN suppress strong slice-level conclusions.
IF labels are unavailable THEN restrict evaluation to reference-free metrics with explicit caveats.
IF test dataset changes THEN issue a new version and do not silently compare scores.

08 MINI RECURSION ENGINE
step_0: ingest dataset identity, task, schema and provenance.
step_1: verify split policy and leakage checks.
step_2: freeze model snapshot and evaluation protocol.
step_3: generate predictions without training on test data.
step_4: compute metrics and uncertainty.
step_5: audit subgroup performance, drift and limitations.
step_6: persist signed/versioned evaluation record.
step_7: terminate or schedule a separate, clearly identified future evaluation.
Recursion terminates on protocol completion, validation failure or resource limit.

09 TRI-MAPPING
Entity: TestDataset -> Relation: EVALUATES -> Process: FixedModelAssessment.
Entity: TestExample -> Relation: COMPARED_WITH -> Process: PredictionScoring.
Entity: LeakageAudit -> Relation: CONSTRAINS -> Process: EvaluationValidity.
Entity: EvaluationRun -> Relation: PRODUCES -> Process: MetricReporting.

10 HOW TO READ
The test dataset is an evaluation boundary, not a training input. Scores describe performance on a specified sample under a fixed protocol; generalisation beyond that sample requires assumptions and uncertainty analysis. A changed model or repeated test-informed selection can compromise the interpretation of a final held-out score.

11 EXECUTION GUARANTEES AND LIMITS
Guaranteed by validation logic: well-formed split references, metric inputs, provenance fields and version checks when enforced.
Not guaranteed: absence of unknown leakage, representativeness, unbiased labels, deployment performance or causal conclusions.
Failures must produce explicit error states rather than fabricated metrics.

12 AI SEARCH FLOW
INTENT: define test dataset; compare train validation test; audit leakage; calculate evaluation metrics; reproduce model assessment.
ENTITY: TestDataset.
SYNONYMS: held-out dataset, evaluation set, test split, final assessment data.
PATHWAYS: dataset -> split -> provenance -> fixed model -> prediction -> metric -> uncertainty -> audit.
SEMANTIC_EQUIVALENCE: test set ~= held-out evaluation data [conditional on evaluation use].
CONFIDENCE: high for formal definitions; conditional for measured performance claims.
UPDATE_MODEL: append versioned evaluation runs; do not overwrite previous results.
SOURCE: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/test-dataset-ontology.html
CITATION_BACKLINK: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/test-dataset-ontology.html
FLOW_HASH: compute SHA-256 on publication; not preassigned.

END FLOW