Evaluation Dataset Ontology

FLOW ALGEBRA — EVALUATION DATASET ONTOLOGY
Semantic Library | AI Core | Entity Graph | Training Process Entities
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/evaluation-dataset-ontology.html
Authors credited on source: Mark Anthony Higgins — Sky Recursive AI
Status: Companion formalisation; page's embedded specification was not available in retrieved HTML.
Date: 2026-10-29 | Encoding: UTF-8 | Language: en-GB

flow "EvaluationDatasetOntology" v1.0

PURPOSE:
  Represent datasets used to measure model behaviour, performance, reliability,
  generalisation, robustness, fairness and limitations under declared protocols.

STATE_MODEL:
  S = (I, R, B, K)
  I = evaluation inputs, targets, predictions, metadata and provenance
  R = partition, metric, comparison, aggregation and reporting rules
  B = dataset boundaries, access policies, leakage controls and metric validity
  K = task definitions, model versions, evaluation protocol and domain knowledge
  E_t = expand(I_t, R)
  V_t = {x in E_t | satisfies(x, B)}
  decision_t = argmax_{x in V_t} P(x | I_t, R, B, K) [only for ranked choices]
  I_(t+1) = update(I_t, decision_t, evidence_t)

ENTITIES:
  EvaluationDataset: identified collection of evaluation examples
  EvaluationExample: (example_id, input, reference?, metadata)
  ReferenceTarget: label, answer, ranking or acceptable-output specification
  Prediction: output produced by a fixed model under declared settings
  EvaluationProtocol: (task, sampling, preprocessing, metrics, aggregation)
  DatasetSplit: train | validation | test | benchmark | holdout | external
  Metric: named function of predictions, references and/or human judgements
  EvaluationRun: (dataset_version, model_version, protocol_version, timestamp)
  Result: (metric, value, sample_count, uncertainty, limitations)
  ProvenanceRecord: source, licence, collection method, transformations, version
  LeakageRecord: overlap, contamination or selection-bias evidence
  Slice: defined subgroup for stratified evaluation

RELATIONSHIPS:
  EvaluationDataset contains EvaluationExample
  EvaluationExample may_have ReferenceTarget
  EvaluationRun uses EvaluationDataset and EvaluationProtocol
  EvaluationRun evaluates ModelVersion
  Metric produces Result
  Result may_be_disaggregated_by Slice
  DatasetSplit constrains permitted tuning and reporting
  ProvenanceRecord documents EvaluationDataset
  LeakageRecord qualifies Result

FORMAL_RULES:
  D_eval = {(x_i, y_i?, m_i)} for i = 1..N
  yhat_i = f_theta(x_i; inference_config)
  score = M({(yhat_i, y_i, m_i)}_(i=1..N))
  accuracy = (1/N) * sum_i 1[yhat_i = y_i] [classification only]
  error_rate = 1 - accuracy [same classification assumptions]
  mean_loss = (1/N) * sum_i loss(yhat_i, y_i) [when targets exist]
  slice_score(g) = M({i : m_i belongs_to g}) [if sample sufficient]
  uncertainty = confidence_interval(score, sampling_design, metric_assumptions)
  overlap = identifiers_or_content_overlap(D_eval, D_train)
  test_validity = protocol_fixed AND leakage_reviewed AND reporting_complete

PATHWAYS:
  P1 Register dataset identity, task, licence and provenance
  P2 Validate schema, target quality and consent/access constraints
  P3 Freeze version and evaluation protocol before final holdout testing
  P4 Check overlap, duplicates, contamination and subgroup coverage
  P5 Run fixed model and inference configuration on eligible examples
  P6 Calculate task-appropriate metrics and uncertainty
  P7 Compare against baselines only under comparable conditions
  P8 Record errors, subgroup gaps, limitations and reproducibility metadata
  P9 Publish results with dataset/model/protocol versions and caveats

INFERENCE_RULES:
  IF dataset used for parameter fitting THEN role includes training, not independent test
  IF model selection uses a split THEN that split is validation, not untouched holdout
  IF evaluation examples overlap training material THEN flag contamination risk
  IF metric assumptions fail THEN mark metric inapplicable
  IF slice count too small THEN suppress unsupported subgroup conclusions
  IF benchmark repeatedly guides development THEN flag adaptive overfitting risk
  IF reference absent THEN use justified reference-free or human evaluation only
  IF comparison protocols differ THEN prohibit unqualified ranking
  IF distribution differs from deployment THEN qualify generalisation claims

MINI_RECURSION_ENGINE:
  observe -> validate -> evaluate -> diagnose -> revise_protocol_or_model -> repeat
  Boundary: never silently reuse an already consulted holdout as untouched test data.
  Stop when evaluation budget, protocol or governance rules require termination.

TRI_MAPPING:
  Entity -> EvaluationDataset -> versioned set of examples
  Relation -> evaluated_by -> fixed protocol and metric
  Flow -> execute -> predictions, aggregate results, provenance and caveats

MAPPING_TABLE:
  dataset_id       -> stable dataset identifier
  dataset_version  -> immutable snapshot or checksum
  example_id       -> deduplication and traceability key
  split_role       -> training/validation/test/benchmark/external designation
  model_version    -> checkpoint or deployment identifier
  protocol_version -> reproducible evaluation settings
  metric_id        -> exact metric and implementation
  result_value     -> measured outcome with unit and direction
  uncertainty      -> interval or uncertainty estimate with assumptions
  provenance       -> data lineage and usage rights

EXECUTION_CONSTRAINTS:
  Require dataset and protocol version identifiers.
  Never infer representativeness solely from dataset size.
  Prevent personal or restricted data disclosure in reports.
  Distinguish measured observations from claims of deployment performance.
  Do not guarantee unbiased, contamination-free or generalisable evaluation.
  Preserve traceable failures and missing values.

AI_SEARCH_FLOW:
  QUERY: identify evaluation dataset, task, split, metric or result
  RESOLVE: map synonyms and identifiers to ontology entities
  FILTER: enforce task compatibility, provenance and access boundaries
  EXPAND: traverse dataset -> protocol -> run -> metric -> result
  VERIFY: check versions, leakage, sample size and uncertainty
  RETURN: grounded answer with limitations and source attribution

CITATION_BACKLINK:
  https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/evaluation-dataset-ontology.html

END FLOW