Benchmark Dataset Ontology

FLOW ALGEBRA — BENCHMARK DATASET ONTOLOGY
Semantic Library | AI Core | Entity Graph | Training Process Entities
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/benchmark-dataset-ontology.html
Authors (source attribution): Mark Anthony Higgins — Sky Recursive AI
Formalisation: companion interpretation; not a verbatim extraction of embedded page rules.
Encoding: UTF-8 | Locale: en-GB

flow "BenchmarkDatasetOntology" v1.0

PURPOSE:
  Represent benchmark datasets as versioned, governed evaluation artefacts.
  Ensure comparisons are reproducible, leakage-aware and protocol-compatible.

STATE_MODEL:
  S_t = (I_t, R, B, K)
  I_t = {dataset, examples, splits, tasks, labels, predictions, metrics, runs}
  R = {schema, membership, provenance, protocol, metric, comparison}
  B = {licence, access, split integrity, leakage constraints, metric validity}
  K = {version, documentation, domain knowledge, evaluation conventions}
  E_t = expand(I_t, R)
  V_t = {x in E_t | satisfies(x, B)}
  choice_t = argmax_{x in V_t} P(x | I_t, R, B, K)
  I_(t+1) = update(I_t, choice_t, observations_t)
  NOTE: P is a ranking formalism, not an empirically calibrated probability unless specified.

ENTITIES:
  BenchmarkDataset: {id, name, version, task_ids, split_ids, provenance, licence}
  BenchmarkExample: {id, input, reference?, metadata, group_id?, split_id}
  Task: {id, input_schema, output_schema, evaluation_goal}
  ReferenceAnnotation: {example_id, label_or_target, annotation_provenance}
  Split: {id, role, example_ids, selection_policy}
  EvaluationProtocol: {id, version, preprocessing, inference_settings, scoring, exclusions}
  Metric: {id, definition, direction, aggregation, uncertainty_method}
  ModelRun: {model_id, model_version, dataset_version, protocol_version, seed?, outputs}
  ScoreReport: {run_id, metric_results, confidence_intervals?, limitations}
  LeakageFinding: {type, affected_examples, evidence, severity}

RELATIONSHIPS:
  BenchmarkDataset contains BenchmarkExample
  BenchmarkDataset defines Task
  BenchmarkDataset partitions_into Split
  BenchmarkExample may_have ReferenceAnnotation
  EvaluationProtocol evaluates ModelRun on BenchmarkDataset
  Metric scores ModelRun
  ModelRun produces ScoreReport
  LeakageFinding qualifies ScoreReport

FORMAL_EQUATIONS:
  D = {e_1, e_2, ..., e_N}
  D_train intersection D_test = empty_set, if disjointness is required by protocol
  D_val intersection D_test = empty_set, if disjointness is required by protocol
  yhat_i = f_theta(x_i; inference_settings)
  s_i = metric(yhat_i, y_i), for metrics admitting per-example scores
  M = aggregate({s_i}, protocol.aggregation)
  Accuracy = (1/N) * sum_i indicator(yhat_i == y_i), for classification
  ErrorRate = 1 - Accuracy, for the same classification setting
  Delta = M_A - M_B, only if protocols and metric directions are compatible
  CI = uncertainty_estimator(scores, sampling_unit, dependence_structure)
  NOTE: For corpus-level metrics, score the full output collection rather than
        assuming per-example decomposition.

PATHWAYS:
  1. identify task and benchmark version
  2. validate licence, provenance, schema and example identifiers
  3. verify evaluation split and contamination controls
  4. lock preprocessing, prompts, decoding and scoring protocol
  5. execute model inference; preserve predictions and run metadata
  6. compute metrics and uncertainty under declared aggregation rules
  7. inspect subgroup behaviour, failures and limitations
  8. publish reproducible score report with version identifiers

INFERENCE_RULES:
  IF dataset_version differs THEN prohibit direct score equivalence by default
  IF metric_definition differs THEN require explicit comparability justification
  IF training contamination is suspected THEN flag result as potentially biased
  IF test labels informed model selection THEN flag test-set leakage
  IF subgroup sample size is insufficient THEN avoid strong subgroup claims
  IF stochastic inference is used THEN report seeds or repeated-run variability
  IF dataset is restricted THEN preserve access and redistribution constraints
  IF annotations are uncertain THEN record uncertainty and adjudication method
  IF higher_is_better(metric) is false THEN reverse comparison direction

MINI_RECURSION_ENGINE:
  observe -> validate -> evaluate -> inspect -> revise_protocol_or_dataset
  Revision must increment version and preserve the original benchmark record.
  Never revise a locked test set using its outcomes while claiming untouched evaluation.

TRI_MAPPING:
  Entity mapping: dataset -> example -> split -> annotation -> task
  Process mapping: protocol -> inference -> metric -> score report
  Evidence mapping: provenance -> leakage checks -> uncertainty -> reproducibility

MAPPING_TABLE:
  Dataset version | immutable benchmark identity | comparability
  Split membership | partition record | leakage control
  Protocol version | scoring and inference contract | reproducibility
  Model run | predictions plus configuration | audit trail
  Score report | metric values and uncertainty | qualified comparison

EXECUTION_CONSTRAINTS:
  Validate identifiers and input/output schema before scoring.
  Require deterministic metric definitions and documented aggregation.
  Preserve run artefacts or hashes where policy permits.
  Do not claim statistical significance without an appropriate analysis.
  Do not equate benchmark performance with deployment safety or general ability.
  Honour dataset licensing, privacy and sensitive-content restrictions.

EXECUTION_GUARANTEE:
  Given valid inputs and a fully specified deterministic protocol, the flow
  defines an auditable evaluation procedure; it does not guarantee model
  quality, unbiased data, generalisation, or calibrated uncertainty.

AI_SEARCH_FLOW:
  QUERY_INTENT: benchmark dataset definition, evaluation, comparison, governance
  MATCH_ENTITIES: BenchmarkDataset, Split, Task, EvaluationProtocol, Metric
  RETRIEVE: versions, provenance, scoring contract, limitations, reports
  FILTER: task compatibility, access rights, split integrity, protocol identity
  RANK: relevance, evidence quality, reproducibility, version consistency
  RETURN: benchmark identity, evaluation procedure, score context, caveats
  CITATION_BACKLINK: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/benchmark-dataset-ontology.html

END FLOW