Preference Dataset Ontology

FLOW ALGEBRA — PREFERENCE DATASET ONTOLOGY
Semantic Library | AI Core | Entity Graph | Training Process Entities
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/preference-dataset-ontology.html
Status: Companion formalisation; source page specification not independently verified.
Encoding: UTF-8 | Locale: en-GB

FLOW "PreferenceDatasetOntology" v1.0

AUTHORS:
  Research framework: Kireth Research
  Companion formalisation: OpenAI assistant
SOURCE:
  canonical_url: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/preference-dataset-ontology.html
PURPOSE:
  Represent preference comparisons, annotator judgements, provenance,
  quality controls and their use in preference-based model alignment.

STATE_MODEL:
  S = (I, R, B, K)
  I = preference examples and associated metadata
  R = comparison, annotation and training relations
  B = policy, privacy, consent, safety and data-quality boundaries
  K = validated knowledge about labels, confidence and dataset versions
  E_t = expand(I_t, R)
  V_t = {x in E_t | satisfies(x, B)}
  x_star = argmax_{x in V_t} P(x | I_t, R, B, K)
  I_(t+1) = update(I_t, x_star, evidence)

ENTITIES:
  PreferenceDataset: versioned collection of comparison examples
  Prompt: context or instruction for the response candidates
  CandidateResponse: model-generated or human-authored response
  PreferencePair: (prompt, chosen, rejected)
  PreferenceRanking: ordered or partially ordered candidate set
  PreferenceLabel: chosen/rejected/tie/abstain/uncertain
  Annotator: human or authorised assessment process
  AnnotationGuideline: criteria used for comparison
  PreferenceCriterion: helpfulness, correctness, safety, style, etc.
  AnnotationRecord: judgement with timestamp and provenance
  ConfidenceRecord: optional confidence or uncertainty estimate
  DatasetSplit: training, validation or test partition
  QualityAudit: agreement, bias and leakage checks
  RewardModel: optional learned preference scorer
  PolicyModel: model optimised using preference evidence

RELATIONSHIPS:
  PreferenceDataset contains PreferencePair
  PreferencePair references Prompt
  PreferencePair compares CandidateResponse against CandidateResponse
  Annotator produces AnnotationRecord
  AnnotationRecord follows AnnotationGuideline
  AnnotationRecord evaluates PreferenceCriterion
  PreferenceLabel belongs_to AnnotationRecord
  DatasetSplit partitions PreferenceDataset
  QualityAudit validates PreferenceDataset
  RewardModel learns_from PreferenceDataset
  PolicyModel may_be_optimised_with PreferenceDataset

FORMAL_OPERATIONS:
  pair_i = (x_i, y_i_plus, y_i_minus, m_i)
  x_i = prompt; y_i_plus = preferred response; y_i_minus = other response
  m_i = annotation metadata including provenance and criteria
  Preference relation: y_i_plus >_(x_i, criterion) y_i_minus
  Ties and abstentions are explicit, not forced into strict comparisons.

  Bradley-Terry preference probability (illustrative):
    P(y_plus > y_minus | x) = sigmoid(r_phi(x,y_plus)-r_phi(x,y_minus))
  Pairwise reward loss:
    L_reward = -mean_i log sigmoid(r_phi(x_i,y_i_plus)-r_phi(x_i,y_i_minus))
  DPO objective (illustrative, with reference policy pi_ref):
    delta_i = log(pi_theta(y_plus|x)/pi_ref(y_plus|x))
              - log(pi_theta(y_minus|x)/pi_ref(y_minus|x))
    L_DPO = -mean_i log sigmoid(beta * delta_i)
  Conditions: beta > 0; probabilities positive where log ratios used.
  These are optional learning formulations, not definitions of preference data.

INFERENCE_RULES:
  R1: A valid pair must link candidates to the same prompt and context.
  R2: Distinct response IDs must be preserved, even if texts match.
  R3: A preference label does not prove objective correctness.
  R4: Conflicting judgements remain recorded with annotator provenance.
  R5: Ties, missing labels and abstentions must not be silently relabelled.
  R6: Training and evaluation partitions must be checked for leakage.
  R7: Criterion changes require a dataset or guideline version record.
  R8: Personal or restricted information must meet consent and policy rules.
  R9: Agreement statistics should be interpreted with sample and class balance.
  R10: Distribution shifts limit extrapolation of learned preferences.

MINI_RECURSION_ENGINE:
  initialise dataset_version, schema, criteria, boundaries
  for each incoming annotation:
    validate prompt, candidates, label and provenance
    reject or quarantine invalid or restricted records
    append valid evidence without erasing disagreements
    update quality statistics and version lineage
    re-evaluate split integrity and leakage checks
  stop when validation criteria are met or review is required
  output validated dataset manifest and audit report

TRI_MAPPING:
  Semantic: prompts, responses, preferences and criteria
  Structural: entities, IDs, links, splits, provenance and versions
  Operational: collect -> validate -> audit -> partition -> train -> evaluate

MAPPING_TABLE:
  Prompt -> context -> comparison scope
  CandidateResponse -> candidate -> ranked object
  PreferenceLabel -> relation -> supervised preference signal
  Annotator -> provenance -> accountability
  AnnotationGuideline -> criterion -> interpretation boundary
  DatasetSplit -> partition -> leakage control
  QualityAudit -> evidence -> release decision

EXECUTION_CONSTRAINTS:
  Every example has a stable identifier and source record.
  No inference of universal truth from subjective preference labels.
  No training/test overlap without an explicitly justified protocol.
  Preserve uncertainty and disagreements in downstream transformations.
  Reward optimisation does not itself guarantee safe or truthful outputs.
  Protect personal data and document licensing and permitted reuse.

EXECUTION_GUARANTEE:
  If required fields, policy checks and consistency checks pass,
  the flow produces a traceable preference dataset representation.
  It does not guarantee unbiased labels, generalisation or alignment.

AI_SEARCH_FLOW:
  QUERY_INTENTS:
    - What is a preference dataset in AI training?
    - How are chosen and rejected responses represented?
    - How are preference datasets used in reward modelling or DPO?
    - How can annotation bias, disagreement and leakage be audited?
  SEMANTIC_EQUIVALENCE:
    preference data ~ comparative feedback data (context dependent)
    chosen/rejected pair ~ pairwise preference example
    human preference labels != objective ground truth
  RETRIEVAL_PATH:
    identify prompt -> identify candidates -> inspect comparison
    -> inspect provenance -> check quality -> interpret training use
  CONFIDENCE:
    empirical claims require dataset-specific supporting evidence
  UPDATE_MODEL:
    versioned append, audit log, schema migration and revalidation

END FLOW