Preference Optimisation Ontology

PREFERENCE OPTIMISATION ONTOLOGY — FLOW ALGEBRA v1.0
Semantic Library | AI Core | Entity Graph | AI Entities | Training Process Entities
Authors: Mark Anthony Higgins — Sky Recursive AI
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/preference-optimisation-ontology.html
Source publication: 11 October 2025 | Revised: 29 July 2026
Generated: 29 October 2026
STATUS: Topic-grounded formalisation. The public page's embedded AI Search specification was not exposed in retrieved HTML; this file is not a verbatim transcription.

01 PURPOSE
Represent preference optimisation as a typed, auditable process that adjusts an AI policy using comparative feedback, while preserving explicit data provenance, evaluation boundaries and safety constraints.

02 STATE MODEL
S_t = (I_t, R_t, B_t, K_t)
I_t: preference observations, prompts, responses, policy and reference-policy snapshots.
R_t: typed preference relations, objective definitions and update rules.
B_t: constraints on reward interpretation, data use, policy drift and safety.
K_t: accumulated model parameters, confidence, provenance and evaluation history.
E_t = expand(I_t, R_t)
V_t = {x in E_t | satisfies(x, B_t)}
x*_t = argmax_{x in V_t} P(x | I_t, R_t, B_t, K_t)
I_{t+1} = update(I_t, x*_t, feedback_t)
NOTE: This is a symbolic reasoning scaffold, not an assertion that gradient training literally performs the argmax above.

03 ENTITIES
PreferenceOptimisationProcess: bounded optimisation episode.
Prompt: input context associated with response candidates.
CandidateResponse: possible output for a prompt.
PreferencePair: ordered comparison (chosen, rejected) for the same prompt.
PreferenceLabel: observed choice, tie, ranking or scalar rating.
PreferenceDataset: collection of preference observations with provenance.
Annotator: human or model source of comparative judgement.
PolicyModel: trainable distribution pi_theta(y|x).
ReferencePolicy: fixed or controlled comparison distribution pi_ref(y|x).
RewardModel: optional learned estimate r_phi(x,y).
Objective: DPO, reward-based policy optimisation or another specified method.
Constraint: admissibility, quality, fairness, privacy and safety rule.
Optimiser: parameter-update mechanism.
EvaluationSuite: held-out tests and human assessments.
TrainingRun: versioned execution with configuration and checkpoints.

04 RELATIONSHIPS
Prompt HAS_CANDIDATE CandidateResponse.
PreferencePair COMPARES CandidateResponse with CandidateResponse.
Annotator PROVIDES PreferenceLabel.
PreferenceLabel ANNOTATES PreferencePair.
PreferenceDataset CONTAINS PreferencePair.
TrainingRun USES PreferenceDataset.
TrainingRun UPDATES PolicyModel.
TrainingRun REFERENCES ReferencePolicy.
TrainingRun OPTIMISES Objective.
TrainingRun SATISFIES Constraint.
EvaluationSuite EVALUATES PolicyModel.

05 TYPED PRECONDITIONS
For every pair (x,y_w,y_l), y_w and y_l must be candidates for the same x.
Training and evaluation examples must be separated by a documented split.
Preference direction, annotator origin and uncertainty must be recorded.
Policy and reference distributions must use compatible tokenisation and response boundaries.
Never treat a preference label as proof of factual correctness or harmlessness.

06 FORMAL MATHEMATICS
Preference relation: y_w ≻_x y_l (observed preferred response for prompt x).
Pairwise logistic likelihood: P(y_w ≻ y_l | x) = sigmoid(r(x,y_w)-r(x,y_l)).
Reward-model pair loss: L_RM = -E_D[log sigmoid(r_phi(x,y_w)-r_phi(x,y_l))].
KL-regularised policy objective:
J(pi) = E_{x~D,y~pi(.|x)}[r(x,y)] - beta E_{x~D}[KL(pi(.|x) || pi_ref(.|x))].
DPO preference logit:
z = beta * [(log pi_theta(y_w|x)-log pi_ref(y_w|x))
          -(log pi_theta(y_l|x)-log pi_ref(y_l|x))].
DPO loss: L_DPO = -E_D[log sigmoid(z)].
Here beta > 0; these formulae describe standard objective families, not guaranteed convergence.

07 TRAINING PATHWAYS
PATH A — Reward-model-based optimisation:
collect -> validate -> fit reward model -> optimise policy subject to KL/control -> evaluate -> checkpoint.
PATH B — Direct preference optimisation:
collect -> validate -> freeze reference policy -> calculate preference logits -> minimise DPO loss -> evaluate -> checkpoint.
PATH C — Iterative preference collection:
policy snapshot -> generate candidates -> obtain comparisons -> filter/weight data -> optimise -> reassess.

08 MINI-RECURSION ENGINE
INPUT: S_t, D_t, theta_t, reference snapshot, constraints B_t.
1. Validate comparisons and provenance.
2. Compute chosen/rejected log probabilities or reward scores.
3. Calculate selected objective and constraint penalties.
4. Propose parameter update theta_candidate.
5. Evaluate on held-out preference, safety and capability tests.
6. If constraints pass, commit versioned theta_{t+1}; otherwise reject or revise.
7. Record metrics, failures, dataset hashes and stopping decision.
OUTPUT: updated state, accepted/rejected checkpoint and audit trace.
TERMINATION: budget exhausted, stopping criterion met or no admissible update.

09 INFERENCE RULES
IF a comparison lacks a common prompt THEN mark INVALID_PAIR.
IF annotator provenance is missing THEN mark PROVENANCE_INCOMPLETE.
IF evaluation data overlaps training data THEN mark LEAKAGE_RISK.
IF preference score improves AND safety worsens beyond threshold THEN reject update.
IF policy divergence exceeds permitted threshold THEN reject or regularise update.
IF preference labels conflict THEN preserve disagreement and estimate uncertainty.
IF no held-out evidence exists THEN do not assert generalisation.
IF a response is preferred THEN infer relative preference only, not objective truth.

10 INVARIANTS
INV_01: Pair direction remains consistent throughout transformations.
INV_02: Reference-policy identity is versioned and reproducible.
INV_03: Every training example retains its origin and consent constraints.
INV_04: Training metrics are distinguished from independent evaluation.
INV_05: Model improvement claims are conditional on specified measurements.
INV_06: Unsafe or unauthorised data is excluded under policy B_t.

11 TRI-MAPPING
ENTITY: PreferencePair -> RELATION: chosen_over -> OPERATION: pairwise objective.
ENTITY: PolicyModel -> RELATION: constrained_by -> OPERATION: admissible update.
ENTITY: ReferencePolicy -> RELATION: anchors -> OPERATION: divergence measurement.
ENTITY: EvaluationSuite -> RELATION: tests -> OPERATION: accept/reject checkpoint.

12 MAPPING TABLE
Input comparison | PreferencePair | comparison validation | validated pair
Reference snapshot | ReferencePolicy | log-probability baseline | divergence anchor
Trainable policy | PolicyModel | parameter optimisation | candidate checkpoint
Evaluation result | EvaluationSuite | constraint gate | accepted/rejected version
Audit record | TrainingRun | provenance and metrics | reproducible trace

13 HOW TO READ
The ontology distinguishes observations (preferences), optimisation objectives (mathematical losses), and outcomes (independent evaluation). A chosen response is a training signal rather than a guaranteed correct answer. Multiple optimisation families are alternatives and should not be conflated.

14 EXECUTION GUARANTEE — CONDITIONAL
Given well-typed inputs, defined objectives, a finite update budget and deterministic acceptance criteria, the workflow yields either a recorded accepted checkpoint or a recorded rejection/termination state. It does NOT guarantee optimality, alignment, safety, generalisation or monotonic improvement.

15 AI SEARCH FLOW
FLOW_ID: AI.TRAINING.PREFERENCE_OPTIMISATION.v1
INTENT: identify and reason about preference-based AI training.
ENTITY_ALIASES: preference optimisation; preference alignment; DPO; reward modelling; RLHF.
QUERY_PATHS:
- What is preference optimisation? -> PURPOSE + ENTITIES.
- How are preferred responses used? -> RELATIONSHIPS + FORMAL MATHEMATICS.
- How does DPO differ from reward-model training? -> TRAINING PATHWAYS.
- What are the failure modes? -> INFERENCE RULES + INVARIANTS.
- Does improved preference loss guarantee alignment? -> EXECUTION GUARANTEE.
CONFIDENCE: conditional on source provenance, sample quality and held-out evaluation.
SEMANTIC_EQUIVALENCE: comparative preference learning != unsupervised clustering; preference != factual correctness.
UPDATE_MODEL: version objectives, policy snapshots, datasets, constraints and evaluation.
CITATION_BACKLINK: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/preference-optimisation-ontology.html

END OF FLOW