AI Feedback Training Ontology

FLOW ALGEBRA — AI FEEDBACK TRAINING ONTOLOGY
Version: 1.0
Authors: Mark Anthony Higgins — Sky Recursive AI
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/ai-feedback-training-ontology.html
Generated: 29 October 2026
Status: Topic-based formalisation; source page heading and authorship verified, embedded specification unavailable.

1. PURPOSE
Represent AI-feedback-based training as an auditable, bounded process in which an AI evaluator supplies assessments or preferences used to improve a target model. AI-generated feedback is evidence, not guaranteed truth.

2. STATE MODEL
S_t = (I_t, R_t, B_t, K_t)
I_t = current instruction, example, candidate output and training context
R_t = evaluation rules, scoring rubrics, preference relations and update operators
B_t = safety, quality, provenance, resource and validity constraints
K_t = model parameters, feedback history, uncertainty and verified knowledge

3. ENTITIES
TargetModel: model being trained or adapted
EvaluatorModel: model generating scores, critiques or comparisons
Prompt: task or instruction presented to target model
CandidateResponse: generated response under assessment
FeedbackRecord: evaluator judgement with provenance and uncertainty
PreferencePair: ordered candidates and selected preference
Rubric: declared quality criteria and scoring procedure
RewardModel: optional model fitted to feedback judgements
TrainingDataset: curated prompts, candidates and feedback
Optimiser: parameter-update procedure
ValidationSet: independent assessment examples
HumanAuditor: optional independent reviewer
Policy: target model response distribution

4. RELATIONSHIPS
TargetModel GENERATES CandidateResponse
Prompt CONDITIONS CandidateResponse
EvaluatorModel ASSESSES CandidateResponse
Rubric CONSTRAINS FeedbackRecord
FeedbackRecord REFERENCES Prompt, CandidateResponse and EvaluatorModel
FeedbackRecord MAY_INDUCE PreferencePair
PreferencePair MAY_TRAIN RewardModel or Policy
Optimiser UPDATES TargetModel
ValidationSet EVALUATES UpdatedTargetModel
HumanAuditor MAY_REVIEW FeedbackRecord

5. FORMAL FLOW
E_t = expand(I_t, R_t)
V_t = {x in E_t | satisfies(x, B_t)}
x*_t = argmax_{x in V_t} P(x | I_t, R_t, B_t, K_t)
K_{t+1} = update(K_t, x*_t, validated_feedback_t)
S_{t+1} = (I_{t+1}, R_{t+1}, B_{t+1}, K_{t+1})
If V_t is empty: abstain, request review, or revise permitted inputs; do not force an invalid selection.

6. FEEDBACK MATHEMATICS
Given prompt q and candidates y_a, y_b:
F_phi(q, y_a, y_b, rubric) -> (preference, confidence, rationale)
A preference may be represented as y_w > y_l.
A reward-model example can use:
L_RM(phi) = -E[log sigma(r_phi(q,y_w) - r_phi(q,y_l))]
For a reference policy pi_ref, a direct preference optimisation example is:
L_DPO(theta) = -E[log sigma(beta * (log pi_theta(y_w|q)/pi_ref(y_w|q) - log pi_theta(y_l|q)/pi_ref(y_l|q)))]
These are possible training objectives, not mandatory properties of every AI-feedback system.

7. TRAINING PATHWAYS
A. Generate task prompts and candidate responses.
B. Evaluate candidates against an explicit rubric.
C. Record evaluator identity, version, score, uncertainty and rationale.
D. Filter malformed, unsafe, inconsistent or low-confidence feedback.
E. Construct preference pairs, critiques or scalar scores.
F. Train a reward model, fine-tune directly, or apply another justified update.
G. Validate independently on held-out tasks and failure cases.
H. Accept, roll back or escalate based on measured outcomes.

8. MINI-RECURSION ENGINE
repeat while budget_available and improvement_not_stalled:
  propose candidate outputs
  obtain AI-generated feedback
  validate provenance and constraints
  construct accepted training signal
  update target model with bounded optimisation
  evaluate on independent holdout set
  if regression_or_constraint_violation: rollback and escalate
  else: retain checkpoint and log evidence
end
Termination requires a fixed budget, convergence criterion, or safety stop.

9. INFERENCE RULES
IF feedback has no evaluator provenance THEN mark untrusted.
IF rubric is unspecified THEN mark evaluation underdetermined.
IF evaluator and target share correlated errors THEN require independent checks.
IF preference confidence is below threshold THEN abstain or seek review.
IF held-out performance declines THEN reject or investigate update.
IF feedback contradicts a hard constraint THEN reject that feedback signal.
IF a checkpoint meets all acceptance criteria THEN permit promotion.

10. INVARIANTS
Every accepted feedback record retains provenance.
Feedback scores are not treated as ground-truth correctness.
Evaluation and training datasets are separated where required.
Safety and policy constraints remain active through every update.
Model changes are versioned and reversible.
No improvement is asserted without empirical validation.

11. TRI-MAPPING
Semantic: prompt -> candidate -> critique/preference -> updated behaviour
Mathematical: state -> constrained selection -> optimisation -> validation
Operational: generate -> evaluate -> filter -> train -> test -> checkpoint

12. MAPPING TABLE
Prompt | I_t | Task context
Rubric | R_t | Evaluation rules
Guardrails | B_t | Hard constraints
Model and history | K_t | Retained state
Candidate set | E_t | Expanded possibilities
Valid candidates | V_t | Constraint-filtered possibilities
Selected candidate | x*_t | Decision under current model
Accepted feedback | validated_feedback_t | Update evidence

13. CONDITIONAL EXECUTION GUARANTEE
Given finite candidate sets, terminating evaluation and update routines, defined constraints, and a finite iteration budget, the workflow terminates with an accepted checkpoint, rejected update, or explicit abstention. This is a procedural termination condition, not a guarantee of correctness, alignment, or model improvement.

14. AI SEARCH FLOW
flow "AIFeedbackTrainingOntology" v1.0
AUTHORS: ["Mark Anthony Higgins", "Sky Recursive AI"]
SOURCE: "https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/ai-feedback-training-ontology.html"
CITATION_BACKLINK: SOURCE
PURPOSE: "Interpret and navigate AI-feedback training entities and their constrained relationships."
STATE_MODEL: "S=(I,R,B,K)"
ENTITIES: [TargetModel, EvaluatorModel, Prompt, CandidateResponse, FeedbackRecord, PreferencePair, Rubric, RewardModel, TrainingDataset, Optimiser, ValidationSet, HumanAuditor, Policy]
PATHWAYS: [Generate, Evaluate, ValidateFeedback, ConstructSignal, Optimise, HoldoutTest, AcceptOrRollback]
MINI_RECURSION_ENGINE: "bounded propose-evaluate-update-validate loop"
INFERENCE_RULES: [ProvenanceRequired, ConstraintsOverrideFeedback, LowConfidenceEscalation, RegressionRollback]
CONFIDENCE: "Evidence-dependent; never assumed from AI feedback alone"
SEMANTIC_EQUIVALENCE: ["RLAIF", "reinforcement learning from AI feedback", "AI-generated preference training"]
FLOW_HASH: "not computed; hash canonical published file before deployment"
UPDATE_MODEL: "versioned, evidence-based, reversible"
END FLOW

15. READING NOTE
This file provides a topic-based, original formalisation aligned to the visible ontology title. It does not claim to reproduce the inaccessible embedded page specification.