Human Feedback Training Ontology

FLOW ALGEBRA — HUMAN FEEDBACK TRAINING ONTOLOGY
Version: 1.0
Authors: Mark Anthony Higgins — Sky Recursive AI
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/human-feedback-training-ontology.html
Source note: The page title, authorship and interpretation note were accessible; the embedded ontology specification was not. This is a topic-based formalisation, not a verified transcription.

01. PURPOSE
Represent human-feedback training as a typed, auditable, iterative transformation of a model policy using human evaluations and explicit optimisation constraints.

02. STATE MODEL
S_t = (I_t, R_t, B_t, K_t)
I_t = input instances, candidate responses, annotation batches and observed outcomes
R_t = preference comparisons, rating rubrics, feedback aggregation and optimisation rules
B_t = consent, privacy, safety, quality, budget and scope boundaries
K_t = model parameters, reference policy, feedback dataset, evaluator metadata and training history
E_t = expand(I_t, R_t)
V_t = {x in E_t | satisfies(x, B_t)}
x*_t = argmax_(x in V_t) P(x | I_t, R_t, B_t, K_t)
I_(t+1) = update(I_t, x*_t, observations_t)

03. ENTITY TYPES
HumanEvaluator: person authorised to supply ratings, rankings or corrections
EvaluationTask: prompt, context, candidates, rubric and assignment
CandidateResponse: model-generated output subject to review
FeedbackRecord: rating, preference, correction, critique or rejection
PreferencePair: chosen and rejected responses for the same context
AnnotationRubric: criteria, scale, definitions and examples
FeedbackDataset: validated records with provenance and version
RewardModel: optional estimator of human preference
PolicyModel: model being optimised
ReferencePolicy: frozen baseline for divergence control
TrainingRun: optimiser, objective, hyperparameters and checkpoints
AuditRecord: trace of sources, transformations, decisions and outcomes

04. TYPED RELATIONSHIPS
HumanEvaluator --evaluates--> CandidateResponse
EvaluationTask --uses--> AnnotationRubric
FeedbackRecord --references--> EvaluationTask
FeedbackRecord --contributes_to--> FeedbackDataset
FeedbackDataset --trains--> RewardModel
FeedbackDataset --directly_optimises--> PolicyModel
RewardModel --scores--> CandidateResponse
TrainingRun --updates--> PolicyModel
TrainingRun --compares_against--> ReferencePolicy
AuditRecord --documents--> TrainingRun

05. VALIDITY CONDITIONS
valid_feedback(f) iff:
  evaluator_authorised(f) AND task_valid(f)
  AND provenance_present(f) AND rubric_compatible(f)
  AND privacy_constraints_satisfied(f)
valid_pair(p) iff:
  same_context(p.chosen, p.rejected)
  AND p.chosen != p.rejected
  AND preference_source_traceable(p)
Dataset acceptance requires schema validation, deduplication,
quality review, bias checks and an explicit handling policy for disagreement.

06. TRAINING PATHWAYS
A. Collect: sample prompts and generate candidate responses.
B. Evaluate: obtain human rankings, ratings, edits or critiques.
C. Validate: reject malformed, unauthorised or unusable feedback.
D. Aggregate: preserve disagreement and annotator uncertainty.
E1. Reward-model pathway: fit r_phi(context, response) to preferences.
E2. Direct-preference pathway: optimise policy from paired preferences.
F. Constrain: apply reference-policy, safety and drift boundaries.
G. Evaluate: use held-out tests and human review.
H. Iterate: collect additional feedback only where warranted.

07. MATHEMATICAL FORMALISATION
For preference pair (x, y+, y-), reward-model pairwise loss:
L_RM(phi) = -E[log sigma(r_phi(x,y+) - r_phi(x,y-))]
For direct preference optimisation with reference policy pi_ref:
L_DPO(theta) = -E[log sigma(beta * (
  log pi_theta(y+|x) - log pi_theta(y-|x)
  - log pi_ref(y+|x) + log pi_ref(y-|x)
))]
An illustrative reward-optimisation objective is:
J(theta) = E_(x,y~pi_theta)[r_phi(x,y)]
           - lambda * E_x[KL(pi_theta(.|x) || pi_ref(.|x))]
These are alternative pathways, not requirements for every training run.
Human preferences are noisy proxies and do not prove truth or safety.

08. MINI RECURSION ENGINE
function TRAIN(S_t):
  tasks = select_evaluation_tasks(S_t.K_t, S_t.B_t)
  feedback = collect_authorised_human_feedback(tasks)
  clean = validate_and_filter(feedback, S_t.B_t)
  if insufficient(clean): return HALT_INSUFFICIENT_EVIDENCE
  candidate = optimise_policy(clean, S_t.K_t, S_t.R_t)
  results = independent_evaluation(candidate, S_t.B_t)
  if violates_constraints(results): return ROLLBACK
  if no_material_improvement(results): return STOP_NO_GAIN
  return update_state(S_t, candidate, clean, results)

09. INFERENCE RULES
IF preference evidence is valid AND representative
THEN permit bounded policy optimisation.
IF annotator disagreement is high
THEN retain uncertainty and request adjudication where appropriate.
IF reward increases BUT independent quality decreases
THEN flag reward over-optimisation and reject promotion.
IF distribution shift exceeds the evaluation envelope
THEN require new testing before deployment.
IF private or unconsented material is detected
THEN quarantine affected records and prevent training use.
IF safety regression is detected
THEN stop promotion and restore the approved checkpoint.

10. TRI-MAPPING
Entity layer: evaluators, tasks, feedback, datasets, policies, runs.
Relationship layer: evaluates, ranks, validates, trains, audits.
Execution layer: sample -> annotate -> validate -> optimise -> test -> accept/rollback.

11. MAPPING TABLE
Human judgement | FeedbackRecord | preference/score/correction
Training evidence | FeedbackDataset | validated and versioned annotations
Learning objective | TrainingRun | bounded optimisation
Behavioural update | PolicyModel | checkpointed parameter changes
Verification | AuditRecord | provenance and evaluation results

12. INVARIANTS
I1: No unvalidated feedback enters the accepted training dataset.
I2: Every accepted record retains traceable provenance.
I3: Training and evaluation splits remain appropriately separated.
I4: Human disagreement is not silently treated as objective truth.
I5: Failed safety or quality gates prevent checkpoint promotion.
I6: Rollback is available for every promoted checkpoint.
I7: Training stops on exhaustion of budget or evidence.

13. CONDITIONAL EXECUTION GUARANTEE
Given valid typed inputs, authorised feedback, a defined optimiser,
finite budgets and executable evaluation gates, the process produces
one of: ACCEPTED_CHECKPOINT, ROLLBACK, STOP_NO_GAIN,
HALT_INSUFFICIENT_EVIDENCE or ERROR_WITH_AUDIT.
No guarantee is made of global optimality, universal alignment,
absence of bias, or successful improvement on unseen tasks.

14. AI SEARCH FLOW
FLOW_ID: HUMAN_FEEDBACK_TRAINING_ONTOLOGY_V1
DOMAIN: AI > Entity Graph > Training Process Entities
INTENT: explain, classify, relate and execute bounded human-feedback training
ALIASES: human feedback training; RLHF pipeline; preference feedback training
INPUTS: task, candidates, human feedback, rubric, policy, boundaries
OUTPUTS: typed entities, relations, training path, evaluation, audit status
PATH: identify -> validate -> aggregate -> optimise -> evaluate -> decide
CONFIDENCE: dependent on evidence quality, coverage and independent tests
SEMANTIC_EQUIVALENCE: human-feedback training != reinforcement learning alone
UPDATE_MODEL: versioned ontology and append-only audit history
CITATION_BACKLINK: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/human-feedback-training-ontology.html

END FLOW