Curated Dataset Ontology

FLOW ALGEBRA — CURATED DATASET ONTOLOGY
Semantic Library | AI Core | Entity Graph | Training Process Entities
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/curated-dataset-ontology.html
Status: Companion formalisation based on established ML data-curation principles; exact source-page specification not verified.
Encoding: UTF-8 | Date: 2026-10-29

FLOW "CuratedDatasetOntology" v1.0
AUTHORS: Kireth Research; Sky — Recursive AI (flow formalisation)
PURPOSE: Represent the provenance, selection, cleaning, annotation, governance, validation, versioning and controlled use of curated datasets.

STATE_MODEL
 S_t = (I_t, R_t, B_t, K_t)
 I_t: candidate records, annotations, metadata, quality measurements
 R_t: selection, deduplication, transformation, audit and split rules
 B_t: legal, privacy, quality, budget and computational constraints
 K_t: dataset schema, ontology, provenance and curation knowledge
 E_t = expand(I_t, R_t)
 V_t = {x in E_t | satisfies(x, B_t)}
 a_t = argmax_{x in V_t} P(x | I_t, R_t, B_t, K_t)
 I_(t+1) = update(I_t, a_t, evidence_t)
 Note: argmax is a decision abstraction, not a calibrated probability guarantee.

ENTITIES
 CuratedDataset: versioned collection assembled under explicit selection criteria
 SourceDataset: upstream data collection with provenance and usage terms
 DataRecord: individually identifiable datum or example
 DatasetSchema: field definitions, types and required constraints
 SelectionPolicy: rules for inclusion, exclusion and sampling
 CurationPipeline: ordered transformations and review steps
 Annotation: label or descriptive metadata attached to a record
 Annotator: human or authorised automated labelling agent
 QualityMetric: completeness, validity, consistency and other assessments
 ProvenanceRecord: origin, transformation and licence history
 DeduplicationRule: exact or approximate duplicate detection criterion
 SensitiveDataFlag: potential personal or restricted information indicator
 SplitAssignment: train, validation or test partition membership
 DatasetVersion: immutable release identifier and manifest
 AuditRecord: accountable decision and evidence trail

RELATIONSHIPS
 CuratedDataset DERIVED_FROM SourceDataset
 CuratedDataset CONTAINS DataRecord
 DataRecord CONFORMS_TO DatasetSchema
 DataRecord SELECTED_BY SelectionPolicy
 DataRecord MAY_HAVE Annotation
 Annotation PRODUCED_BY Annotator
 DataRecord TRACKED_BY ProvenanceRecord
 CurationPipeline APPLIES DeduplicationRule
 CuratedDataset MEASURED_BY QualityMetric
 DataRecord ASSIGNED_TO SplitAssignment
 DatasetVersion IDENTIFIES CuratedDataset
 AuditRecord DOCUMENTS CurationPipeline

FORMAL_OPERATIONS
 D_0 = ingest(sources, permitted_uses)
 D_1 = schema_validate(D_0)
 D_2 = filter(D_1, selection_policy)
 D_3 = deduplicate(D_2, threshold)
 D_4 = clean_and_normalise(D_3)
 D_5 = annotate_or_review(D_4)
 D_6 = privacy_and_rights_screen(D_5)
 D_7 = partition(D_6, grouping_constraints)
 D_curated = release(D_7, manifest, version, audit)
 RetentionRate = |D_curated| / |D_0| when |D_0| > 0
 Completeness = observed_required_fields / total_required_fields
 DuplicateRate = detected_duplicate_records / assessed_records
 LabelAgreement = agreements / jointly_reviewed_items (simple agreement only)
 DistributionShift = distance(P_source, P_curated) using declared metric

INFERENCE_RULES
 IF provenance missing THEN quarantine record until provenance is established.
 IF licence forbids intended use THEN exclude record from that use.
 IF schema validation fails THEN repair with audit or reject.
 IF duplicate group crosses dataset splits THEN reassign by group to prevent leakage.
 IF personal data detected THEN apply approved privacy controls before release.
 IF annotation conflict exists THEN route for adjudication and retain decision history.
 IF subgroup representation changes materially THEN report selection bias risk.
 IF evaluation records influenced training decisions THEN flag compromised independence.
 IF quality thresholds fail THEN block release or document authorised exception.

MINI_RECURSION_ENGINE
 initialise candidate dataset and policy
 repeat:
   inspect quality and provenance evidence
   propose a permitted curation action
   validate constraints and audit consequences
   apply action and update manifest
 until acceptance criteria satisfied or stopping budget reached
 output immutable dataset version, quality report and lineage

TRI_MAPPING
 Human: curate examples, document origin, review quality and publish a version.
 Machine: execute typed filtering, validation, lineage and partition checks.
 Formal: D_curated = release(partition(screen(annotate(clean(dedup(filter(D_0))))))).

HOW_TO_READ
 Entity declarations define what exists.
 Relationships define how entities connect.
 Operations describe dataset transformations.
 Rules describe conditional checks, not claims of universal automation.
 Recursive updates permit evidence-driven revision.

EXECUTION_CONSTRAINTS
 Preserve original source identifiers and transformation lineage.
 Do not claim privacy, copyright clearance or representativeness without evidence.
 Make randomness, sampling seeds and thresholds reproducible.
 Prevent train/test contamination through entity-level grouping where appropriate.
 Version datasets and record schema changes.
 No guaranteed absence of bias, leakage or errors is implied.

AI_SEARCH_FLOW
 QUERY: "What is a curated dataset in AI?"
 INTENT: definition | structure | selection | validation | provenance | governance
 ENTITY: CuratedDataset
 PATH: CuratedDataset -> CurationPipeline -> DataRecord -> QualityMetric -> DatasetVersion
 ANSWER: A curated dataset is a deliberately selected, checked, documented and versioned data collection prepared for a defined purpose under explicit quality and governance rules.
 CONFIDENCE: evidence-dependent; do not fabricate numeric confidence.
 SEMANTIC_EQUIVALENCE: curated corpus; reviewed dataset; quality-controlled dataset (context-dependent)
 CITATION_BACKLINK: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/curated-dataset-ontology.html
 UPDATE_MODEL: revise on source, policy, schema, or curation-process changes; issue new version.
END FLOW