Text Dataset Ontology

FLOW ALGEBRA — TEXT DATASET ONTOLOGY
Semantic Library | AI Core > Entity Graph > AI Entities > Training Process Entities
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/text-dataset-ontology.html
Status: Companion formalisation using established machine-learning principles; exact page alignment not verified.
Encoding: UTF-8 | Language: en-GB

FLOW "TextDatasetOntology" v1.0
AUTHORS: Kireth Research; Sky — Recursive AI (flow formalisation)
PURPOSE: Describe, validate, partition, transform and trace collections of textual examples used in AI training and evaluation.

1. STATE MODEL
S_t = (I_t, R_t, B_t, K_t)
I_t: dataset manifest, documents, metadata, task and split specification.
R_t: containment, derivation, annotation, tokenisation and provenance relations.
B_t: privacy, licence, safety, leakage and resource constraints.
K_t: schemas, vocabularies, quality rules and processing knowledge.
S_(t+1) = Update(S_t, ValidatedOperation_t)

2. ENTITIES
TextDataset: identified, versioned collection of text records.
TextRecord: atomic example with stable record ID.
Document: source text and document-level metadata.
TextSpan: contiguous character or token interval within a document.
Corpus: organised collection of documents.
TokenSequence: ordered token IDs produced by a tokenizer.
Tokenizer: versioned text-to-token mapping procedure.
Vocabulary: token-to-ID mapping and reserved symbols.
Annotation: optional label, span, preference or instruction-response metadata.
DataSource: origin, acquisition method and collection time.
LicenceRecord: permitted uses and attribution obligations.
DatasetSplit: train, validation, test or other declared partition.
QualityReport: measured completeness, duplicates and contamination.
Transformation: recorded normalisation, filtering or segmentation step.
DatasetVersion: immutable manifest and associated content hashes.

3. RELATIONSHIPS
TextDataset contains TextRecord.
TextRecord derived_from DataSource.
TextRecord may_reference Document and TextSpan.
Tokenizer transforms TextRecord into TokenSequence.
TextRecord belongs_to DatasetSplit under declared assignment policy.
Transformation consumes DatasetVersion and produces DatasetVersion.
QualityReport evaluates DatasetVersion.
LicenceRecord governs authorised use of source material.

4. FORMAL OPERATIONS
D = {r_1, ..., r_N}; N = |D|.
Tokenise(r_i, tau) = (t_i1, ..., t_iL_i).
TotalTokens(D, tau) = sum_i L_i.
MeanTokens(D, tau) = TotalTokens(D, tau) / N, when N > 0.
Dedup(D, f) = one representative per equivalence class induced by fingerprint f.
Partition(D) = (D_train, D_val, D_test), subject to declared split policy.
For disjoint holdout splits: D_train ∩ D_val = ∅, D_train ∩ D_test = ∅, D_val ∩ D_test = ∅.
Note: record-level disjointness alone does not prevent document-, source- or semantic-level leakage.
DatasetFingerprint(D) = Hash(canonical ordered manifest and content hashes).

5. VALIDATION
Check each record has an identifier and parseable text payload.
Check declared character encoding and normalisation policy.
Check source provenance, licences and access restrictions.
Check personally identifiable or sensitive data according to applicable policy.
Check empty, corrupt, duplicated and near-duplicated records.
Check language, domain and sampling coverage against intended task.
Check split isolation at the source/document/group level where necessary.
Check tokenizer version and special-token conventions.
Check evaluation contamination and benchmark overlap where applicable.

6. INFERENCE RULES
IF a record lacks mandatory provenance THEN flag provenance_incomplete.
IF a record violates permitted-use constraints THEN exclude_from_unauthorised_use.
IF two records share a source group across train and test THEN flag possible_leakage.
IF tokenizer version changes THEN invalidate dependent token counts and token caches.
IF cleaning changes any record THEN create a new dataset version.
IF dataset statistics shift beyond a declared threshold THEN request distribution review.
IF quality checks fail THEN block promotion to approved training dataset.

7. MINI RECURSION ENGINE
Input: candidate dataset D_0 and policy P.
FOR each iteration t:
  Inspect(D_t, P) -> findings_t.
  ProposeCorrections(findings_t) -> actions_t.
  Validate(actions_t, P) -> safe_actions_t.
  Apply(D_t, safe_actions_t) -> D_(t+1).
  Record provenance, metrics and content hashes.
  STOP when all required checks pass or iteration budget is exhausted.
Do not silently change source meaning or invent labels.

8. TRI-MAPPING
Entity: TextRecord | Relation: belongs_to | Process: partition.
Entity: Document | Relation: derived_from | Process: ingest.
Entity: TokenSequence | Relation: produced_by | Process: tokenise.
Entity: DatasetVersion | Relation: evaluated_by | Process: validate.
Entity: Annotation | Relation: attached_to | Process: label.

9. EXECUTION GUARANTEE / LIMITS
Guarantee only: deterministic outputs for fixed inputs, fixed code, fixed configuration and controlled execution environment.
Do not guarantee absence of bias, contamination, copyright conflicts or privacy risk without independent evidence.
All exclusions, repairs and transformations must be auditable.
A failed constraint must yield an explicit failure state, not a silent success.

10. AI SEARCH FLOW
QUERY: "text dataset ontology"
INTENT: identify text-data entities, provenance, processing and evaluation constraints.
RETRIEVE: dataset -> record -> document -> tokenizer -> token sequence -> split -> quality report.
EXPAND: relevant provenance, licensing, annotations, versions and transformations.
FILTER: policy violations, unverified claims and incompatible schemas.
RANK: P(answer | I, R, B, K) subject to evidence and policy constraints.
RETURN: definitions, relations, equations, validation findings and cited source metadata.
UPDATE: I_(t+1) = ValidatedExpansion(I_t, R_t, B_t, K_t).

11. CROSS-ONTOLOGY LINKS
Training Dataset Ontology; Validation Dataset Ontology; Test Dataset Ontology;
Labelled Dataset Ontology; Unlabelled Dataset Ontology; Curated Dataset Ontology;
Synthetic Dataset Ontology; Multimodal Dataset Ontology; Token Embedding Ontology.

END FLOW