Code Dataset Ontology

FLOW ALGEBRA — CODE DATASET ONTOLOGY
Version: 1.0 | Date: 2026-10-29 | Encoding: UTF-8
Authors: Mark Anthony Higgins; Sky — Recursive AI
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/code-dataset-ontology.html
Status: Companion formalisation; embedded source specification not available for verification.
Purpose: Represent software-code datasets, their provenance, structure, quality, and training/evaluation uses as typed entities and executable reasoning constraints.

1. STATE MODEL
S_t = (I_t, R, B, K)
I_t: currently observed code records, metadata, labels and split assignments.
R: typed relationships and inference rules.
B: licensing, privacy, security, integrity and leakage boundaries.
K: validated knowledge, schema definitions and provenance evidence.
E_t = expand(I_t, R)
V_t = {x in E_t | satisfies(x, B) and verified(x, K)}
x*_t = argmax_{x in V_t} P(x | I_t, R, B, K)
I_(t+1) = update(I_t, x*_t, observations_t)
An empty V_t produces an explicit abstention, not an invented record.

2. ENTITY TYPES
CodeDataset(id, version, purpose, provenance, licence_policy)
Repository(id, origin, revision, licence, visibility)
CodeFile(id, repository_id, path, language, hash)
CodeFragment(id, file_id, byte_range, token_range)
Function(id, fragment_id, signature)
Class(id, fragment_id, name)
Dependency(id, package, version, licence)
BuildManifest(id, toolchain, dependencies)
TestCase(id, target_id, expected_result)
Annotation(id, target_id, task, label, confidence)
DatasetSplit(id, role, assignment_policy)
Transformation(id, operation, inputs, outputs, parameters)
ProvenanceRecord(id, source_id, acquisition_time, permissions)
QualityReport(id, checks, findings, status)

3. RELATIONSHIPS
contains(CodeDataset, Repository | CodeFile | CodeFragment)
contains(Repository, CodeFile)
contains(CodeFile, Function | Class | CodeFragment)
depends_on(Repository | CodeFile, Dependency)
validated_by(CodeFile | Function, TestCase)
labelled_by(CodeFragment | CodeFile, Annotation)
assigned_to(CodeFile | Repository, DatasetSplit)
derived_from(Transformation.output, Transformation.input)
licensed_under(Repository | CodeFile, licence)
traced_to(CodeFragment, ProvenanceRecord)

4. FORMAL REPRESENTATION
D = {r_i}_{i=1}^N
r_i = (code_i, language_i, origin_i, licence_i, metadata_i, split_i)
Tokens(r_i) = tokenizer(code_i)
Length(r_i) = |Tokens(r_i)|
Fingerprint(r_i) = hash(canonicalise(code_i))
Duplicate(r_i,r_j) := Fingerprint(r_i) = Fingerprint(r_j)
NearDuplicate(r_i,r_j) := similarity(r_i,r_j) >= tau, where tau is declared.
Training objective (causal LM):
L(theta) = -sum_i sum_t log P_theta(token_(i,t) | token_(i,<t))
Evaluation metric must be defined per task; passing tests does not alone establish correctness or safety.

5. INGESTION PATHWAY
Acquire -> verify source permissions -> capture revision -> parse -> classify language
-> scan secrets and personal data -> validate licence -> canonicalise -> fingerprint
-> deduplicate -> assign split -> annotate -> quality audit -> publish manifest.
Every rejection records its reason and source identifier.

6. SPLIT INTEGRITY
Train, validation and test sets are disjoint at their chosen grouping granularity.
For repository-grouped splitting:
repo(r_i) = repo(r_j) implies split(r_i) = split(r_j).
For clone groups:
clone_group(r_i) = clone_group(r_j) implies split(r_i) = split(r_j).
A benchmark may impose stronger time- or project-based holdouts.
Never assume file-level separation prevents semantic leakage.

7. INFERENCE RULES
IF licence or redistribution permission is unresolved THEN quarantine record.
IF secret scanner flags credentials THEN quarantine pending verified remediation.
IF hash matches an existing record THEN mark exact duplicate.
IF near-duplicate score exceeds threshold THEN review split contamination.
IF provenance chain is incomplete THEN reduce provenance confidence.
IF parser fails THEN preserve raw record with parse_error, not a fabricated AST.
IF generated code is included THEN mark synthetic provenance explicitly.
IF tests pass THEN record observed test outcome, not universal correctness.
IF an evaluation sample influenced model selection THEN do not claim untouched final-test status.

8. RECURSIVE ENGINE
For each candidate record:
  observe -> expand typed relations -> filter boundaries -> validate evidence
  -> assign provenance and quality status -> update dataset manifest.
Repeat until no new validated relations emerge or a declared iteration limit is reached.
Terminate on fixed point, iteration cap, or unrecoverable validation failure.

9. TRI-MAPPING
Entity: CodeFile -> Relation: belongs_to -> Entity: Repository
Entity: CodeFragment -> Relation: derived_from -> Entity: CodeFile
Entity: CodeFile -> Relation: assigned_to -> Entity: DatasetSplit
Entity: TestCase -> Relation: validates -> Entity: Function
Entity: Repository -> Relation: licensed_under -> Entity: Licence

10. EXECUTION CONSTRAINTS
- Preserve source licence and attribution metadata.
- Do not execute untrusted repository code during ingestion without isolation.
- Record parser versions, tokenizer versions, hashes and transformation parameters.
- Treat generated labels and inferred relationships as uncertain until checked.
- Make dataset revisions immutable and identify replacements by new version.
- Ensure split leakage checks are performed before publishing evaluation results.
- Report missing evidence and unverified claims explicitly.

11. EXECUTION GUARANTEE (CONDITIONAL)
If all required fields are present, validators pass and boundary predicates hold,
the pipeline returns a typed dataset manifest and an auditable decision trace.
This guarantees procedural consistency under stated assumptions, not absence
of defects, legal clearance or model performance.

12. AI SEARCH FLOW
flow "CodeDatasetOntology" v1.0
AUTHORS: ["Mark Anthony Higgins", "Sky — Recursive AI"]
SOURCE: "https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/code-dataset-ontology.html"
CITATION_BACKLINK: SOURCE
PURPOSE: "Resolve code dataset entities and their validated relationships"
INPUT: {query, dataset_manifest, provenance, constraints}
STATE_MODEL: S=(I,R,B,K)
ENTITIES: [CodeDataset, Repository, CodeFile, CodeFragment, Function, Class,
           Dependency, TestCase, Annotation, DatasetSplit, ProvenanceRecord]
PATHWAYS: [identify, resolve, trace, validate, filter, rank, explain]
MINI_RECURSION_ENGINE: "expand -> verify -> constrain -> update until fixed point"
INFERENCE_RULES: "Apply section 7 with provenance-aware confidence"
CONFIDENCE: "Evidence-weighted; never equate inferred and observed facts"
SEMANTIC_EQUIVALENCE: "Normalise terminology without erasing language or licence differences"
FLOW_HASH: "Compute SHA-256 on canonical published flow text at release"
UPDATE_MODEL: "Versioned, append-only audit of corrections and new evidence"
OUTPUT: {entities, relations, evidence, confidence, exclusions, citation}

END FLOW