Multimodal Dataset Ontology

FLOW ALGEBRA — MULTIMODAL DATASET ONTOLOGY
Semantic Library | AI Core > Entity Graph > AI Entities > Training Process Entities
Source: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/multimodal-dataset-ontology.html
Status: Companion formalisation based on established ML principles; exact page specification not verified.
Encoding: UTF-8 | Locale: en-GB

FLOW "MultimodalDatasetOntology" v1.0
AUTHORS: Kireth Research; Sky — Recursive AI (companion formalisation)
PURPOSE: Represent, validate, partition and process datasets containing two or more modalities with explicit alignment, provenance and missingness semantics.

STATE_MODEL
 S_t = (I_t, R_t, B_t, K_t)
 I_t = dataset samples, modality payloads, annotations and metadata
 R_t = modality membership, alignment, temporal and semantic relations
 B_t = access, consent, schema, quality, privacy and resource constraints
 K_t = known modality types, transformations, split policies and validation rules
 E_t = expand(I_t, R_t)
 V_t = {x in E_t | satisfies(x, B_t)}
 x*_t = argmax_{x in V_t} P(x | I_t, R_t, B_t, K_t)
 I_(t+1) = update(I_t, x*_t)
 NOTE: argmax is a conceptual selection operator, not a calibrated probability guarantee.

ENTITIES
 MultimodalDataset: id, version, licence, provenance, splits, schema
 Sample: sample_id, group_id, timestamp, modality_set, annotations
 Modality: type in {text, image, audio, video, sensor, spatial, tabular, other}
 Payload: uri_or_blob, media_type, encoding, checksum, dimensions
 Alignment: source_sample, target_sample, relation, confidence, offset
 Annotation: label, span_or_region, annotator, method, confidence
 Missingness: modality, reason, expected, available
 Transformation: operation, inputs, outputs, parameters, version
 DatasetSplit: name, sample_ids, policy, seed
 QualityRecord: checks, results, thresholds, reviewer
 ConsentRecord: subject_or_source, permissions, restrictions, expiry

RELATIONSHIPS
 Dataset HAS_SAMPLE Sample
 Sample HAS_MODALITY Modality
 Modality HAS_PAYLOAD Payload
 Sample HAS_ANNOTATION Annotation
 Sample ALIGNED_WITH Sample
 Sample MEMBER_OF DatasetSplit
 Payload DERIVED_BY Transformation
 Payload GOVERNED_BY ConsentRecord
 Sample HAS_QUALITY QualityRecord

FORMAL_OPERATIONS
 M(s) = set of observed modalities in sample s
 multimodal(s) iff |M(s)| >= 2
 aligned(a,b) iff shared_entity(a,b) or valid_temporal_link(a,b)
 D = {s_1, ..., s_N}
 D_train union D_val union D_test = D (when all samples assigned)
 D_train intersect D_val = empty; likewise other split pairs
 group(s_i) = group(s_j) => split(s_i) = split(s_j) when group isolation required
 x_m = encode_m(payload_m)
 z = fuse({x_m : m in M(s)}, alignment(s), missingness(s))
 L = sum_{s in D_train} w_s * loss(f_theta(s), target(s)) / sum_{s in D_train} w_s
 NOTE: Fusion and loss functions are task-dependent; target(s) may be absent in self-supervised tasks.

INFERENCE_RULES
 IF |M(s)| < 2 THEN flag NOT_MULTIMODAL_SAMPLE; dataset may still be multimodal overall.
 IF alignment is claimed AND evidence is absent THEN mark UNVERIFIED_ALIGNMENT.
 IF sample_group spans evaluation splits THEN flag POTENTIAL_LEAKAGE.
 IF modality payload is absent THEN record explicit missingness; do not invent content.
 IF payload checksum differs THEN quarantine for integrity review.
 IF licence or consent disallows use THEN exclude from disallowed processing.
 IF modality preprocessing changes THEN record version and revalidate comparability.
 IF labels are model-generated THEN retain synthetic-label provenance.

MINI_RECURSION_ENGINE
 1. Ingest dataset manifest and declared modalities.
 2. Resolve sample identifiers, payload references and permissions.
 3. Validate schemas, hashes, alignment and modality availability.
 4. Deduplicate and partition by appropriate leakage groups.
 5. Transform modalities with versioned pipelines.
 6. Assess representation, quality and missingness per split.
 7. Emit validated manifest and audit record; iterate on failed checks.
 TERMINATE when checks pass, limits are reached or unresolved items are quarantined.

TRI_MAPPING
 Entity: MultimodalDataset -> Relation: HAS_SAMPLE -> Process: ingest
 Entity: Sample -> Relation: HAS_MODALITY -> Process: encode
 Entity: Sample -> Relation: ALIGNED_WITH -> Process: align
 Entity: Payload -> Relation: DERIVED_BY -> Process: transform
 Entity: DatasetSplit -> Relation: CONTAINS -> Process: evaluate

HOW_TO_READ
 Each entity is a typed record; relations connect records; rules specify checks;
 operations describe possible computational steps, not universal implementation requirements.

EXECUTION_CONSTRAINTS
 Preserve stable sample IDs and lineage across all transformations.
 Respect licence, consent, privacy and data minimisation restrictions.
 Prevent subject, session, scene or near-duplicate leakage where relevant.
 Require deterministic split manifests for reproducibility.
 Treat alignment confidence as evidence metadata, not proof.
 Do not assume every sample has every modality.

EXECUTION_GUARANTEE
 If inputs satisfy the declared schema and checks execute successfully,
 output manifests retain traceable modality, split and provenance metadata.
 No guarantee of dataset truth, fairness, model accuracy or generalisation.

AI_SEARCH_FLOW
 INTENT: explain, locate, validate or compare multimodal dataset entities
 QUERY_TERMS: multimodal dataset, paired modalities, alignment, fusion, provenance,
              missing modality, dataset split, data leakage, cross-modal learning
 PATHWAY: query -> identify entity -> inspect relations -> apply constraints
          -> validate evidence -> return answer with uncertainty and provenance
 SEMANTIC_EQUIVALENCE:
   multimodal data ~ multi-modal dataset (context dependent)
   paired image-text ~ aligned image and text (only if pairing verified)
 CONFIDENCE: evidence-based; unknown when supporting records absent
 UPDATE_MODEL: append versioned changes, retain prior provenance and audit trail
 CITATION_BACKLINK: https://kirethresearch.org/booksai-core/entity-graph/ai-entities/training-process-entities/multimodal-dataset-ontology.html
END FLOW