Skip to content

IncQL RFC 040: Interoperability semantic profiles

  • Status: Draft
  • Created: 2026-05-30
  • Author(s): Danny Meijer (@dannymeijer)
  • Related:
  • IncQL RFC 000 (core language model and layer boundaries)
  • IncQL RFC 002 (Apache Substrait integration)
  • IncQL RFC 004 (execution context)
  • IncQL RFC 007 (Prism logical planning and optimization engine)
  • IncQL RFC 008 (optimizer boundary, statistics, cost-based optimization, and adaptive execution)
  • IncQL RFC 012 (unified scalar expression surface)
  • IncQL RFC 013 (function catalog program)
  • IncQL RFC 024 (function extension policy)
  • IncQL RFC 027 (relational evidence program)
  • IncQL RFC 028 (semantic identity and target model)
  • IncQL RFC 029 (typed metadata attachments)
  • IncQL RFC 030 (Prism lineage graph)
  • IncQL RFC 031 (local inspection APIs and artifacts)
  • IncQL RFC 032 (execution observations)
  • IncQL RFC 033 (adapter requirements and coverage)
  • IncQL RFC 036 (governed plan bundle)
  • IncQL RFC 038 (evidence exchange bridges)
  • IncQL RFC 041 (Prism plan ingress and external client frontends)
  • Issue: IncQL #74
  • RFC PR: IncQL #60
  • Written against: Incan v0.3-era IncQL
  • Shipped in: —

Summary

This RFC defines interoperability semantic profiles for IncQL evidence. A profile describes the semantic environment a plan is being received from, compared with, targeted at, or observed under: an IncQL baseline, client protocol, plan ingress frontend, execution engine, adapter binding, SQL dialect, catalog/schema system, transformation project, interchange consumer, or conformance baseline. Profiles give ingress coverage records, adapter requirements, coverage records, execution observations, plan diffs, bundles, and exchanges a shared context without making any external system the owner of IncQL relational meaning.

Motivation

Interoperability requires more than lowering a plan and asking whether an adapter has a support flag. Different target environments can share the same relational vocabulary while differing on edge semantics: type coercion, decimal overflow, timestamp and timezone behavior, identifier resolution, null and NaN ordering, collation, case sensitivity, function definitions, aggregate edge cases, window defaults, nested data behavior, row ordering, and fallback execution.

If IncQL does not name the semantic profile used for an inspection or execution, those assumptions will be scattered across adapters, Substrait metadata, docs, and runtime diagnostics. That would make coverage hard to trust. A plan could appear portable while relying on target-specific behavior that was never recorded as evidence.

Profiles provide the missing layer between IncQL-authored semantics, plan ingress, and adapter coverage. Prism remains the source of authored and rewritten relational meaning. Profiles describe source and target environments well enough for IncQL to produce ingress diagnostics, requirements, coverage records, and observations against them.

Profiles are intentionally ecosystem-neutral, but concrete profiles may describe systems and formats such as Oracle, PostgreSQL, SQL Server, MySQL, Athena, Presto, Trino, Spark, Snowflake, BigQuery, Redshift, Databricks, Glue Data Catalog, Hive Metastore, dbt, Airflow, MWAA, Dagster, Prefect, OpenLineage, DataHub, OpenMetadata, or Great Expectations. Listing a system as a possible profile target does not make that system normative for IncQL semantics.

Goals

  • Define semantic profiles as versioned evidence records.
  • Allow profiles for IncQL baselines, client protocols, plan ingress frontends, execution engines, adapter bindings, SQL dialects, catalog/schema systems, transformation projects, interchange consumers, and conformance baselines.
  • Name the semantic dimensions that affect relational correctness and evidence interpretation.
  • Let adapter requirements and coverage records state which profile they were evaluated against.
  • Let execution observations report the profile requested before execution and the profile observed at runtime when available.
  • Keep profiles local and open, without requiring a hosted registry or managed control plane.
  • Keep external target profiles non-authoritative for IncQL semantics.

Non-Goals

  • Defining a profile for one specific external engine.
  • Making any external engine, SQL dialect, or interchange format the normative IncQL semantic model.
  • Defining SQL transpilation, physical planning, or backend execution strategies.
  • Defining transformation-project semantics as IncQL semantics.
  • Defining a full conformance test suite.
  • Defining a global registry of every engine version or deployment configuration.
  • Guaranteeing semantic equivalence merely because a profile name is present.

Guide-level explanation (how authors think about it)

Most authors should encounter profiles through inspection, coverage, and execution evidence:

from pub::incql.inspect import inspect_plan

inspection = inspect_plan(summary)
profile = inspection.semantic_profile("portable_relational")

requirements = inspection.adapter_requirements(profile)
coverage = session.check_coverage(summary, target_profile=profile)

The exact API names are illustrative. The important model is that the target profile is explicit. A coverage report should be able to say which semantic profile was used and which dimensions are covered, constrained, mismatched, or unknown.

Execution can attach the same evidence context:

result = session.collect(summary, target_profile=profile)
observation = result.execution_observation()

assert observation.requested_profile == profile.id

If the runtime adapter reports a different engine version, configuration, or semantic mode than the requested profile expected, the observation should record that difference as structured evidence.

Reference-level explanation (precise rules)

IncQL must define an interoperability semantic profile record. A profile record must include:

  • profile identity
  • profile schema version
  • target class
  • profile name or family
  • source
  • target version information when available
  • target configuration fingerprint when available
  • semantic dimensions
  • evidence references
  • confidence or completeness
  • diagnostics

Target class must distinguish at least:

  • incql_baseline
  • client_protocol
  • plan_ingress_frontend
  • execution_engine
  • adapter_binding
  • sql_dialect
  • catalog_schema_system
  • transformation_project
  • interchange_consumer
  • conformance_baseline

Concrete profile families may be narrower than target class names. For example, a sql_dialect target class may include Oracle, PostgreSQL, SQL Server, or MySQL profiles; an execution_engine target class may include Athena, Presto, Trino, Spark, Snowflake, BigQuery, Redshift, or Databricks profiles; a catalog_schema_system target class may include Glue Data Catalog or Hive Metastore profiles; and a transformation_project target class may include dbt-shaped project profiles.

Semantic dimensions must be represented as structured records rather than free-form prose. Initial dimensions should include, where applicable:

  • type system and implicit coercion
  • numeric and decimal semantics
  • temporal, timezone, and calendar semantics
  • boolean, null, and NaN semantics
  • string comparison, collation, and case sensitivity
  • identifier resolution and catalog naming
  • schema catalog, partition, and external table metadata semantics
  • transformation project selection, materialization, test, and metadata semantics
  • client session state and configuration semantics
  • relation ordering and determinism
  • aggregate and grouping edge semantics
  • window frame and ordering semantics
  • nested, variant, and semi-structured data semantics
  • function and operator identity
  • extension and fallback behavior
  • plan-stage observability

A semantic dimension record must include dimension identity, lifecycle, declared behavior when known, source, evidence references, confidence, and diagnostics. A dimension may be exact, constrained, unknown, or not_applicable. Unknown dimensions must not be treated as matching IncQL semantics.

IncQL may define profile assessments that compare a plan or bundle with a profile. A profile assessment must include the plan target, profile identity, affected semantic targets, assessed dimensions, result state, evidence references, confidence, and diagnostics.

Profile assessment result state must distinguish at least:

  • matched: IncQL can determine that the plan's required semantics match the profile for the assessed dimension
  • constrained: the profile can satisfy the dimension only under recorded restrictions
  • mismatched: the profile does not satisfy the plan's required semantics for the dimension
  • unknown: IncQL cannot determine whether the profile satisfies the dimension
  • not_applicable: the dimension does not apply to the plan or target profile

Adapter requirements and coverage records may cite profile records and profile assessments. If coverage depends on a profile, the coverage record must identify the profile. Coverage evaluated under one profile must not be reused under a different profile unless the evidence proves that the relevant semantic dimensions are equivalent.

Execution observations may include a requested profile and an observed profile. The requested profile is the semantic profile used during pre-execution inspection or coverage checks. The observed profile records runtime facts reported by the adapter, such as engine version, adapter version, semantic mode, or relevant configuration. A mismatch between requested and observed profiles must be diagnostic evidence. It must not silently rewrite the plan's authored semantics.

Profiles must not replace Prism semantic targets, lineage edges, adapter requirements, or execution observations. They provide context for evidence. They do not create fields, lineage, policy decisions, quality observations, or coverage states by themselves.

Serialized artifacts that include profile records must distinguish missing profile evidence from an empty or fully matching profile assessment.

Design details

Syntax

This RFC introduces no authoring syntax.

Semantics

Semantic profiles are evidence contexts. They describe the target environment against which IncQL evidence is checked, exported, or observed. They do not define IncQL relational meaning.

Profiles may be authored, built into IncQL, imported from artifacts, produced by adapters, or observed during execution. The source and lifecycle must be recorded so tools can distinguish a trusted built-in profile from an adapter-reported runtime observation or an imported profile.

Interaction with other IncQL surfaces

Prism remains the source of authored and rewritten relational meaning. Profile assessments consume Prism targets, lineage, schema flow, function registry facts, ingress coverage records, and adapter requirements. They must not infer semantic structure from backend plan strings or external client protocol node identifiers.

Plan ingress frontends may use profile evidence when decoding and analyzing external client plans. A Spark Connect frontend, for example, may use a client protocol profile to decide identifier resolution, function aliases, coercion behavior, and unsupported-feature diagnostics before Prism produces an analyzed plan.

Substrait lowering may carry or reference profile evidence, but Substrait must not be the only profile evidence store.

Function registry entries may contribute profile dimensions when function identity, determinism, null behavior, extension behavior, or backend availability affects compatibility.

Adapter coverage records should cite the profile used for evaluation when the answer depends on target semantics. Execution observations should report runtime profile facts when adapters can provide them.

Governed plan bundles may include profile records and profile assessments so downstream tools can understand which target environments were checked.

Evidence exchange bridges may project profile evidence into external formats or ingest external project artifacts with profile context. Lossy exports and lossy imports must report dimensions that could not be represented.

Compatibility / migration

Existing plans and adapters remain valid without profile evidence. Tools that require profile evidence must report missing profiles as unsupported or unknown evidence rather than assuming portability.

Profile schemas must be versioned from the start. Profile names that appear in serialized artifacts must be stable public vocabulary or explicitly marked as local/private.

Alternatives considered

  • Use adapter support flags only. Rejected because support depends on target semantics, engine version, configuration, and execution mode.
  • Use Substrait as the profile model. Rejected because Substrait is an interchange boundary and does not capture every IncQL evidence dimension.
  • Make one external engine profile normative. Rejected because IncQL needs to interoperate with multiple targets without importing one target's semantics as the language definition.
  • Rely only on conformance tests. Rejected because tests are valuable evidence but do not replace structured profile records, coverage states, or diagnostics.
  • Leave profiles to downstream integrations. Rejected because independent profile reconstruction would cause drift across adapters, CI, notebooks, agents, transformation projects, and governance exchanges.

Drawbacks

  • Profiles add another evidence concept that must stay distinct from requirements and coverage.
  • Profile dimension vocabulary will require maintenance as IncQL and target environments grow.
  • Early profiles may contain many unknown dimensions, which can make reports feel conservative.
  • Runtime-observed profiles can differ from requested profiles, requiring clear diagnostics.

Layers affected

  • IncQL specification — semantic profile records, dimensions, and assessment states become part of the relational evidence vocabulary.
  • IncQL library package — inspection, coverage, bundle, and export APIs must be able to expose profile records when available.
  • Execution / interchange — sessions and adapters may report requested and observed profile evidence without owning IncQL semantics.
  • Documentation — docs must explain profiles as evidence contexts, not as alternative semantic authorities.

Unresolved questions

  • Which semantic dimensions are mandatory in the first implementation?
  • Should built-in IncQL profiles live in core or in optional integration packages?
  • How should profile records compare target configurations without leaking sensitive deployment details?
  • Should conformance test results become profile evidence in this RFC or a later RFC?
  • Which transformation-project profile dimensions are needed before exchange bridges can safely emit test and metadata suggestions?