From intent to execution

One typed path through the system

This is the Prism-backed LazyFrame path from checked intent to a materialized DataFusion result. Each boundary has one job, so changing a surface or backend cannot silently change the plan’s meaning.

The representation path is not the literal public call stack: callers hand a Prism-backed `LazyFrame` to Session; Session triggers the Prism-to-Substrait handoff, validates logical reads, supplies registrations, and dispatches its configured adapter.

The separation is deliberate. Authoring surfaces express relational work. Prism is IncQL’s internal logical planning engine: it stores the authored plan, derives narrow semantics-preserving views, and keeps origin information intact. It is not the IncQL brand mark, the interchange format, or the execution engine.

Substrait carries portable logical meaning across the boundary. Session then supplies the runtime context that the logical plan intentionally lacks: registered sources, its configured backend selection, execution, materialization, and writes. The adapter may plan for its engine, but it does not become the semantic owner of the query.

Read the exact contracts in Prism RFC 007, Substrait RFC 002, and Execution context.

Correlated locally

Plan evidence and execution evidence stay distinct

For a Prism-backed LazyFrame, local plan inspection and the later execution observation expose the same plan_target. That correlation does not add evidence to the Substrait payload.

Before execution · local plan lane

Inspect authored meaning

inspect_plan(...) reads Prism-backed state without binding a physical source or running a backend. It returns a structured PlanInspection.

SchemaOutput fields + typed shape
Lineage + originValue, control, grouping + authored mappings
Plan + rewritesNodes, applied rules + origin map
RequirementsAdapter needs, diagnostics + unsupported markers

Plan anchorplan_target

During and after · runtime lane

Record one concrete attempt

Session derives the same plan target before lowering, records it on the ExecutionObservation, and gives the concrete run its own distinct attempt_target.

Attempt: execute, collect, or write; terminal status
Runtime: Context targets, backend, and adapter or profile identity
Timing: Start, end + monotonic duration
Outcome: Diagnostics; optional counts, traces + evidence references

Plan anchorplan_target

Substrait remains the logical-plan boundary. It carries the portable Plan / Rel. Neither the PlanInspection, its origin map, nor the correlation key is serialized into that plan. Adapter coverage is checked separately and is not automatically attached by collect_observed.

Quality checks stay explicit and policy-neutral. A quality block declares typed assertions; it does not filter, quarantine, or mutate a relation on its own. A caller evaluates those declarations through Session.observe_quality(...) or observe_quality_pair(...), which return quality observations. Enforcement remains the responsibility of policy, CI, or orchestration code.

A governance checkpoint is evidence, not a hidden policy engine. It records an observation or decision against a semantic target so downstream tooling can explain and verify what happened.

Continue with Plan inspection, Execution observations, quality assertions and observations, or governed evidence.

One concrete execution trace

From query block to observed result

This trace follows one Prism-backed LazyFrame[Order]. It shows the representation at each boundary, the transformation performed there, and the evidence that connects a concrete execution back to the authored Prism plan produced by the query.

Authoringquery { } + LazyFrame
PrismAuthored nodes + canonical view
SubstraitPlan / Rel
SessionPlan + registrations
Adaptermaterialization

Authoring · Surface input

A query over `orders`

from pub::incql import (
    LazyFrame,
    Session,
)
from models import (
    Order,
    OrderSummary,
)

session = Session.default()
orders: LazyFrame[Order] = session.read_csv(
    "orders",
    "orders.csv",
)?

paid_by_region: LazyFrame[OrderSummary] = (
    query {
        FROM orders
        WHERE .status == "paid"
        GROUP BY .region
        SELECT
            .region as region,
            sum(.amount) as total
    }
)

observed = session.collect_observed(
    paid_by_region,
)

Input carrier: LazyFrame[Order]
Logical source: orders
Output intent: LazyFrame[OrderSummary]
Output fields: region, total

Authoring · Desugar + typecheck

The surface becomes ordinary carrier calls

paid_by_region = orders
    .filter(eq(col("status"), "paid"))
    .group_by([col("region")])
    .agg([
        aggregate_as(
            sum(col("amount")),
            "total",
        ),
    ])
    .select([
        with_column_assignment(
            "region",
            col("region"),
        ),
        with_column_assignment(
            "total",
            col("total"),
        ),
    ])

The generated calls follow the normal Incan typechecker. In this CSV-backed trace, the registered orders source supplies the starting logical schema; later references resolve against each stage’s current logical schema during lowering. OrderSummary documents the intended output row model; full field/type compatibility validation against that annotation remains follow-up work.

Prism → Substrait · Logical handoff

See the concrete artifact at every step

The authored Prism graph is immutable. A narrow canonical rewrite fuses grouping and aggregation for lowering while retaining an authored-origin map.

Authored view: 5 nodes
Applied rule: fuse_group_by_aggregate
Rewritten view: 4 nodes
Explainability: Origin map retained

Clause-by-clause transformation from the authored query into the portable logical plan
Query clause and carrier call	Prism state	Substrait relation	Inspectable evidence
`FROM orders`Source carrier: `orders`	Existing carrier root n0`ReadNamedTable("orders")FROM` selects the existing `orders` carrier; it appends no node. The root is unchanged in the rewritten view.	`ReadRelNamedTable(["orders"])`	Read rootLogical source and input-field targets
`WHERE .status == "paid".filter(eq(col("status"), "paid"))`	Authored n1`Filter`Origin n1 after rewrite	`FilterRelequal(status, "paid")`	Control lineage`orders.status` controls which rows survive
`GROUP BY .region.group_by(...).agg(sum(amount))`	Authored n2 + n3`GroupBy` then `Aggregate`Rewritten as one `Aggregate(group, measures)`, origin n3	`AggregateRel`Grouping: `region` Measure: `sum(amount)` as `total`	Grouping + value lineage`region` groups; `amount` contributes to `total`
`SELECT region, total.select(...)`	Authored n4`SelectProject`Origin n4 after rewrite	`ProjectRelRelRoot` names `region`, `total`	Output shapeSchema and output-field targets

How to read the plan: the Plan root wraps ProjectRel, which nests down through AggregateRel, FilterRel, and ReadRel. The tree is displayed root-to-source; rows later move source-to-root. The labels n0…n4 illustrate a fresh cursor and are not stable global IDs.

Session → adapter · Runtime handoff

The portable plan meets concrete runtime state

Substrait carries the logical read, not its concrete TableSource or backend selection. Session maps orders to TableSource { source_kind, uri } and dispatches the selected adapter.

Runtime ownership and the artifact produced by each handoff
Stage and owner	Receives	Performs	Produces or preserves
Session · BindRuntime context	`Plan`Registration for logical table `orders` and selected backend	Validate + dispatchChecks that `orders` is registered, then passes backend registrations and plan to the adapter	Backend dispatch inputsValidated `Plan` + `BackendRegistration[]`. Session separately carries the cursor-derived `plan_target` into the later observation.
Adapter · ExecuteDataFusion	`Plan` + registrations`orders` resolves to a concrete `TableSource`	Backend planning + executionRegisters the source, consumes the Substrait plan, builds a DataFusion logical plan, executes, and collects record batches	`DataFrameMaterializationresolved_columns`, runtime `row_count`, and rendered preview text
Session · Observe returnRuntime evidence	Collected `DataFrame[OrderSummary]` or `SessionError`The result of this exact collect attempt	Normalize public result + evidenceWraps data or error and records attempt status, backend, duration, diagnostics, and row count when available	`ObservedDataFrame[OrderSummary]data`, `observation`, and `error`

Separate diagnostic path session.check_plan_coverage(paid_by_region)

Coverage checking is explicit today. It reports inferred adapter requirements separately; collect_observed does not gate execution on those records or attach them automatically.

Evidence spine

Inspection and execution meet at `plan_target`

Before executioninspect_plan(paid_by_region)

Reports authored and rewritten nodes, the applied rewrite rule, output fields, lineage, adapter requirements, diagnostics, and unsupported-evidence markers—without binding a source or running a backend.

Shared correlation keyplan_targetPrism store + tip derived identity

After collectExecutionObservation

Reports Collect, success or failure, datafusion, runtime duration, diagnostics, and a row count when materialization succeeds.

What changes at each boundary. The query block desugars into carrier calls; those calls append immutable Prism nodes; a canonical view fuses GroupBy and Aggregate; lowering emits one Substrait AggregateRel; Session binds the logical read; and DataFusion performs backend-specific planning and execution.

What stays traceable. Prism retains rewritten-to-authored origin mappings. Local inspection explains that status provides control lineage, region provides grouping lineage, and amount provides value lineage to total. The later observation correlates to that local evidence through the same plan_target; the lineage itself is not runtime telemetry.

Boundary rule: Prism’s origin map and semantic targets remain local IncQL evidence; they are not smuggled into the Substrait payload. Substrait carries relational meaning. Session adds runtime binding and reconnects the concrete attempt to the inspected plan.

See the query-block grammar, inspection surface, and execution API.

Clear seams

Each layer owns one kind of decision

Portability only works when language mechanics, relational meaning, interchange, and execution do not collapse into one another.

Authoring

Language host and semantic package

The compiler hosts the extension point; IncQL supplies the vocabulary and public relational contracts.

Generic language mechanics

Incan compiler

Decides: generic vocabulary hosting, parsing and typechecking generated Incan, language lowering and Rust emission, and language tooling.

Must not decide: IncQL vocabulary semantics, carrier contracts, relational meaning, or execution behavior.

IncQL vocabulary + package

IncQL

Decides: query: and quality: desugaring, public carriers and functions, relational semantics, and evidence contracts.

Must not decide: generic compiler semantics, workflow orchestration, credentials, or engine-specific behavior.

Handoffgeneric vocabulary ASTIncQL desugaringchecked carrier calls

Planning + interchange

Semantic graph and portable representation

Prism owns local planning state; Substrait is the representation exchanged with runtime.

Semantic planning

Prism

Decides: immutable authored graph state, narrow canonical rewrites, rewritten-to-authored origin mapping, and the state from which schema and lineage are derived.

Must not decide: syntax or desugaring, interchange representation, runtime binding, or physical execution.

Portable interchange

Substrait

Carries: portable Plan / Rel, schemas and expressions, extension declarations, and conformance-relevant plan shape.

Must not carry: source bindings, backend selection, local Prism evidence, or physical execution policy.

HandoffPrism rewritten viewloweringSubstrait Plan / Rel

Runtime

Binding context and engine integration

Session owns the attempt; the adapter owns the concrete engine bridge.

Runtime context

Session

Decides: registrations, selected backend, binding validation, dispatch, public result wrapping, execution observations, and explicit coverage evaluation when requested.

Must not decide: semantic rewrites or backend-specific physical planning.

Engine integration

Backend adapter

Decides: concrete source registration, backend bridging and planning, engine calls, collection or writes, and typed backend errors.

Must not decide: IncQL semantics, Session policy, or execution-observation creation.

HandoffPlan + BackendRegistration[]adapter dispatchmaterialization or typed error

What crosses the seams: IncQL vocabulary desugaring produces checked carrier calls. A Prism-backed carrier can derive a rewritten view and lower it to a Substrait Plan or Rel. Session validates registered logical reads, supplies BackendRegistration[], and dispatches the configured adapter, which returns a materialization or typed error.

What does not cross them: Prism’s origin map and inspection artifacts remain local. Session creates the runtime observation and correlates it with local plan evidence through plan_target; the adapter neither receives that evidence nor creates the observation.

The cross-cutting invariant: local plan evidence and runtime observations remain correlatable without turning Substrait or the adapter into an evidence transport.

Read the boundary contracts in Prism RFC 007, Substrait RFC 002, Execution context, and the optimizer-boundary record.

Inside the package

The repository follows the same seams

Most boundaries map to focused modules. Two cross-cuts matter: IncQL-specific syntax lives in vocab_companion/, while shared evidence records connect Prism inspection to Session observations.

Implementation cutaway

Two carrier paths converge at Substrait

Only LazyFrame is Prism-backed today. Concrete DataFrame and DataStream carriers lower directly to the portable boundary.

Boundary	Prism-backed `LazyFrame`	Direct `DataFrame` + `DataStream`
AuthoringIncQL surface	`vocab_companion/`IncQL vocabulary registration and desugaring`src/lib.incn + src/dataset/ + src/functions/`Public exports, carrier types, and relational calls	`src/lib.incn + src/dataset/ + src/functions/`Concrete carriers and relational calls; query vocabulary may produce the same calls
PlanningLocal semantic state	`src/prism/`Authored graph, narrow rewrites, origin mapping, and planning state	No Prism storeThis carrier path converges during Substrait lowering.
ExchangePortable logical plan	`src/substrait/`Schema and expression lowering, relation construction, extensions, plan assembly, and conformance	`src/substrait/`Direct carrier lowering into the same portable `Plan` / `Rel` boundary
RuntimeBind + dispatch	`src/session/ + src/backends.incn`Public Session API, source and sink domain, registration validation, adapter selection, and dispatch	`src/session/ + src/backends.incn`The same runtime context receives the portable plan and registrations
AdapterConcrete engine	`src/session/datafusion_backend.incn`Current DataFusion bridge, execution, collection, writes, and typed backend errors	`src/session/datafusion_backend.incn`The same current adapter consumes the direct path’s portable plan

IncQL repositoryPackage code, tests, documentation, conformance material, and normative RFCs.

Incan repositoryCompiler implementation for syntax, typechecking, lowering, Rust emission, language tooling, and the test runner.

Go deeper

Contributor architecture

Browse the implementation map, build and test path, repository/compiler boundary, and source-level reading order without crowding the conceptual architecture.

Open contributor architecture