THE INCQL BOOK

4. Collect an observed result

Let the Session bind and execute the plan, then retain evidence about that concrete attempt.

Part II

Explain and execute it

Chapter 2 of 3

Goal

Materialize the review as a DataFrame[Order] and inspect the accompanying ExecutionObservation. Relate that runtime attempt back to the plan inspected in Chapter 3.

This chapter keeps inspection and execution visibly separate. inspect_plan(plan.clone()) and inspect_lineage(plan.clone()) receive clones so the original deferred plan remains available; neither call materializes its rows. Passing that plan to collect_observed(...) crosses the execution boundary: the Session validates and runs the plan through the selected backend, then materializes the result locally.

With this example, you are collecting at most three orders through DataFusion while retaining evidence about that specific attempt. The returned ObservedDataFrame[Order] always carries an observation; its data is Some(DataFrame[Order]) only when materialization succeeds, and its optional error carries the typed failure otherwise. The trace keeps those artifacts separate so a runtime record never has to pretend that row data exists.

01
Chapter changeDescribe and inspect the deferred work
```
plan = orders.limit(3)
inspection = inspect_plan(plan.clone()) # (1)!
lineage = inspect_lineage(plan.clone())
```
1. Inspect without consuming the plan. clone() gives the local inspection functions the same deferred work while leaving plan available for the later execution call. Both inspect_plan(...) and inspect_lineage(...) describe local plan evidence; neither materializes rows.
Owner
Authoring surface

Artifact
Prism-backed deferred plan

Knowable now
Plan identity, authored nodes, schema, and lineage
02
Session dispatchLower, bind, and submit one attempt
```
observed = session.collect_observed(plan) # (1)!
```
1. Cross the execution boundary and keep its receipt. Session validates, lowers, binds, and submits this plan as a concrete collection attempt. The return value is an ObservedDataFrame[Order]: it always carries the attempt's observation, while data and error distinguish successful materialization from a typed failure.
Owner
Session

Artifact
ObservedDataFrame[Order]

Knowable now
A distinct attempt exists and is correlated with the plan
03
DataFusion attemptExecute and return materialisation evidence

Backend
datafusion

Resolved shape
4 columns

Materialised rows
3

ExecutionObservation DataFrame[Order]

The returned data and observation are deliberately separate. On success, data exposes the supported materialisation surface—resolved columns, row count, and preview text. IncQL does not currently expose typed Order iteration here, so the book does not imply that it does. On failure, the attempt can still return typed diagnostics without pretending that materialised data exists.

The observation has its own attempt identity while retaining the plan target. This allows multiple concrete attempts to remain correlated with one logical plan without collapsing the plan and its executions into the same event.

Open the complete Chapter 4 checkpoint

"""Chapter 4: collect through DataFusion and preserve execution evidence."""

from domain import Order
from pub::incql import LazyFrame, Session, SessionError, inspect_lineage, inspect_plan, report_session_error


def collect_orders(mut session: Session) -> Result[None, SessionError]:
    """Inspect, execute, and print the structured materialization surface."""
    orders: LazyFrame[Order] = session.read_csv("tutorial_orders", "orders.csv")?
    plan = orders.limit(3)
    inspection = inspect_plan(plan.clone())
    lineage = inspect_lineage(plan.clone())
    observed = session.collect_observed(plan)

    println(f"plan id: {inspection.plan_id}")
    println(f"lineage edges: {len(lineage.edges)}")
    println(f"backend: {observed.observation.backend_name}")
    match observed.data:
        Some(frame) =>
            println(f"columns: {frame.resolved_columns():?}")
            println(f"rows: {frame.row_count()}")
            println(frame.preview_text())
        None =>
            match observed.error:
                Some(error) => return Err(error)
                None => println("collect_observed returned no data and no typed error")
    return Ok(None)


def main() -> None:
    mut session = Session.default()
    match collect_orders(session):
        Ok(_) => println("Chapter 4: DataFusion collection completed")
        Err(error) => report_session_error("tutorial.chapter_04", error)

Run it from the included tutorial project:

incan run src/chapter_04.incn

Expected evidence

The checkpoint reports the local plan ID and lineage count, names datafusion as the backend, and prints four resolved columns, three materialised rows, and a compact preview. The final line confirms that DataFusion collection completed.

For all observation fields and failure behavior, see Execution context.

Try it

Print the observation's attempt_target next to its plan_target. Explain why the two targets must remain distinct even when this chapter performs only one attempt.

Chapter complete

You can distinguish the Prism plan, the DataFusion execution attempt, and the local DataFrame, and you can show how the attempt remains correlated with the plan.