THE INCQL BOOK

6. Observe data quality

Evaluate explicit assertions and keep a failed observation separate from execution errors and policy enforcement.

Part III

Decide what happens next

Chapter 1 of 2

Goal

Evaluate one row-count assertion that passes and another that deliberately fails. Read their metrics and statuses without treating the failed check as an automatic exception or row filter.

Assertions describe; Session observes

A quality assertion records the condition a caller wants evaluated. Here, row_count(min_count=Some(1), max_count=Some(3)) describes an inclusive range that the three-row result satisfies, while row_count(min_count=Some(4)) deliberately asks for more rows than the same plan produces. session.observe_quality(...) executes the work needed to evaluate each condition and returns one QualityObservation per assertion.

With this example, you are performing three collections: the baseline collect_observed(...) call, one fresh collection for the passing quality check, and another for the failing check. caller_accepts(...) is local tutorial policy that interprets those records; it is not IncQL enforcement. The helper surface is broader than this row-count example: null_rate(...), unique(...), and group_row_count(...) use the single-relation API, while cross-relation checks use observe_quality_pair(...).

01
Passing observationEvaluate the accepted range in a fresh collection
```
passing = session.observe_quality(
    plan.clone(), # (1)!
    [row_count(min_count=Some(1), max_count=Some(3))], # (2)!
)
```
1. Evaluate this deferred carrier. The first argument is the LazyFrame[Order] whose rows the assertion observes. The clone leaves the original plan available for the deliberately failing check below; it does not reuse previously materialized rows.
2. Supply an assertion list with explicit optional bounds. observe_quality(...) returns one QualityObservation for each item in this list. Some(1) and Some(3) set inclusive minimum and maximum row counts; use None for a bound you do not want to constrain.
Owner
Session quality evaluation

Artifact
One QualityObservation and its execution references

Knowable now
The freshly collected row count satisfies the inclusive range
02
Deliberate failing observationEvaluate a stricter assertion in another collection
```
failing = session.observe_quality( # (1)!
    plan,
    [row_count(min_count=Some(4))],
)
```
1. Start another collection. This call does not inspect the baseline result or the earlier passing observation. Session collects the plan again, evaluates a minimum of four with no maximum bound, and records Failed because the fresh result has three rows.
Owner
Session quality evaluation

Artifact
A separate QualityObservation and execution attempt

Knowable now
Three rows do not satisfy a minimum of four, so the status is Failed
03
Tutorial policyInterpret the evidence in caller-owned code

Owner
caller_accepts(...)

Passing list
true

Failing list
false

QualityObservation × 2 Tutorial policy result

The assertion mode can express intended handling, but IncQL does not silently quarantine rows, stop a pipeline, or approve an output. A Failed observation means the predicate was evaluated and returned false; it is evidence, not an exception or automatic gate. Errored means the relational work needed by the check could not be executed.

Open the complete Chapter 6 checkpoint

"""Chapter 6: observe quality and leave the handling decision with the caller."""

from domain import Order
from pub::incql import (
    LazyFrame,
    QualityObservation,
    QualityObservationStatus,
    Session,
    SessionError,
    inspect_lineage,
    inspect_plan,
    report_session_error,
    row_count,
)


def caller_accepts(observations: list[QualityObservation]) -> bool:
    """Apply this tutorial caller's policy without changing IncQL evidence."""
    for observation in observations:
        if observation.status != QualityObservationStatus.Passed:
            return false
    return true


def evaluate_orders(mut session: Session) -> Result[None, SessionError]:
    """Collect, check coverage, and demonstrate passing and failing quality evidence."""
    orders: LazyFrame[Order] = session.read_csv("tutorial_orders", "orders.csv")?
    plan = orders.limit(3)
    inspection = inspect_plan(plan.clone())
    lineage = inspect_lineage(plan.clone())
    observed = session.collect_observed(plan.clone())

    println(f"plan id: {inspection.plan_id}")
    println(f"lineage edges: {len(lineage.edges)}")
    println(f"backend: {observed.observation.backend_name}")
    match observed.data:
        Some(frame) =>
            println(f"columns: {frame.resolved_columns():?}")
            println(f"rows: {frame.row_count()}")
            println(frame.preview_text())
        None =>
            match observed.error:
                Some(error) => return Err(error)
                None => println("collect_observed returned no data and no typed error")

    coverage = session.check_plan_coverage(plan.clone())
    println(f"coverage records: {len(coverage)}")

    passing = session.observe_quality(plan.clone(), [row_count(min_count=Some(1), max_count=Some(3))])
    failing = session.observe_quality(plan, [row_count(min_count=Some(4))])
    println(f"passing observation: {passing[0].status.value()}")
    println(f"deliberate failing observation: {failing[0].status.value()}")
    println(f"caller accepts passing policy: {caller_accepts(passing)}")
    println(f"caller accepts failing policy: {caller_accepts(failing)}")
    return Ok(None)


def main() -> None:
    mut session = Session.default()
    match evaluate_orders(session):
        Ok(_) => println("Chapter 6: quality produced evidence; the caller made the decision")
        Err(error) => report_session_error("tutorial.chapter_06", error)

Run it:

incan run src/chapter_06.incn

The run reports passed for the inclusive range, failed for the deliberate minimum of four, and the corresponding true and false results from the tutorial's caller_accepts(...) function.

For the assertion helpers, status vocabulary, and cross-relation path, see Quality.

Try it

Print the metrics and execution-observation references carried by both records. Then explain why the final chapter still needs caller-owned decision logic if a failed observation should prevent a write.

Chapter complete

You can distinguish Passed, Failed, and Errored, read the reported metrics, and state clearly that a quality observation is evidence rather than enforcement.