Skip to content

IncQL v0.1 release notes

Status: Unreleased. IncQL v0.1 is the milestone where the RFC 000–004 series becomes meaningfully usable end-to-end. The current package already exposes dataset carriers, a real Substrait boundary, Prism-backed LazyFrame planning, and Session-backed execution. See the RFC index for normative design.

Features and enhancements

Entries will be filled in as work lands (link RFCs and PRs when applicable).

  • Language: Foundational IncQL syntax and semantics (naming, query schema, layer boundaries).
  • Carriers: DataSet[T] hierarchy including bounded vs unbounded traits and concrete frame/stream types.
  • Plans: Apache Substrait as the logical interchange contract.
  • Authoring: LazyFrame method chains are Prism-backed, and RFC 003 query {} blocks desugar into the same carrier calls before lowering through the current carrier planning paths and Substrait boundary. Query blocks support the brace spelling and expression-position query: spelling, including SELECT aliases, lateral alias reuse, grouped aggregates, SELECT DISTINCT, post-SELECT filters, ordering, limits, inner and left joins, generator clauses, and named window expressions.
  • Aggregates: builder-based col, sum, count, count_expr, count_distinct, count_if, avg, min, and max helpers now lower grouped and global aggregates through Prism, Substrait, and Session execution. count() counts rows, count(expr) counts non-null expression values, count_expr(expr) remains a compatibility spelling, and the first aggregate modifier slice supports DISTINCT plus aggregate-local FILTER where valid.
  • Scalar expressions: RFC 012 unifies filter predicates, computed projection values, grouping keys, and aggregate inputs around one ColumnExpr surface with canonical lit(...) and typed literal helpers.
  • Core scalar functions: RFC 015 adds registry-backed scalar function applications and the first core helper slice for casts, comparisons, boolean logic, null/NaN predicates, arithmetic, conditionals, membership/range predicates, and ordering expressions. Primitive cast targets can use source-level type tokens such as cast(col("amount_text"), float), while explicit string target spellings remain available for compatibility aliases such as int64 and float64. Implemented helpers lower to Substrait IR through registry metadata, built-in Rex shapes, or structural sort-field lowering; DataFusion remains the first execution adapter rather than the semantic boundary.
  • Common scalar functions: RFC 018 adds registry-backed common math, string, text encoding, regex, and date/time helpers, including compatibility aliases such as substr(...), ucase(...), lcase(...), dateadd(...), datediff(...), and safe_cast(...). Helpers lower through registry-owned Substrait metadata, with DataFusion adapter UDFs only where the first backend has no direct API-level equivalent for the IncQL semantic contract.
  • Nested data functions: RFC 020 adds registry-backed scalar helpers for array construction/access, cardinality, containment, overlap, sorting, set-like operations, joining, slicing, reversing, scalar array flattening, map construction/access, map key/value/entry extraction, map key containment, and named struct construction. These helpers lower through Substrait extension metadata without introducing generator semantics, with representative DataFusion-backed Session coverage for composable array projection paths.
  • Generator functions: RFC 021 adds registry-backed generator applications for explode(...), explode_outer(...), posexplode(...), posexplode_outer(...), inline(...), inline_outer(...), portable flatten(...), and stack(...). Generators remain relation-shaping operations applied with generate(...); they preserve input columns, require explicit output aliases, lower through the current Substrait extension-relation gap encoding, and execute through the DataFusion Session adapter with concrete output-column materialization.
  • Window functions: RFC 019 adds window() specs, explicit row/range frame bounds, ranking and distribution helpers (row_number, rank, dense_rank, percent_rank, cume_dist, ntile), offset and value helpers (lag, lead, first_value, last_value, nth_value), and aggregate-over-window placement through with_window_column(...). Portable window helpers require explicit ordering where appropriate, lower through Substrait ConsistentPartitionWindowRel, and execute through the DataFusion session adapter.
  • Format functions: RFC 022 adds scalar payload helpers for deterministic hashes (md5, sha1, sha224, sha256, sha384, sha512, sha2, crc32, and xxhash64), URL parsing/encoding/decoding, JSON validation/path/schema helpers, and CSV row/schema helpers. Format helpers lower through registry-owned Substrait metadata; the DataFusion adapter executes the full helper set with native functions where available and Incan-authored adapter callbacks for non-native helpers.
  • Approximate functions: RFC 023 adds explicit approximate aggregate helpers for approx_count_distinct(...) and approx_percentile(...). They carry approximation policy in registry metadata, lower through IncQL-owned Substrait extension names, and keep DataFusion implementation-name rewrites inside the backend adapter.
  • Typed sketches: RFC 025 adds typed HyperLogLog sketch logical values with SketchLogicalType, SketchExpr, hll_type(...), sketch_col(...), hll_sketch(...), hll_merge(...), hll_estimate(...), hll_serialize(...), and hll_deserialize(...). Sketch metadata remains IncQL-owned through registry and Substrait options; DataFusion reports a backend planning diagnostic because it has no sketch runtime implementation.
  • Typed variants: RFC 026 adds typed semi-structured variant logical values with VariantLogicalType, VariantExpr, variant_type(...), variant_col(...), variant_value(...), parse_variant_json(...), try_parse_variant_json(...), variant_get(...), typeof(...), and kind predicates for null, boolean, integer, float, string, timestamp, array, and object values. Variant parse helpers accept string value-or-column inputs, variant metadata remains IncQL-owned through registry and Substrait options, and DataFusion reports a backend planning diagnostic because it has no variant runtime implementation.
  • Function registry: RFC 014 adds declaration-site registry decorators for the current public helper surface, including stable function references, checked signature projection, lifecycle metadata, behavior categories, alias policy, Substrait mapping categories, and checked API metadata drift validation.
  • Function extension policy: IncQL RFC 024 policy metadata now distinguishes portable core functions, namespaced extension-only functions, opt-in compatibility aliases, engine-specific functions, and rejected compatibility requests without adding an extension plugin system or backend-owned semantics.
  • Projection: builder-based with_column, add, mul, and literal expression helpers now lower derived columns through Prism, Substrait, and Session execution.
  • Substrait internals: RFC 002 helpers are now split into focused owner modules for relation building, plan assembly, inspection, schema registry, extension bookkeeping, and expression lowering instead of one substrait.plan godmodule.
  • Prism: LazyFrame lowering applies safe canonical rewrites (Filter(true) elimination and adjacent Limit/Project/OrderBy collapse) before RFC 002 plan emission.
  • Inspection: RFCs 028–031 now have a first local evidence spine for Prism-backed LazyFrame plans. inspect_plan(...) and inspect_lineage(...) expose semantic targets, output schema, authored and rewritten Prism node records, lineage edges, artifact-family summaries, metadata attachment records, diagnostics shape, and explicit unsupported-evidence markers without executing or backend-binding the plan.
  • Execution observations and adapter coverage: RFC 032 adds observed Session variants for execute, collect, and write so callers can capture structured runtime evidence for success and failure attempts. RFC 033 adds adapter requirement and coverage records, inspection-inferred adapter requirements, Session.check_coverage(requirements) for explicit capability checks, and Session.check_plan_coverage(data) / Session.check_inspection_coverage(inspection) for plan-evidence-driven coverage checks with covered, partially covered, uncovered, and unknown states.
  • Quality observations: RFC 034 adds quality assertion declarations, quality { ... } / quality: assertion-list syntax, and Session.observe_quality(...) / Session.observe_quality_pair(...) so relation, field, group, and explicit cross-relation checks produce structured quality observations. The first helper set covers row-count thresholds, null-rate thresholds, uniqueness, group row-count thresholds, and cross-relation row-count equality without treating failed checks as implicit filters or pipeline gates.
  • Governed evidence: RFC 035 adds governed attribute and policy checkpoint records so callers can attach provenance, confidence, status, authority, lifetime, visibility, diagnostics, and evidence references to semantic targets. Local plan inspection now emits conservative schema-derived governed attributes and planning observation checkpoints without turning IncQL into a policy engine.
  • Execution: Session-oriented read, execute, and write (reference backend per RFC 004), with collect(...) now producing structured DataFrame materialization metadata plus preview text instead of treating rendered text as the canonical contract. Session execution dispatch now routes through a backend adapter boundary over Substrait plans; DataFusion remains the first adapter rather than being encoded directly into Session state.
  • Session API: Session.write(data, target) now accepts typed sink descriptors such as csv_sink(uri) and parquet_sink(uri), while the file-specific write_csv(...) and write_parquet(...) helpers remain as convenience methods.
  • Documentation: Current package behavior is documented under docs/language/, while RFCs remain design records rather than implementation diaries.

Pipe-forward (|>) is specified in RFC 005 but out of scope for v0.1.

Bugfixes

  • Substrait boundary: RFC 002 now exposes explicit join/set operation helpers, preserves ReferenceRel ordinals, and uses a registered incql.io extension URI for the current EXPLODE gap encoding.

Documentation