Skip to content

IncQL RFC 016: Core aggregate functions

  • Status: Implemented
  • Created: 2026-04-27
  • Author(s): Danny Meijer (@dannymeijer)
  • Related:
  • IncQL RFC 001 (dataset carriers and aggregation surface)
  • IncQL RFC 003 (query {} aggregate rules)
  • IncQL RFC 012 (scalar expressions and aggregate measures)
  • IncQL RFC 013 (function catalog program)
  • IncQL RFC 014 (function registry and catalog governance)
  • Issue: IncQL #33
  • RFC PR: IncQL #44
  • Written against: Incan v0.2
  • Shipped in: v0.1

Summary

This RFC defines IncQL's core aggregate function set: count, sum, avg, min, and max, including argument forms, input type rules, null behavior, empty-input behavior, result type rules, aliases, and required diagnostics. These aggregates form the minimum portable aggregate surface for dataframe and query-block work.

Motivation

IncQL already exposes sum and argument-free count, but real grouped analysis needs a stable minimum aggregate family. Beam's SQL aggregate surface centers on COUNT, AVG, SUM, MAX, and MIN; DataFusion, Spark, and Snowflake also support these as foundational aggregates. IncQL should define these before broader statistical, collection, approximate, or ordered aggregates.

Without explicit rules, aggregate behavior can drift across authoring surfaces and backends. Null skipping, empty groups, count(*) versus count(expr), and numeric result type rules are observable semantics, not implementation details.

Goals

  • Define canonical core aggregate functions.
  • Distinguish count() / count(*) from count(expr).
  • Define null behavior and empty-input behavior.
  • Define aggregate output naming expectations.
  • Require explicit errors for invalid aggregate positions and invalid argument types.

Non-Goals

  • Defining aggregate DISTINCT, FILTER, or ordered aggregate modifiers.
  • Defining statistical aggregates beyond avg.
  • Defining collection, string, approximate, or sketch aggregates.
  • Defining window function semantics.
  • Defining streaming trigger or watermark semantics.

Guide-level explanation (how authors think about it)

Authors can summarize grouped or whole-relation data with the core aggregate set:

from pub::incql.functions import avg, col, count, max, min, sum

summary = (
    orders
        .group_by([col("customer_id")])
        .agg([
            count(),
            count(col("discount_code")),
            sum(col("amount")),
            avg(col("amount")),
            min(col("created_at")),
            max(col("created_at")),
        ])
)

count() counts rows. count(col("discount_code")) counts non-null values in that expression. Numeric aggregates ignore null input values unless every value is null or the group is empty, in which case result behavior follows the rules below.

Reference-level explanation (precise rules)

IncQL must define canonical aggregate entries for count, sum, avg, min, and max.

count() must count input rows in the current relation or group. count(expr) must count rows where expr evaluates to a non-null value. count_expr(expr) remains available as a compatibility spelling for count(expr), but must lower through the same canonical aggregate mapping. count must return a non-null integer count. For an empty input relation or empty group, count() and expression-count semantics must return zero.

sum(expr) must accept numeric input expressions. It must ignore null input values. If no non-null value exists in the group or relation, sum must return null unless a later RFC defines an explicit defaulting aggregate. The result type must be derived from the input numeric type according to the numeric type policy.

avg(expr) must accept numeric input expressions. It must ignore null input values. If no non-null value exists in the group or relation, avg must return null. Its result type must be capable of representing fractional results unless the input type family explicitly defines otherwise.

min(expr) and max(expr) must accept orderable input expressions. They must ignore null input values. If no non-null value exists in the group or relation, they must return null.

Aggregate outputs must be aggregate measures, not row-level scalar expressions. They may appear only in aggregate-capable positions defined by query blocks, dataframe aggregation methods, or later RFCs.

The registry must record compatibility aliases where useful. mean may be an alias for avg if IncQL accepts dataframe-style naming. Warehouse compatibility names such as count_if remain outside this core RFC unless modeled as aggregate modifiers by IncQL RFC 017.

Design details

Syntax

This RFC requires importable aggregate functions. Query syntax may support SQL spellings such as COUNT(*), but that spelling must map to the same count() aggregate semantics.

Semantics

Core aggregates skip null values except for count(), which counts rows regardless of null values. Expression-count semantics count non-null expression results and are exposed by count(expr).

Aggregate argument expressions must be scalar expressions under IncQL RFC 012. Aggregate arguments must not themselves contain aggregate outputs unless a later RFC defines nested aggregate semantics.

Interaction with other IncQL surfaces

query {} blocks must apply grouped-reference rules consistently with these aggregate definitions. Dataframe .group_by(...).agg(...) calls must produce the same aggregate semantics as equivalent query-block aggregates.

Compatibility / migration

Existing sum and count helpers should be treated as compatibility-compatible forms of the canonical registry entries. count() behavior remains compatible with the current row-count intent. count_expr(expr) remains as a compatibility spelling for expression-count semantics.

Alternatives considered

  • Define all Spark aggregates at once. Rejected because core aggregate semantics should be stabilized before statistics, collection, approximate, and sketch families.
  • Make count accept no arguments forever. Rejected because count(expr) is foundational SQL/dataframe behavior.
  • Return zero from sum on empty input. Rejected for core semantics because SQL-style null-on-no-value behavior better distinguishes "no values" from "values that sum to zero."

Drawbacks

  • Null and empty-input behavior can surprise authors coming from APIs that default missing sums to zero.
  • Result type policy for numeric aggregates is a cross-cutting dependency on scalar numeric types.
  • Supporting both count() and count(expr) makes one helper carry row-count and expression-count semantics, so tests must keep both call shapes covered.

Layers affected

  • IncQL specification — aggregate measures must remain distinct from scalar expressions and consistent with IncQL RFC 012.
  • IncQL library package — public aggregate helpers should support the core aggregate set and argument forms.
  • Incan compiler — query-block aggregate checking must validate grouped and aggregated positions against these rules.
  • Execution / interchange — Prism and Substrait lowering must preserve null skipping, empty-input behavior, and count forms.
  • Documentation — aggregate reference docs should document null and empty-input behavior explicitly.

Implementation log

  • Added registry-backed helpers for sum, count, count_expr, avg, min, and max.
  • Kept aggregate measures distinct from row-level scalar expressions.
  • Added Substrait extension mappings and aggregate relation lowering for all implemented core aggregate helpers.
  • Added DataFusion-backed grouped and global aggregate session tests with materialized output assertions.
  • Preserved existing sum and count() compatibility.

Design Decisions

  • Expression-count spelling: v0.1 exposes canonical count(expr) for expression-count semantics. count_expr(expr) remains a compatibility spelling that returns the same canonical aggregate measure.
  • Count result type: the v0.1 package records count as a non-null aggregate count measure and validates concrete execution through DataFusion-backed session tests. A more precise static numeric return type belongs with the broader IncQL numeric type policy.
  • Average result type: the v0.1 package records avg as a numeric aggregate and relies on the backend/interchange path for concrete materialized numeric representation. Static decimal/floating promotion rules remain tied to the broader numeric type policy rather than this aggregate helper slice.