Announcing Polars 2.1
Polars 2.1 is out. This post gives an overview of the highlights grouped by area. Everything else is in the release notes.
Upgrade with:
`pip install -U polars`
Before you upgrade
Two changes may need your attention:
- `#30071` – Disallow mixed string <=> numeric arithmetic in eager. `pl.Series([1, 2]) + "4"` used to return `["14", "24"]` and now raises, as it already did in lazy
- `#29888` – Drop support for Python 3.10
Out-of-core
In 2.0 we enabled spilling to disk by default, but the most common reason for running out of memory, a high-cardinality group-by, couldn’t spill yet. Now it can.
- `#29911` – Run the streaming group-by out of core, with shuffles
- `#30282` – Evaluate the streaming window node (`over`) out of core
- `#30104` – Add an out-of-memory kill threshold (`POLARS_OOMKILL_THRESHOLD_FRACTION`)
- `#29259` – Add a reverse node to the streaming engine
- `#29792` – Add query-level IO active time and thread count metrics
To see it working, give Polars a small memory budget and turn on the spill metrics:
``` import os
os.environ["POLARS_OOC_MEMORY_BUDGET_MB"] = "200" os.environ["POLARS_OOC_LOG_METRICS"] = "1"
import polars as pl
n = 30_000_000 lf = pl.LazyFrame({ "k": pl.int_range(n, eager=True).shuffle(seed=0), "v": pl.int_range(n, eager=True), })
lf.group_by("k").agg(pl.col("v").sum(), pl.col("v").n_unique()).collect() ```
``` spill_stats(group-by): ... spill(succ=100.0%, n=312) ... prefetch_stats(group-by): file_mb(written=132.8, read=132.8) ... ```
The state for 30 million groups doesn’t fit in 200 MB, so the engine writes part of it to disk and reads it back to finish the query.
SQL
Since 2.0 we treat SQL as a first-class citizen, and this release adds a lot of it.
- `#30188` – Support recursive CTEs (`WITH RECURSIVE`)
- `#30256` – Support `CREATE [OR REPLACE] TEMP TABLE`
- `#29834` – Add `ASOF JOIN ... MATCH_CONDITION (...)`
- `#30131` – Support correlated subqueries in `JOIN ... ON` and correlated scalar subqueries without an aggregate
- `#30141` – Support uncorrelated subqueries in `JOIN ... ON`
- `#29912` – Add `generate_series` and `range`
- `#29967` – Add `REGEXP_REPLACE` and `DATE_TRUNC`
- `#30144` – Support `CURRENT_DATE`, `CURRENT_TIME` and `CURRENT_TIMESTAMP`
- `#29899` – Support time-zone-aware timestamps
- `#29951` – Support `STRUCT` and `MAP` type names
- `#29963` – Support qualified table names
- `#29954` – Add a `read_ndjson` table function with named options
- `#30047` – Accept a list of paths in the `read_*` table functions
Iceberg
You can now manage Iceberg tables end to end from Polars: create them, write to them, query them and drop them.
- `#30113` – Attach Iceberg catalogs to `SQLContext` via `register_catalog`
- `#30260` – Support `CREATE TABLE` with column definitions in attached catalogs
- `#30247` – Support `DROP TABLE [PURGE]` on tables of attached catalogs
- `#29909` – Add `overwrite_filter` and dynamic partition overwrite to `sink_iceberg`
- `#29908` – Add `create_if_not_exists` to `sink_iceberg`
- `#30087` – Add `create_namespace_if_not_exists` to `sink_iceberg`
- `#30091` – Support sinking to Iceberg tables with a bucket partitioned field
- `#30032` – Add `pl.delete_iceberg`
- `#29753` – Add a Rust resolver for `scan_iceberg`
Parquet
- `#29606` – Read encrypted Parquet files
- `#29905` – Write encrypted Parquet files
- `#29987` – Add experimental GeoParquet read and write support
Smaller additions
- `#30169` – Support full joins in `join_where`
- `#30138` – Add `Expr.truncdiv` for division rounding toward zero
- `#30232` – Add conversion to and from masked numpy arrays
- `#30133` – Allow `reinterpret` between `Int128` and `Decimal`
- `#29424` – Add sampling to the KLL approx-quantile sketch
Encryption, `register_catalog`, `delete_iceberg` and masked numpy conversion are still **unstable**, so their API may change in a later release.
Performance
This release has close to 90 performance PRs. These are the ones you are most likely to notice:
Out-of-core
- `#30296` – Make file readers wait for spills while over the memory budget
- `#30294` – Wait for spills while over the out-of-core memory budget
- `#30280` – Return freed memory to the OS right away while out-of-core data is spilled
- `#30237` – Set the default spill compression level to 1
- `#30292` – Lower the default out-of-core prefetch fraction to 0.5
Optimizer
- `#29852` – Track sortedness through joins
- `#29860` – Rewrite `sort().first()` to `min()` and single-key `sort_by().first()` to `min_by()`
- `#30056` – Merge union inputs that scan files of the same kind into one multi-file scan
- `#30007` – Push predicates through order-keeping columns
- `#30123` – Push as-of join key filters to the right input
Parquet
- `#29953` – Use the Parquet string prefilter for literal `str.contains` and SQL `LIKE`
- `#30006` – Skip Parquet row groups with the running top-k threshold
- `#29882` – Filter `Categorical` and `Enum` Parquet columns while decoding
- `#30245` – Speed up writing nested Parquet columns
Cloud and Iceberg
- `#29863` – Reduce requests and raise concurrency for cloud CSV scans
- `#29857` – Reduce requests and raise concurrency for cloud NDJSON scans
- `#29798` – Use Iceberg statistics
- `#29790` – Reuse the Iceberg catalog and cache REST catalog scans
Kernels and construction
- `#29813` – Huge speedup creating wide `Struct` data
- `#29739` – Huge reduction in `from_rows` and `from_dicts` memory usage during nested value conversion
- `#30281` – Aggregate `unique` and `n_unique` in the streaming group-by itself
- `#29849` – Speed up `arg_min` and `arg_max`, especially on data with nulls
Bug fixes
Some of the bug fixes in this release:
Wrong results
- `#29887` – Fix incorrect filter expression rewrites
- `#29793` – Don’t push filters into a cache past a projection
- `#30199` – Correct unsigned Parquet statistics merging
- `#29868` – Write valid Parquet statistics and column index for NaN and nested values
- `#30290` – Compare `Array(Struct)` rows field by field
- `#30060` – Correct rolling window when offset is not the negation of period
- `#30068` – Don’t silently wrap when casting far future dates
- `#30069` – Truncate when casting `Decimal` to integer
- `#30200` – Fix wrong JSON output for `Struct` and `Array` inside list
- `#29965` – Fix wrong results and name clashes for computed SQL `GROUP BY` keys
- `#29823` – Fix wrong results in SQL `REPLACE` and `ARRAY_AGG`, `join_where` on decimals and sorted group-by keys
Hangs and panics
- `#29824` – Fix lost wakeup in the async executor when waking an already-scheduled task
- `#29819` – Fix livelock in `TaskParker` on very low thread counts
- `#29804` – Fix panic in `when/then` that masks a multi-chunk struct column
- `#29850` – Fix row-estimation panic for aggregations over an empty scan
- `#29918` – Raise instead of panicking on out-of-range datetimes in `to_list`
- `#29945` – Raise instead of panicking when a sink is used as input of another operation
Other fixes
- `#29828` – Call `statx` via raw syscall so wheels load on glibc < 2.28
- `#30266` – Respect memory limits of nested cgroups
- `#30065` – Disallow scanning and sinking to the same file in one query
- `#30001` – Fix reading Parquet files with empty row groups
- `#30204` – Raise on Delta tables with column mapping
Last words
Thanks to everyone who contributed code or filed an issue for this release. If something breaks after upgrading, please let us know.