> ## Documentation Index > Fetch the complete documentation index at: https://private-7c7dfe99-vortex-format.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # v26.8 changelog for Cloud > Changelog for v26.8

Backward Incompatible Changes

Query, function, and syntax changes

* For views whose body is a plain `SELECT` over a single `Distributed` table, the whole outer query is now pushed to the shards (new setting `optimize_trivial_view_pushdown_to_distributed`, enabled by default). This changes observable behaviour for such views: `FINAL` and `SAMPLE` written on the view reference are now propagated to the shard-local table instead of being ignored, and `extremes` is not reported on single-shard clusters. Set `optimize_trivial_view_pushdown_to_distributed = 0` to restore the previous behaviour. [#101791](https://github.com/ClickHouse/ClickHouse/pull/101791) ([simonmichal](https://github.com/simonmichal)). * Lightweight `UPDATE` patch parts now use a new v2 on-disk format sorted by (`sorting_key..., _block_number, _block_offset`) and applied with a new merging algorithm. Peak memory is bounded by the largest equal-sort-key run instead of the full patch, and updates that cross merge boundaries no longer fall back to in-memory Join apply. Old-format patch parts remain readable. During a rolling upgrade from a version before 26.8, keep `patch_parts_version = 'v1'` or use the `compatibility` setting until all replicas are upgraded. [#103182](https://github.com/ClickHouse/ClickHouse/pull/103182) ([CurtizJ](https://github.com/CurtizJ)). * `AggregatingMergeTree` now rejects, at table creation time, schemas where a column is neither part of the sorting key nor an aggregate-state measure (`AggregateFunction`/`SimpleAggregateFunction`). Such columns are silently collapsed to an arbitrary value during background merges, producing wrong results for queries that `GROUP BY` or filter on them. Set `allow_dimensions_outside_sorting_key = 1` to restore the previous behavior. [#108087](https://github.com/ClickHouse/ClickHouse/pull/108087) ([alexey-milovidov](https://github.com/alexey-milovidov)). * ClickHouse now always uses the unified insert deduplication hash for both synchronous and asynchronous inserts; the legacy per-insert deduplication behaviours are removed. The server setting `insert_deduplication_version` is kept as a migration guard: the server refuses to start if it is set to a legacy value (`old_separate_hashes` or `compatible_double_hashes`). To upgrade from a version that used a legacy value, first run a release that supports `compatible_double_hashes` (which writes both the legacy and unified hashes). For replicated tables run it for at least `replicated_deduplication_window_seconds` (one hour by default; the default windows retain the unified hashes of all inserts for that window, which is considered enough for an insert retry loop). For non-replicated tables with `non_replicated_deduplication_window` > 0 that window is count-based rather than time-based, so run `compatible_double_hashes` for at least that many inserts. Then remove the setting (or set it to `new_unified_hash`) before upgrading to this version. [#108361](https://github.com/ClickHouse/ClickHouse/pull/108361) ([CheSema](https://github.com/CheSema)). * The setting `use_legacy_to_time` is now `0` by default, so `toTime` converts values into the `Time` data type instead of converting a date with time to a fixed date. The legacy behavior is still available via the `toTimeWithFixedDate` function or by setting `use_legacy_to_time = 1`. [#108729](https://github.com/ClickHouse/ClickHouse/pull/108729) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Naive Bayes models (used by `naiveBayesClassifier`) are now configured as a dictionary with the `NAIVE_BAYES` layout, built at load time from a table of pre-aggregated per-class n-gram counts, instead of the previous server-side configuration (an XML config file referencing serialized `.bin` model files), which is no longer supported — existing models must be recreated as dictionaries. Three new functions are added alongside `naiveBayesClassifier`: `naiveBayesClassifierWithProb` returns the predicted class together with its probability, `naiveBayesClassifierWithAllProbs` returns every class with its probability, and `naiveBayesNgrams` tokenizes text into n-grams the same way the dictionary does, for building the training data from raw labelled text. Additionally, several performance optimizations have been applied: on a 9.7 MiB code-point trigram language model, it uses \~49× less memory (from 1.84 GiB to 38 MiB), loads \~11× faster, and classifies \~11× faster. [#108773](https://github.com/ClickHouse/ClickHouse/pull/108773) ([nihalzp](https://github.com/nihalzp)). * The default value of the `max_insert_threads` setting has been changed from `1` (no parallelism) to `auto`, which resolves to the number of CPU cores available to the server (reduced under memory pressure via `max_insert_threads_min_free_memory_per_thread`). This parallelizes `INSERT SELECT` by default and can also parallelize the writing side of an eligible plain `INSERT` when the destination write path can safely fan out. It can change the number of parts created and the order of inserted rows. To restore the previous behavior, set `max_insert_threads` to `1`, or set `compatibility` to a version below `26.8`. [#109000](https://github.com/ClickHouse/ClickHouse/pull/109000) ([alexey-milovidov](https://github.com/alexey-milovidov)). [#109006](https://github.com/ClickHouse/ClickHouse/pull/109006) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Reading a CSV bare `Tuple` now takes one field per element in every position, so a `\N` in the field of a top-level element is that element and `input_format_null_as_default` applies to it. Previously a leading `\N` was taken for the whole column being null, so the row was replaced by the column default and the remaining element cells then failed to parse. This is backward incompatible for a row that supplies a single field for a whole bare `Tuple`: it is now short by the remaining elements and is rejected, and for a one-element `Tuple`, where the field count is the same either way, it silently yields the element default instead of the column `DEFAULT`. Set `input_format_csv_deserialize_separate_columns_into_tuple = 0` to read such a field as the whole column again. [#109744](https://github.com/ClickHouse/ClickHouse/pull/109744) ([groeneai](https://github.com/groeneai)). * Added the `VALID FOR ` clause to `CREATE USER` and `ALTER USER` as a shorthand for `VALID UNTIL`. The expiration deadline is computed as the current time plus the given interval at query execution time and stored in the `VALID UNTIL` form. The `valid_until` column of the `system.users` table now has the type `Array(DateTime64(0))` instead of `Array(DateTime)`, so that deadlines beyond the year 2106 are represented exactly; tooling that reads this column should handle the new type. [#110171](https://github.com/ClickHouse/ClickHouse/pull/110171) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `EXPLAIN SYNTAX` now returns the reformatted query as a single `String` record (with embedded newlines) instead of one record per line, so its output is a recoverable single row that is directly usable (for example, `SELECT count() FROM (EXPLAIN SYNTAX ...)` returns `1`). This is controlled by the new `single_record` option, which defaults to `1`; set `single_record = 0` to restore the historical one-record-per-line output. Other `EXPLAIN` kinds (`PLAN`/`PIPELINE`/`AST`) keep their per-line tree output. [#110479](https://github.com/ClickHouse/ClickHouse/pull/110479) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The `hasColumnInTable` function no longer accepts the optional `hostname`, `username`, and `password` arguments for checking a column on an arbitrary remote server; only the `hasColumnInTable(database, table, column)` form remains. The removed remote mode was a security concern: it let any user trigger outbound connections to arbitrary hosts and leaked credentials into query logs, and it was not gated by any privilege. [#110881](https://github.com/ClickHouse/ClickHouse/pull/110881) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The `PostgreSQL` and `MaterializedPostgreSQL` database engines now respect the server's `remote_url_allow_hosts` configuration, like the `PostgreSQL` table engine, the `postgresql` table function, and DDL-created dictionaries already do. The policy is enforced on `CREATE DATABASE` and on a user-issued `ATTACH DATABASE`: if `remote_url_allow_hosts` is configured on your server, creating or attaching a `PostgreSQL` or `MaterializedPostgreSQL` database pointing at a host outside the whitelist is now rejected with `UNACCEPTABLE_URL`. Databases that already exist keep loading at server startup, so an upgrade cannot leave the server unable to boot. [#111135](https://github.com/ClickHouse/ClickHouse/pull/111135) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Removed the Apache Arrow library-based reader and writer for the `Arrow` and `ArrowStream` formats. The native ClickHouse implementation, which has been the default since 26.7, is now the only one. The settings `input_format_arrow_use_native_reader` and `output_format_arrow_use_native_writer` are obsolete: they are still accepted, but have no effect, so a query that set them to `0` to force the Apache Arrow implementation now uses the native one. [#111996](https://github.com/ClickHouse/ClickHouse/pull/111996) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The TLS credentials of a MySQL source (`ssl_ca`, `ssl_cert`, `ssl_key`) can no longer be specified as file paths from SQL — in a named collection created with `CREATE NAMED COLLECTION`, as a query argument, or in a `CREATE DICTIONARY` query — because the server opens those files with its own privileges. Paths remain supported in the server configuration file. Elsewhere, pass the contents of the certificate or key in the new `ssl_ca_pem`, `ssl_cert_pem` and `ssl_key_pem` parameters, which are masked in logs and in `SHOW` queries like passwords are. [#112070](https://github.com/ClickHouse/ClickHouse/pull/112070) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `disable_insertion_and_mutation` now prevents background consumption from `Kafka`, `RabbitMQ`, and `NATS` tables while still allowing direct writes to external storage. Gated `Kafka2`, `NATS`, and `RabbitMQ` tables do not initialize consumers for direct `SELECT`. The recently introduced `message_queue_disable_insertion` setting now requires a server restart. [#113660](https://github.com/ClickHouse/ClickHouse/pull/113660) ([cwurm](https://github.com/cwurm)). * A window `PARTITION BY` or `ORDER BY` over an `AggregateFunction` column is now rejected with `ILLEGAL_COLUMN`, as top-level `ORDER BY` over such a column already was. Previously such a query was accepted by at least one analyzer, and window `PARTITION BY` partitioned differently depending on `max_threads`. The refusal also covers a state nested in `Array`, `Tuple`, `Map`, `Variant` or `SimpleAggregateFunction`. A `SimpleAggregateFunction` over an ordinary type, `QBit`, and `GROUP BY` and `DISTINCT` over a state, are unaffected. [#113878](https://github.com/ClickHouse/ClickHouse/pull/113878) ([groeneai](https://github.com/groeneai)). * Fix access checks for the `SYSTEM ... CACHE ON CLUSTER` commands. They required the `SYSTEM DROP CACHE` privilege group instead of the privilege of the individual command, which both denied holders of a single granular cache privilege and let a holder of the group run `SYSTEM SYNC FILESYSTEM CACHE ON CLUSTER` without holding `SYSTEM SYNC FILESYSTEM CACHE`. [#114042](https://github.com/ClickHouse/ClickHouse/pull/114042) ([groeneai](https://github.com/groeneai)). * The `NATS` table engine accepts credentials inline in the new `nats_credentials` setting (the same payload as a `.creds` file), and no longer accepts `nats_credential_file` from SQL: the path is a reference to a file on the server filesystem, which the server opens with its own privileges, so it can only be specified in a named collection defined in the server configuration file, or as `nats.credential_file` in the server configuration itself. A query may replace such a configured path with inline `nats_credentials`, unless the operator pinned it with ``. Tables created before this restriction keep working after an upgrade. [#114644](https://github.com/ClickHouse/ClickHouse/pull/114644) ([alexey-milovidov](https://github.com/alexey-milovidov)).

Data type and format changes

* Reject the `zip`/`zipx` backup archive format for backups stored on object storage — direct `S3(...)`/`AzureBlobStorage(...)` destinations and `Disk(...)` destinations backed by S3 or Azure — for both `BACKUP` and `RESTORE`. Zip requires seeking to read its central directory, which is very slow over object storage. Use a tar-based format such as `tar.gz` instead. [#101770](https://github.com/ClickHouse/ClickHouse/pull/101770) ([Onyx2406](https://github.com/Onyx2406)). * Extended the supported range of `DateTime64` from `[1900-01-01, 2299-12-31]` to `[0000-01-01, 9999-12-31]`. Values outside the former range are now computed correctly (via cctz) instead of being clamped to the boundary. With precision 8 or 9 the range remains narrower because the ticks are stored as `Int64` (with nanosecond precision the maximum is still `2262-04-11`). Backward compatibility: if you used out-of-range date-times before, but relied on the very specific saturation rules inside the old range, keep in mind that now the results are changing to be more correct. [#107907](https://github.com/ClickHouse/ClickHouse/pull/107907) ([alexey-milovidov](https://github.com/alexey-milovidov)). * An unquoted JSON number for a `DateTime64`/`DateTime` column in `JSONEachRow` and similar formats is now interpreted as a Unix timestamp (seconds since the epoch) with optional sub-second precision, consistent with the `Values` format, `CAST` and `toDateTime64`. Previously a number with a fractional part (e.g. `1703363853.035`) was rejected with `CANNOT_PARSE_INPUT_ASSERTION_FAILED`, and a bare integer (e.g. `1703363853`) was read into the raw scaled value of a `DateTime64`, producing a `1970-...` timestamp. This is a backward incompatible change for unquoted integers fed to `DateTime64` columns; quoted strings and ClickHouse's own (always quoted) JSON output are unaffected. [#108091](https://github.com/ClickHouse/ClickHouse/pull/108091) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The experimental `ALP` codec now performs `Float32` scaling arithmetic in `Float64`, which eliminates exceptions on decimal data and significantly improves compression ratios. [#111627](https://github.com/ClickHouse/ClickHouse/pull/111627) ([rienath](https://github.com/rienath)). * Reject timespan setting (milliseconds/seconds) values that overflow Int64 microseconds [#112757](https://github.com/ClickHouse/ClickHouse/pull/112757) ([azat](https://github.com/azat)). * Asynchronous metrics can now have the `Map` data type, and the per-CPU-core and per-device metrics were converted to single key-value metrics: `OSUserTimeCPU0`, `OSUserTimeCPU1`, ... became a single `OSUserTimeCPU` metric with a map from the CPU core number to the value (and similarly for the other `OS*TimeCPU*` metrics, `CPUFrequencyMHz_*`, `Temperature*`, `EDAC*`, `Block*_*`, `Network(Receive|Send)*_*`, `Disk*_*`, `*BlobsQueueEstimate`, and `AsyncLogging*QueueSize`). `system.asynchronous_metrics` has a new `key_values Map(LowCardinality(String), Float64)` column (the `value` column is `NaN` for such metrics), `system.asynchronous_metric_log` logs them as one row per key using a new `key` column, the Prometheus endpoint exports them with a label (e.g. `ClickHouseAsyncMetrics_BlockReadBytes{device="sda"}`), and the Graphite `MetricsTransmitter` sends them as `..`. If your monitoring reads the old metric names, set `asynchronous_metrics_key_values_mode` to `legacy_names` to retain them or to `both` during migration; apply this server setting with `SYSTEM RELOAD CONFIG` without restarting. [#115333](https://github.com/ClickHouse/ClickHouse/pull/115333) ([alexey-milovidov](https://github.com/alexey-milovidov)). [#116791](https://github.com/ClickHouse/ClickHouse/pull/116791) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Object-storage disks now use native metadata-storage transactions by default, improving the consistency of their metadata updates. [#89658](https://github.com/ClickHouse/ClickHouse/pull/89658) ([Michicosun](https://github.com/Michicosun)).

Settings and configuration changes

* Removed the long-deprecated functions `snowflakeToDateTime`, `snowflakeToDateTime64`, `dateTimeToSnowflake` and `dateTime64ToSnowflake`. They were deprecated back in v24.6 in favor of `snowflakeIDToDateTime`, `snowflakeIDToDateTime64`, `dateTimeToSnowflakeID` and `dateTime64ToSnowflakeID`, which should be used instead. The setting `allow_deprecated_snowflake_conversion_functions` (which used to re-enable them) is now obsolete and has no effect. [#108711](https://github.com/ClickHouse/ClickHouse/pull/108711) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Extended the supported range of `Date32` from `[1900-01-01, 2299-12-31]` to `[0000-01-01, 9999-12-31]`, matching `DateTime64`. Parsing and conversions now accept the extended range instead of silently clamping to the old boundaries. Backward compatibility notes: in the numeric conversion `toDate32(N)`, values in `[120530, 2932896]` are now interpreted as day numbers (dates from `2300-01-01` to `9999-12-31`) instead of Unix timestamps in early 1970, matching the rule that a number that fits into the day-number range is a day number; numbers below the day number of `0000-01-01` and timestamps after `9999-12-31` saturate to the new boundaries. [#111534](https://github.com/ClickHouse/ClickHouse/pull/111534) ([alexey-milovidov](https://github.com/alexey-milovidov)).

Other backward incompatible changes

* `arrayIntersect` and `arraySymmetricDifference` no longer treat a value repeated inside a single argument as if it appeared in several arguments. Queries relying on the previous behavior can now return different results: `arrayIntersect([1], [2], [1, 1])` returns `[]` instead of `[1]`, `arrayIntersect([1, 2], [2], [1, 1, 2])` returns `[2]` instead of `[1, 2]`, and `arraySymmetricDifference([1], [2], [1, 1])` returns `[2, 1]` instead of `[2]`. For `arraySymmetricDifference` two arguments are already enough: `arraySymmetricDifference([1], [2, 2])` returns `[2, 1]` instead of `[1]`. A value is now counted for an argument only when it was present in every argument before it, so the result contains exactly the values present in all of the arguments. As part of the same change, `arrayIntersect` builds its hash table from the smallest argument rather than from all of them, which makes it up to 1.85x faster and use a third less memory when the arguments differ a lot in size. `arrayUnion` is not affected. [#113021](https://github.com/ClickHouse/ClickHouse/pull/113021) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Deprecated the `minmax` column statistics type. [#108680](https://github.com/ClickHouse/ClickHouse/pull/108680) ([hanfei1991](https://github.com/hanfei1991)). * Fix the access check for `SYSTEM PREWARM PRIMARY INDEX CACHE ... ON CLUSTER`, which incorrectly required the `SYSTEM PREWARM MARK CACHE` privilege instead of `SYSTEM PREWARM PRIMARY INDEX CACHE`. [#109198](https://github.com/ClickHouse/ClickHouse/pull/109198) ([Algunenano](https://github.com/Algunenano)). * Fixed `SYSTEM STOP/START CLEANUP` and `SYSTEM STOP/START VIRTUAL PARTS UPDATE ON CLUSTER` requiring `SYSTEM PULLING REPLICATION LOG` instead of `SYSTEM CLEANUP` / `SYSTEM VIRTUAL PARTS UPDATE`. [#110289](https://github.com/ClickHouse/ClickHouse/pull/110289) ([groeneai](https://github.com/groeneai)). * The `/` glob (any amount of directories) now also matches the same directory. Previously, `/` in glob patterns was not handled as a special case, so `data/**/file.txt` would not match `data/file.txt` (zero directory levels). [#97676](https://github.com/ClickHouse/ClickHouse/pull/97676) ([alexey-milovidov](https://github.com/alexey-milovidov)).

New Features

Functions and data types

* Added the `parseQueryToJSON` and `formatQueryFromJSON` functions to serialize and deserialize ASTs as JSON, and an experimental `clickhouse_json` dialect enabled by `enable_json_ast_dialect`. [#100412](https://github.com/ClickHouse/ClickHouse/pull/100412) ([alexey-milovidov](https://github.com/alexey-milovidov)). [#113480](https://github.com/ClickHouse/ClickHouse/pull/113480) ([fm4v](https://github.com/fm4v)). * Added system table `system.stemmers` which shows all available stemmers languages that can be specified for the `stem` function. [#100611](https://github.com/ClickHouse/ClickHouse/pull/100611) ([Ergus](https://github.com/Ergus)). * Added support for the standard SQL `AT TIME ZONE` and `AT LOCAL` postfix operators as syntactic sugar for `toTimeZone`. The expression `expr AT TIME ZONE zone` is now equivalent to `toTimeZone(expr, zone)`, and `expr AT LOCAL` is equivalent to `toTimeZone(expr, timeZone())`. [#106092](https://github.com/ClickHouse/ClickHouse/pull/106092) ([lvzhipin03](https://github.com/lvzhipin03)). * The `URL` table engine and `url` table function now dispatch to the appropriate backend based on the URL scheme: `file://` is served by the `File` engine, `s3://`/`gs://`/`gcs://`/`oss://` by `S3`, `az://`/`azure://`/`abfss://`/`abfs://` by `AzureBlobStorage`, `hdfs://` by `HDFS`, and `http(s)://` by the `URL` engine as before. The `url_base` setting is applied before scheme dispatch. Only the S3 schemes resolved by the default `url_scheme_mappers` are dispatched; other S3-compatible vendor schemes (`cos`, `obs`, ...) are not, and require using the `s3` engine/function directly. [#106093](https://github.com/ClickHouse/ClickHouse/pull/106093) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added the `dotProductTransposed` function (alias `scalarProductTransposed`) that computes the approximate inner product between a `QBit` column and a reference vector, complementing the existing `L2DistanceTransposed` and `cosineDistanceTransposed` functions. [#108100](https://github.com/ClickHouse/ClickHouse/pull/108100) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added an optional stride parameter to the `QBit` data type (`QBit(T, dimension, stride)`) that stores groups of dimensions in separate streams, so a vector search can read only the first dimensions efficiently (e.g. for Matryoshka embeddings). The transposed distance functions accept an optional fourth `used_dims` argument to read a reduced number of dimensions. [#108103](https://github.com/ClickHouse/ClickHouse/pull/108103) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added support for the `Int8` element type in the `QBit` data type, enabling storage and transposed-distance vector search (`L2DistanceTransposed`, `cosineDistanceTransposed`) over quantized 8-bit integer vectors. [#108105](https://github.com/ClickHouse/ClickHouse/pull/108105) ([alexey-milovidov](https://github.com/alexey-milovidov)). * New function `randomHadamardTransform(vector[, seed[, output_dims]])`: a deterministic randomized Hadamard transform of a float vector — an orthogonal, norm-preserving rotation useful for preprocessing embeddings before quantization, and (when truncated) as a Johnson–Lindenstrauss / subsampled-randomized-Hadamard random projection. [#108227](https://github.com/ClickHouse/ClickHouse/pull/108227) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Implement aggregate function `mergedJSONPatch` for RFC 7396-style merge-patch aggregation over JSON values, including state merge support and AggregatingMergeTree usage. Users can aggregate a JSON column using a timestamp or another ordering column as the sort key to compute the latest state. [#108349](https://github.com/ClickHouse/ClickHouse/pull/108349) ([larryluogit](https://github.com/larryluogit)). * Added function `xxHash64Spark`, which computes Spark-compatible `xxHash64` values for `String` and `NULL` inputs using seed `42` and returns `Int64`. [#108436](https://github.com/ClickHouse/ClickHouse/pull/108436) ([lalitium](https://github.com/lalitium)). * Add `digits(n, offset[, length])` function which returns the number in `n` which starts at index `offset(1-indexed)` and has `length` number of digits. If `length` is not provided, the function returns the number from offset till end. [#109012](https://github.com/ClickHouse/ClickHouse/pull/109012) ([1000ms](https://github.com/1000ms)). * Add the `sqr` arithmetic function for calculating the square of a number. [#109061](https://github.com/ClickHouse/ClickHouse/pull/109061) ([lalitium](https://github.com/lalitium)). * Added quantized transposed distance functions `cosineDistanceTransposedQuantized`, `L2DistanceTransposedQuantized` and `dotProductTransposedQuantized` that operate on a `QBit(Int8)` of `quantizeBFloat16ToInt8` Lloyd-Max codes, dequantizing the stored codes on the fly. A floating-point reference vector is the full-precision query, compared at `Float32` precision (a `Float64` query is narrowed); an `Array(Int8)` reference is itself dequantized, for a symmetric quantized-vs-quantized distance. [#109405](https://github.com/ClickHouse/ClickHouse/pull/109405) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added a new function `dictGetRoot` which returns the topmost ancestor (the root) of a key in a hierarchical dictionary. It is a convenient equivalent of `dictGetHierarchy(dict_name, key)[-1]`. [#109459](https://github.com/ClickHouse/ClickHouse/pull/109459) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added the `mergeTreeCodecBlockCounts(database, table)` table function that reports, per (part, column, substream) of a `MergeTree` table, how many compressed blocks use each codec. [#109623](https://github.com/ClickHouse/ClickHouse/pull/109623) ([rienath](https://github.com/rienath)). * Add function `notHas`, the negation of `has` for arrays, maps, and JSON. When the haystack is a constant array, `notHas(constant_array, x)` is rewritten to `x NOT IN constant_array` by `optimize_rewrite_has_to_in` (enabled by default), so it executes via a set lookup and can prune by the primary key index like `NOT IN`. [#109926](https://github.com/ClickHouse/ClickHouse/pull/109926) ([nihalzp](https://github.com/nihalzp)). * Added the `MultiPoint` geo data type, stored as `Array(Point)`, and included it in the `Geometry` type. [#109951](https://github.com/ClickHouse/ClickHouse/pull/109951) ([davidmenggx](https://github.com/davidmenggx)). * Functions `arrayElement` (the `vec[n]` operator) and `arraySlice` now work for the `QBit` data type: `qbit[n]` returns the n-th vector element at full precision, and `arraySlice(qbit, offset, length)` returns a projection to a subset of dimensions. Both read only the bit planes of the stride groups they need, and slices aligned to stride-group boundaries reuse the stored streams without copying. [#109953](https://github.com/ClickHouse/ClickHouse/pull/109953) ([Utkal059](https://github.com/Utkal059)). * Added a `splitByRegexp` tokenizer for text indexes and the `tokens` function. It splits the input into tokens using a regular expression as the separator, for example `tokenizer = splitByRegexp('[^\p{L}\p{N}#+]+')`, which allows preserving tokens containing special characters such as `C++` or `C#` that other tokenizers would break apart. [#110002](https://github.com/ClickHouse/ClickHouse/pull/110002) ([Ergus](https://github.com/Ergus)). * Added functions `geometryIntersectCartesian` and `geometryIntersectSpherical` that return whether two geometries intersect. Unlike `polygonsIntersectCartesian`/`polygonsIntersectSpherical`, they accept any geometry data type (`Point`, `LineString`, `MultiLineString`, `Ring`, `Polygon`, `MultiPolygon`), including the common `Geometry` type, and the two arguments may be of different types. [#110062](https://github.com/ClickHouse/ClickHouse/pull/110062) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The native ORC reader now supports ORC's `uniontype`, mapping it to the ClickHouse `Variant` type (previously such columns were rejected with `Unsupported ORC type`). [#110078](https://github.com/ClickHouse/ClickHouse/pull/110078) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The native ORC output format can now write the ClickHouse `Variant` type, mapping it to an ORC `uniontype` (previously it failed with `ILLEGAL_COLUMN`). [#110085](https://github.com/ClickHouse/ClickHouse/pull/110085) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Add framing formats, selected by the new setting `framing_output_format`: they multiplex different response parts of the query in a single HTTP response stream — chunks of data, totals and extremes, progress packets, profile events, server logs, and exceptions. Implemented framing formats: `None` (default, everything works as before), `EventStream` (HTTP server-sent events), `JSONEachPacketBase64`, and `JSONEachPacketString` (a JSON object per packet with base64-encoded or string data). [#110127](https://github.com/ClickHouse/ClickHouse/pull/110127) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added the `bigquery` table function and the `BigQuery` table engine for reading from and writing to Google BigQuery tables, including public datasets. The table structure is inferred from the BigQuery table schema. Authentication supports an OAuth access token, a service account key in JSON format, and an OAuth client with a refresh token. [#110166](https://github.com/ClickHouse/ClickHouse/pull/110166) ([alexey-milovidov](https://github.com/alexey-milovidov)). * New function `aiRedact` that detects and redacts personally identifiable information (PII) in text using an LLM provider. Specify the categories to redact (e.g. `['email', 'name']`) or pass an empty array to use a default set of PII categories. Matched values are replaced with a token (`[REDACTED]` by default). [#110464](https://github.com/ClickHouse/ClickHouse/pull/110464) ([davidmenggx](https://github.com/davidmenggx)). * Add `aiFilter` function that evaluates a natural-language condition against text with an LLM and returns `UInt8` for use in `WHERE`, `PREWHERE`, and `JOIN ... ON`. [#110594](https://github.com/ClickHouse/ClickHouse/pull/110594) ([ylw510](https://github.com/ylw510)). * Support TLS/SSL connections to PostgreSQL for the `PostgreSQL` table engine, the `postgresql` table function, the `PostgreSQL` and `MaterializedPostgreSQL` database engines, and `PostgreSQL` dictionaries: `sslmode` plus the certificates and the key, given either as literal contents (`sslrootcert_pem`, `sslcert_pem`, `sslkey_pem`; masked like passwords) or as paths (`sslrootcert`, `sslcert`, `sslkey`; accepted only from a named collection defined in the server configuration file). [#110615](https://github.com/ClickHouse/ClickHouse/pull/110615) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The `postgresql` table function and `PostgreSQL` table engine can now be used to connect to another ClickHouse server over the PostgreSQL protocol (when a table name is used; the `query(...)` variant is not supported yet). Added the PostgreSQL-compatibility functions `format_type` and `current_setting`. [#110760](https://github.com/ClickHouse/ClickHouse/pull/110760) ([alexey-milovidov](https://github.com/alexey-milovidov)). * New function `aiSimilarity` that computes the semantic similarity between two texts using an embedding model. Returns a `Nullable(Float32)` in `[-1, 1]` where `1` means the texts are identical, or `NULL` if an operand is `NULL`/empty or its embedding failed. [#110777](https://github.com/ClickHouse/ClickHouse/pull/110777) ([davidmenggx](https://github.com/davidmenggx)). * Added the `Remote` and `RemoteSecure` database engines that provide real-time access to the tables of a database on a remote ClickHouse server, forwarding `SELECT` and `INSERT` queries to it. They are the ClickHouse-to-ClickHouse counterparts of the `MySQL` and `PostgreSQL` database engines. [#110975](https://github.com/ClickHouse/ClickHouse/pull/110975) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Support pipe operators in SQL queries: `FROM t |> WHERE x > 1 |> AGGREGATE count() AS c GROUP BY y |> ORDER BY c DESC |> LIMIT 10`, similar to the pipe syntax of GoogleSQL. Any `SELECT` query can be followed by a chain of `|>` operators, and each operator wraps the query before it into a subquery, so the resulting AST is the same as for the equivalent query with nested subqueries. In a query that starts with the `FROM` clause, the `SELECT` clause is now optional and defaults to `SELECT *`. [#111151](https://github.com/ClickHouse/ClickHouse/pull/111151) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added aggregate function `gini` to calculate the Gini coefficient of finite, non-negative numeric values, including grouped and distributed aggregations. [#112280](https://github.com/ClickHouse/ClickHouse/pull/112280) ([amirreza1307](https://github.com/amirreza1307)). [#114643](https://github.com/ClickHouse/ClickHouse/pull/114643) ([groeneai](https://github.com/groeneai)). * Support animated PNG in the `PNG` output format. A `t` column turns the result of a query into an animation, with the time scale of `t` set by `output_format_image_time_multiplier_seconds` and `output_format_image_time_divisor_seconds`, and `output_format_image_streaming_animation` to write the frames out as the query produces them instead of buffering them in memory. [#112846](https://github.com/ClickHouse/ClickHouse/pull/112846) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added a `finish_time` column to `system.mutations` that records when a mutation was completed. Unfinished mutations and mutations whose completion time is unknown report zero. [#113474](https://github.com/ClickHouse/ClickHouse/pull/113474) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Add a new `bucketed` schema type for `system.metric_log`, which stores all metrics in a single `Map(Enum16(...), Int64)` column using the bucketed `Map` serialization with 128 buckets, plus a per-metric `ALIAS` column for compatibility: the table consists of a few columns instead of thousands, zero values are not stored, and reading a single metric reads only one of the 128 buckets. In addition, a `Map` with `Enum` keys can now be indexed by the string name of the enum value, e.g. `map['name']`. [#115380](https://github.com/ClickHouse/ClickHouse/pull/115380) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added a `chinese` tokenizer for the `tokens` function and `MergeTree` text indexes. It segments Chinese text into words using a dictionary and a Hidden Markov Model (the algorithm follows [jieba](https://github.com/fxsjy/jieba)), with `coarse_grained` (default) and `fine_grained` granularities. [#89945](https://github.com/ClickHouse/ClickHouse/pull/89945) ([amosbird](https://github.com/amosbird)). * Add `groupFormat` aggregate function that formats rows in each group using a specified output format and returns the result as a string. [#93201](https://github.com/ClickHouse/ClickHouse/pull/93201) ([wandersofb](https://github.com/wandersofb)). * Add aggregate function combinator `-Tuple`, which applies the underlying aggregate function to each element of a `Tuple` column independently and returns a `Tuple` of the results, preserving element names: `sumTuple(t)` for `t = (a, b)` returns `(sum(a), sum(b))`. Aggregate functions with several arguments take one tuple per argument, paired by position: `corrTuple((a1, a2), (b1, b2))` returns `(corr(a1, b1), corr(a2, b2))`. Unlike `-ForEach` over arrays, elements may have different types, and per-element result types and names are preserved. [#98190](https://github.com/ClickHouse/ClickHouse/pull/98190) ([RinChanNOWWW](https://github.com/RinChanNOWWW)).

SQL and query features

* Added support for WHERE clauses in projection definitions. Projections with WHERE only materialize rows matching the predicate, and the optimizer can use them (cost-based) when the query's WHERE implies the projection's WHERE. [#102347](https://github.com/ClickHouse/ClickHouse/pull/102347) ([SBALAVIGNESH123](https://github.com/SBALAVIGNESH123)). * Added a way to access tables and databases via URL paths in the HTTP interface (e.g. \`/database/table.format.gz?filter=a>0\`), plus new settings (\`http\_allow\_database\_as\_path\`, \`http\_allow\_table\_as\_file\`, \`http\_allow\_filters\_as\_path\`, \`http\_allow\_filters\_as\_unrecognized\_url\_parameters\`, \`select\`, \`order\`, \`sort\`, \`filter\`, \`page\`, \`compression\`, \`format\`, \`input\_format\`, \`output\_format\`, \`default\_format\`, \`database\`) that compose with one another and with the existing \`query\` URL parameter. [#105249](https://github.com/ClickHouse/ClickHouse/pull/105249) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added the `Remote` and `RemoteSecure` table engines, the persistent counterparts of the `remote` and `remoteSecure` table functions. `CREATE TABLE ... ENGINE = Remote('addresses', db, table, ...)` now works in addition to `CREATE TABLE ... AS remote(...)`. [#106189](https://github.com/ClickHouse/ClickHouse/pull/106189) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added `EXPLAIN ANALYZE` for examining query performance: the query is executed and the actual execution metrics are rendered in the familiar query-plan format. [#106586](https://github.com/ClickHouse/ClickHouse/pull/106586) ([Fgrtue](https://github.com/Fgrtue)). [#110668](https://github.com/ClickHouse/ClickHouse/pull/110668) ([Fgrtue](https://github.com/Fgrtue)). * Added engine-agnostic `SYSTEM STOP`, `SYSTEM START`, `SYSTEM PAUSE`, `SYSTEM CANCEL`, and `SYSTEM REFRESH` commands, and their `... ALL BACKGROUND` server-wide forms, to control the background activity of `Kafka`, `RabbitMQ`, `NATS`, `S3Queue`/`AzureQueue` tables and refreshable materialized views through one unified interface. For refreshable materialized views they alias the existing `SYSTEM ... VIEW` commands. As part of supporting these controls on NATS, JetStream tables now acknowledge messages only after a successful insert (at-least-once, previously messages were auto-acknowledged on delivery and could be lost if the insert failed). A new `nats_wait_for_flush_interval` setting (default `false`, preserving the previous low-latency behaviour) optionally keeps a consumption cycle open for the whole flush interval instead of flushing as soon as the queue drains. A new `nats_commit_on_select` setting makes a direct `SELECT` on a JetStream table consume (acknowledge) the messages it reads. [#107476](https://github.com/ClickHouse/ClickHouse/pull/107476) ([NIKTONIKTO717](https://github.com/NIKTONIKTO717)). * Added support for the `mysql`, `postgresql`, and `sqlite` table functions and table engines to accept a user's query (instead of a table name) and pass it to the external database as is, written either as a subquery `(SELECT ...)` or as `query('SELECT ...')`. The structure of the resulting table is inferred from the query result, and such a table is read-only. [#107740](https://github.com/ClickHouse/ClickHouse/pull/107740) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Introduce the `QueryRunner` table engine. Records inserted into a `QueryRunner` table represent queries that the engine executes. The engine can be used for asynchronous query execution, batch execution of generated queries, directing queries to remote clusters, benchmarks, fuzzing, and testing with shadow traffic. [#107888](https://github.com/ClickHouse/ClickHouse/pull/107888) ([mstetsyuk](https://github.com/mstetsyuk)). * Support the `GROUPS` frame mode for window functions (SQL:2011), e.g. `any(price) OVER (PARTITION BY symbol ORDER BY ts GROUPS BETWEEN CURRENT ROW AND 1 FOLLOWING)`. In a `GROUPS` frame the boundaries count whole peer groups — sets of rows that are equal on the `ORDER BY` key — so `N PRECEDING`/`N FOLLOWING` mean `N` peer groups before/after the current row's peer group, rather than physical rows (`ROWS`) or `ORDER BY` value distances (`RANGE`). [#108653](https://github.com/ClickHouse/ClickHouse/pull/108653) ([nihalzp](https://github.com/nihalzp)). * Plain `CREATE MATERIALIZED VIEW ... POPULATE` is now locally atomic: the view is subscribed to new inserts of the source table and the existing data is snapshotted together under a brief exclusive lock on the source, so rows inserted through the same server concurrently with the population are no longer missed or duplicated (new setting `materialized_views_populate_atomically`, enabled by default). The guarantee covers the local insert path only - inserts arriving on another replica or through a distributed write path are outside the cut - and it requires a source that can provide a pinned snapshot (the `MergeTree` family and `Memory`); other sources, as well as `CREATE OR REPLACE` / `REPLACE` and views created in `Replicated` databases, keep the legacy non-atomic population. `POPULATE` can now also be used together with `TO` to backfill the target table. [#108715](https://github.com/ClickHouse/ClickHouse/pull/108715) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `ALTER USER`, `ALTER ROLE` and `ALTER SETTINGS PROFILE` now accept `SET name = value` as an alias for `MODIFY SETTING name = value`. It changes individual settings in place while keeping the rest, unlike the bare `SETTINGS` clause which replaces the whole settings list. This makes it less likely to accidentally wipe a user's other settings. [#108722](https://github.com/ClickHouse/ClickHouse/pull/108722) ([groeneai](https://github.com/groeneai)). * Support `ALTER TABLE ... MODIFY CONSTRAINT [IF EXISTS] name CHECK expr` to change the expression of an existing constraint in place. [#108768](https://github.com/ClickHouse/ClickHouse/pull/108768) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added the sort-based IEJoin algorithm for joins whose `ON` section has two inequality comparisons (`<`, `<=`, `>`, `>=`) between the joined tables, enabled by adding `ie_join` to the `join_algorithm` setting. Supported kinds are `ALL INNER/LEFT/RIGHT/FULL JOIN` and `SEMI`/`ANTILEFT/RIGHT JOIN`. Previously such queries were executed as a `CROSS JOIN` with a filter (`INNER` only), which is much slower on large tables. [#109920](https://github.com/ClickHouse/ClickHouse/pull/109920) ([vdimir](https://github.com/vdimir)). * Added a new system table, `system.user_query_log`, which shows every user their own query log records without requiring access to the query log table. The original implementation and the idea belong to Yue Ni (@niyue). [#110156](https://github.com/ClickHouse/ClickHouse/pull/110156) ([alexey-milovidov](https://github.com/alexey-milovidov)). * New setting `run_query_in_background`. The server accepts the query, immediately returns an empty result, and runs it to completion regardless of what happens to the connection. The result is discarded. Track the query by its `query_id` in `system.processes` and `system.query_log`. Intended for long queries like `INSERT ... SELECT`, `CREATE TABLE … AS SELECT`, or `CREATE MATERIALIZED VIEW … POPULATE` that must not die with a dropped connection. [#112816](https://github.com/ClickHouse/ClickHouse/pull/112816) ([mstetsyuk](https://github.com/mstetsyuk)). * Support ALTER TABLE ... MODIFY PROJECTION to change projection-level settings (e.g. index\_granularity) of an existing projection without rebuilding it. The new settings apply lazily via merges. [#113343](https://github.com/ClickHouse/ClickHouse/pull/113343) ([diegomestre2](https://github.com/diegomestre2)). * The S3 tables-engine catalog for data lakes now supports `INSERT` as well as querying existing tables. [#113505](https://github.com/ClickHouse/ClickHouse/pull/113505) ([scanhex12](https://github.com/scanhex12)). * The `Iceberg` catalog now supports Snowflake Horizon Catalog with `catalog_type = 'horizon'`, allowing reads and writes to Snowflake-managed `Iceberg` tables. [#114547](https://github.com/ClickHouse/ClickHouse/pull/114547) ([melvynator](https://github.com/melvynator)). * Added the `skip_unavailable_shards_mode` setting (also available as a `Distributed` engine setting) to control which exceptions from a remote shard are silently ignored when `skip_unavailable_shards` is enabled. This provides finer control over query behavior in distributed environments. [#79091](https://github.com/ClickHouse/ClickHouse/pull/79091) ([byte-sourcerer](https://github.com/byte-sourcerer)).

Storage, data lakes, and object storage

* Added support for Puffin file format ([https://iceberg.apache.org/puffin-spec/](https://iceberg.apache.org/puffin-spec/)). Now this format can be used in table functions like `file`, `url`, `s3` and so on.. [#103936](https://github.com/ClickHouse/ClickHouse/pull/103936) ([scanhex12](https://github.com/scanhex12)). * Added "packed" data part storage for `MergeTree` tables, which stores all of a part's files in a single archive (`data.packed`) instead of a file per stream. It is controlled by the `min_bytes_for_full_part_storage`, `min_rows_for_full_part_storage` and `min_level_for_full_part_storage` settings and is disabled by default. Compatibility note: once a table writes packed parts, older server versions cannot read them, so enabling this format prevents downgrading the server. [#108118](https://github.com/ClickHouse/ClickHouse/pull/108118) ([Algunenano](https://github.com/Algunenano)). * Add `nats_credentials` setting to the NATS table engine, allowing users to specify NATS credentials inline as a string (matching the payload of a `.creds` file). [#110733](https://github.com/ClickHouse/ClickHouse/pull/110733) ([addshore](https://github.com/addshore)).

Data formats and ingestion

* Added the `HiveText` output format that writes data in Apache Hive `LazySimpleSerDe` text form. Fields are separated by `\x01`, rows by the new `format_hive_text_rows_delimiter` setting (default `\n`), and nested `Array`, `Map`, and `Tuple` values use Hive's separator list. The output targets Hive's default `LazySimpleSerDe` and is not symmetric with ClickHouse's `HiveText` input for nested values or custom row delimiters. [#107582](https://github.com/ClickHouse/ClickHouse/pull/107582) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added the `GeoJSON` output format for writing one-feature-per-row GeoJSON `FeatureCollection` documents. The new `format_geojson_validate_geometry` setting (enabled by default) validates GeoJSON shapes on reads and writes; the `GeoJSON` input format now infers `id` as `Nullable(String)`, distinguishing absent or `null` IDs from empty strings. [#108065](https://github.com/ClickHouse/ClickHouse/pull/108065) ([nihalzp](https://github.com/nihalzp)).

Settings, access, and observability

* Added a built-in documentation search page, available at the `/docs` path of the HTTP interface, that provides instant search over the `system.documentation` table and renders the reference documentation (with syntax highlighting, math, and cross-links). [#108345](https://github.com/ClickHouse/ClickHouse/pull/108345) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added a japanese tokenizer for text indexes and the tokens/hasAnyTokens/hasAllTokens functions, using the MeCab morphological analyzer. The dictionary is loaded at runtime from a location set in the server configuration and verified against a SHA-256 checksum before use. [#111420](https://github.com/ClickHouse/ClickHouse/pull/111420) ([Ergus](https://github.com/Ergus)). * Added a new filesystem cache setting `system_cache_extensions` that controls which files are stored in the system cache when `use_split_cache` is enabled. [#111575](https://github.com/ClickHouse/ClickHouse/pull/111575) ([kirillgarbar](https://github.com/kirillgarbar)). * Add the `always_fetch_mutated_part` setting, which allows a replica to download mutated parts from other replicas instead of executing mutations locally. [#113020](https://github.com/ClickHouse/ClickHouse/pull/113020) ([rorylshanks](https://github.com/rorylshanks)).

Other new features

* Add support for `WITH TIES` for negative `LIMIT`. [#100930](https://github.com/ClickHouse/ClickHouse/pull/100930) ([nihalzp](https://github.com/nihalzp)). * Add `restore_table_data`, `restore_access_entities`, and `restore_functions` restore settings for granular control over what gets restored. These can override `structure_only` for individual categories — e.g. `structure_only=true, restore_access_entities=true` restores table definitions and access entities without table data. [#102402](https://github.com/ClickHouse/ClickHouse/pull/102402) ([fm4v](https://github.com/fm4v)). * Dictionaries can now be configured for lazy loading individually with `dictionary_lazy_load` in the dictionary definition, overriding the global `dictionaries_lazy_load` server setting. This allows some dictionaries to load lazily while others load eagerly. [#108314](https://github.com/ClickHouse/ClickHouse/pull/108314) ([mstetsyuk](https://github.com/mstetsyuk)). * The array subscript operator supports an array of integers as the index: `arr[indexes]` returns the elements at all of the given positions, equivalently to `arrayMap(i -> arr[i], indexes)`. [#108371](https://github.com/ClickHouse/ClickHouse/pull/108371) ([ucasfl](https://github.com/ucasfl)). * Added functions `geoToUTM`, `UTMToGeo`, `geoToMGRS` and `MGRSToGeo` for converting between WGS84 geographic coordinates and the UTM and MGRS coordinate systems. [#108939](https://github.com/ClickHouse/ClickHouse/pull/108939) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Add `SYSTEM UNLOAD DICTIONARY` and `SYSTEM UNLOAD DICTIONARIES` commands to release dictionary memory without dropping the dictionary definition. Dictionaries will be reloaded lazily on next access. [#109639](https://github.com/ClickHouse/ClickHouse/pull/109639) ([Manerone](https://github.com/Manerone)). * Added a new `icu` tokenizer for text indexes and the `tokens`, `hasAnyTokens`, and `hasAllTokens` functions, providing locale-aware Unicode word segmentation (including dictionary-based segmentation for languages such as Chinese, Japanese, and Thai). [#109940](https://github.com/ClickHouse/ClickHouse/pull/109940) ([Ergus](https://github.com/Ergus)). * Added a new codec `ZXC` — an asymmetric LZ codec with slow compression and very fast decompression, with a compression ratio between `LZ4` and `ZSTD`. [#110620](https://github.com/ClickHouse/ClickHouse/pull/110620) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added the `s3_base` setting so relative URLs in the `s3`, `s3Cluster`, `gcs`, and `oss` table functions and the `S3` table engine can be resolved against an S3 base URL. Added the `URL` database engine, which resolves table names as URLs against an optional base and dispatches them to the appropriate backend. [#111512](https://github.com/ClickHouse/ClickHouse/pull/111512) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Enable wildcard expansion for `url()` and the `URL` engine by listing HTTP index pages (HTML or plaintext directory listings) and extracting matching URLs. Wildcards are matched against the URL path component, with limits on the index page size and on the number of directories read. [#95181](https://github.com/ClickHouse/ClickHouse/pull/95181) ([niyue](https://github.com/niyue)). * Added `AWS_MSK_IAM` as a supported value for `kafka_sasl_mechanism`, enabling ClickHouse to authenticate with Amazon MSK using IAM roles without managing SASL/SCRAM credentials. [#96100](https://github.com/ClickHouse/ClickHouse/pull/96100) ([kalavt](https://github.com/kalavt)). * Added a postprocessor to the text index which transforms the tokens after tokenization. [#98939](https://github.com/ClickHouse/ClickHouse/pull/98939) ([Ergus](https://github.com/Ergus)).

Experimental Features

Query planning and execution

* Added "drivers" for executable user-defined functions. A driver declared in `` can be used in `CREATE FUNCTION name ARGUMENTS (...) RETURNS T ENGINE = DriverName(...) AS '...code...'` to compile or otherwise process a user code snippet at function-creation time and produce a runnable executable UDF. The resulting configuration is stored in ``, and the originating query is persisted as `ATTACH FUNCTION` so the function survives server restarts. A proof-of-concept `c_function_body` driver compiles and runs C function bodies inside sandboxed Docker containers in `executable_pool` mode. [#105131](https://github.com/ClickHouse/ClickHouse/pull/105131) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Prometheus Query API requests to `/api/v1/query` and `/api/v1/query_range` are now recorded in `system.query_log` with `read_rows` and `read_bytes` metrics. [#106611](https://github.com/ClickHouse/ClickHouse/pull/106611) ([JTCunning](https://github.com/JTCunning)). * Support distributed query-plan reads for `SELECT ... FINAL` on MergeTree-family tables when `make_distributed_plan` is enabled. [#108148](https://github.com/ClickHouse/ClickHouse/pull/108148) ([davenger](https://github.com/davenger)). * Add experimental plan-based parallel replicas execution for MergeTree queries. [#108504](https://github.com/ClickHouse/ClickHouse/pull/108504) ([devcrafter](https://github.com/devcrafter)). * Added an experimental table function `eval`, which evaluates a constant expression to a query string and executes the resulting single `SELECT` query. The feature is disabled by default and can be enabled with the setting `allow_experimental_eval_table_function`. Author: Yue Ni [#110132](https://github.com/ClickHouse/ClickHouse/pull/110132) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Plan-based parallel replicas (`parallel_replicas_plan_based`): the local/remote boundary is now chosen by a plan-level optimization over a normal query plan, and queries over views with a `UNION` are parallelized without `parallel_replicas_allow_view_over_mergetree`. Queries that use `IN (subquery)` now fall back to a local plan instead of failing. [#111063](https://github.com/ClickHouse/ClickHouse/pull/111063) ([devcrafter](https://github.com/devcrafter)). [#112443](https://github.com/ClickHouse/ClickHouse/pull/112443) ([devcrafter](https://github.com/devcrafter)). * Harden the experimental AI functions: deny insecure (`http`) endpoints to remote hosts by default (new setting `ai_function_allow_insecure_endpoint`), bound outbound API calls per query by default (`ai_function_max_api_calls_per_query` now defaults to `1000`), and sanitize provider error responses written to the logs. [#111286](https://github.com/ClickHouse/ClickHouse/pull/111286) ([george-larionov](https://github.com/george-larionov)). * Plan-based parallel replicas (`parallel_replicas_plan_based`) now distribute eligible JOINs (`INNER`, `LEFT`, `RIGHT`, including `SEMI`/`ANTI`): the whole join is executed in a distributed plan fragment with one side coordinated across replicas and the other broadcast. [#112268](https://github.com/ClickHouse/ClickHouse/pull/112268) ([devcrafter](https://github.com/devcrafter)). * Report the error that cancelled a distributed query (`make_distributed_plan = 1`) instead of a bare `QUERY_WAS_CANCELLED`. A failing worker task's real error was previously visible only in the server log. [#112590](https://github.com/ClickHouse/ClickHouse/pull/112590) ([davenger](https://github.com/davenger)). * Support distributed execution of query plans with `IN (subquery)` without rewriting it to a `JOIN`. The set is built once on the initiator and its values are shipped with the worker tasks. The forced `rewrite_in_to_join` override is removed; the rewrite remains available as an explicit setting. [#113826](https://github.com/ClickHouse/ClickHouse/pull/113826) ([davenger](https://github.com/davenger)). * A distributed plan query (`make_distributed_plan`) now stops its upstream stages when a `LIMIT` is satisfied, instead of computing the full result and discarding it. [#114806](https://github.com/ClickHouse/ClickHouse/pull/114806) ([davenger](https://github.com/davenger)). * Fix the error `Illegal type Decimal(18, 0) of argument of function toIntervalNanosecond` when a PromQL query uses `offset` inside a range selector, e.g. `rate(m[2m] offset 5m)`. [#114997](https://github.com/ClickHouse/ClickHouse/pull/114997) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Support `GROUPING SETS` aggregation in distributed plans built by the Cascades optimizer (`make_distributed_plan = 1, enable_cascades_optimizer = 1`). [#115426](https://github.com/ClickHouse/ClickHouse/pull/115426) ([davenger](https://github.com/davenger)). * `TimeSeries` tables can now keep a "recent samples" table: when the new engine setting `recent_samples_ttl_seconds` is non-zero, the engine creates a TTL'd copy of the samples table (partitioned by `recent_samples_partition_by`, by day by default) written on every insert, and PromQL queries whose time range fits in the TTL window automatically read from it, making alert/recording rules and short-range dashboards several times faster on large tables. The preference can be disabled with the new query-level setting `time_series_prefer_recent_samples_table`. [#115441](https://github.com/ClickHouse/ClickHouse/pull/115441) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Distributed query plans (`make_distributed_plan`) now stop idle upstream stages promptly after a satisfied `LIMIT`: an idle exchange sink reads `NoMoreDataNeeded` as soon as it arrives, and an idle exchange source notices its closed output and forwards the stop upstream. Previously such stages kept reading and computing data that nobody consumes. [#115690](https://github.com/ClickHouse/ClickHouse/pull/115690) ([davenger](https://github.com/davenger)). * Automatic parallel replicas can now collect runtime statistics and enable parallel replicas for supported queries when `parallel_replicas_plan_based` is enabled. [#115788](https://github.com/ClickHouse/ClickHouse/pull/115788) ([devcrafter](https://github.com/devcrafter)). * Added an experimental Cascades cost-based optimizer for distributed query plans, enabled by `enable_cascades_optimizer = 1` together with `make_distributed_plan = 1`. It chooses between shuffle, broadcast, replicated, and local join strategies, two-phase, shuffle, and local aggregation, two-stage distributed top-N, and parallel and replicated reads by estimated cost, inserting exchange operators as needed. [#86353](https://github.com/ClickHouse/ClickHouse/pull/86353) ([davenger](https://github.com/davenger)).

Storage, data lakes, and indexes

* Add an in-memory SLRU cache for deserialized Paimon metadata files (manifest lists and manifests). When enabled via the `use_paimon_metadata_files_cache` setting, repeated queries against the same Paimon table skip re-downloading and re-parsing metadata from object storage. [#104657](https://github.com/ClickHouse/ClickHouse/pull/104657) ([Binnn-MX](https://github.com/Binnn-MX)). * The experimental `ReaderExecutor` (`use_reader_executor`, off by default) can now hold a remote source connection open and reuse it across sequential reads, reducing the number of object storage requests on scans. [#107735](https://github.com/ClickHouse/ClickHouse/pull/107735) ([CheSema](https://github.com/CheSema)). * Added a read-through filesystem cache to the experimental `ReaderExecutor` read path (`use_reader_executor`, disabled by default). [#110029](https://github.com/ClickHouse/ClickHouse/pull/110029) ([CheSema](https://github.com/CheSema)). * Added an experimental `MergeTree` setting `allow_experimental_adaptive_codec_selection`. When enabled, merges and mutations pick the smallest-output codec per block for columns that use the default codec. [#111834](https://github.com/ClickHouse/ClickHouse/pull/111834) ([rienath](https://github.com/rienath)). * Added a userspace page cache tier to the experimental `ReaderExecutor` read path (`use_reader_executor`, off by default). On a miss the executor now populates the userspace page cache and serves subsequent reads from it, zero-copy. [#114890](https://github.com/ClickHouse/ClickHouse/pull/114890) ([CheSema](https://github.com/CheSema)). * `Iceberg` manifest-file compaction is now available through `OPTIMIZE TABLE ... MANIFEST` to reduce the number of metadata files. It is gated by the `allow_experimental_iceberg_compaction` setting. [#98178](https://github.com/ClickHouse/ClickHouse/pull/98178) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)).

Functions, formats, and AI

* Experimental: serve `SELECT count(*) FROM t WHERE col default(col)` from per-column sparsity statistics in `serialization.json` without any data scan when the predicate exactly partitions the column into defaults and non-defaults. [#105890](https://github.com/ClickHouse/ClickHouse/pull/105890) ([Algunenano](https://github.com/Algunenano)). * Added the experimental `Quantize(...)` vector codec family and an opt-in two-stage approximate vector-search rewrite over quantized companion streams. [#108565](https://github.com/ClickHouse/ClickHouse/pull/108565) ([shankar-iyer](https://github.com/shankar-iyer)). * Revive the experimental `SZ3` error-bounded lossy compression codec for `Float32`/`Float64` (and arrays of them) columns. It requires `allow_experimental_codecs`. [#108788](https://github.com/ClickHouse/ClickHouse/pull/108788) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed reading data compressed with the experimental `SZ3('ALGO_LORENZO_REG', ...)` codec, which previously failed with `CORRUPTED_DATA`, and avoided undefined behavior when compressing non-finite floating-point values (`NaN` and infinities) with the experimental `SZ3` codec. [#110762](https://github.com/ClickHouse/ClickHouse/pull/110762) ([alexey-milovidov](https://github.com/alexey-milovidov)). * PromQL: date/time functions (`hour()`, `minute()`, `month()`, `year()`, `day_of_week()`, `day_of_month()`, `day_of_year()`, `days_in_month()`) can now be called without arguments, defaulting to `vector(time())` as in Prometheus. [#111869](https://github.com/ClickHouse/ClickHouse/pull/111869) ([valerypetrov](https://github.com/valerypetrov)). * PromQL: fixed operations on instant vectors without tags, e.g. `vector(1) + vector(2)`, `topk(1, vector(1))`, `label_replace(vector(1), ...)` and binary operators with `on()`, which failed with a type error. [#111872](https://github.com/ClickHouse/ClickHouse/pull/111872) ([valerypetrov](https://github.com/valerypetrov)). * PromQL: implemented function `deriv`, fixing a bug where its backing aggregate function returned values 1000x too small with millisecond-resolution timestamps. [#112360](https://github.com/ClickHouse/ClickHouse/pull/112360) ([valerypetrov](https://github.com/valerypetrov)). * Apply the read-in-order optimization to `ORDER BY` queries when `parallel_replicas_plan_based` is enabled: the sort is now shipped to the replicas and merged on the initiator, so each replica reads in the sorting key order. Also fixes `Unknown function __topKFilter` for `ORDER BY` on a non-primary-key column with this setting enabled. [#114315](https://github.com/ClickHouse/ClickHouse/pull/114315) ([devcrafter](https://github.com/devcrafter)). * Fixed vector-search reads after reloading a data part (for example after `ATTACH` or a server restart): parts no longer lose their `Quantized` codec subcolumns, so searches read the quantized subcolumns rather than falling back to the full-precision column. [#114763](https://github.com/ClickHouse/ClickHouse/pull/114763) ([shankar-iyer](https://github.com/shankar-iyer)). * The experimental `TimeSeries` table engine no longer applies the `Gorilla` codec to auto-created `value` columns of the samples inner table; they now default to `CODEC(ZSTD(1))`. Explicitly declared columns keep whatever codecs the user wrote. The `timestamp` column default is unchanged (`CODEC(DoubleDelta, ZSTD(1))`). [#112110](https://github.com/ClickHouse/ClickHouse/pull/112110) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). [#114790](https://github.com/ClickHouse/ClickHouse/pull/114790) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Support `ROLLUP`, `CUBE`, `GROUPING SETS` and the `grouping` function in queries executed with `make_distributed_plan = 1`. [#114950](https://github.com/ClickHouse/ClickHouse/pull/114950) ([davenger](https://github.com/davenger)). * Add a Real Doubles (RD) variant and an optional `ALP(AUTO|STD|RD)` argument to the experimental ALP codec. Bare ALP now auto-selects between the existing STD scheme and RD instead of always using STD. [#99654](https://github.com/ClickHouse/ClickHouse/pull/99654) ([nazarii-piontko](https://github.com/nazarii-piontko)).

Other experimental features

* Add static partition-to-shard affinity for StorageKafka2 via kafka\_partition\_shard\_num and kafka\_shard\_count settings, allowing multiple ClickHouse shards to deterministically consume disjoint subsets of Kafka partitions. [#108886](https://github.com/ClickHouse/ClickHouse/pull/108886) ([JingYanchao](https://github.com/JingYanchao)). * You can now authenticate to OneLake using a pre-obtained bearer token via the `onelake_bearer_token` setting, instead of a `onelake_client_id` and `onelake_client_secret`. This avoids sharing a long-lived client secret. The token is not refreshed, so the database must be recreated once it expires. [#109104](https://github.com/ClickHouse/ClickHouse/pull/109104) ([asya-ch](https://github.com/asya-ch)). * Fix `data_uncompressed_bytes` for skip indices packed into `skp_idx.packed` (via `packed_skip_index_max_bytes`): the reported uncompressed size was the compressed size, which also could prevent `distributed_index_analysis` from activating. [#109272](https://github.com/ClickHouse/ClickHouse/pull/109272) ([Algunenano](https://github.com/Algunenano)). * Added Zstandard compression support to the experimental Prometheus remote-write v1 HTTP handler. [#110907](https://github.com/ClickHouse/ClickHouse/pull/110907) ([JTCunning](https://github.com/JTCunning)). * PromQL: implemented functions `clamp`, `clamp_min`, `clamp_max`, and `round`. [#111870](https://github.com/ClickHouse/ClickHouse/pull/111870) ([valerypetrov](https://github.com/valerypetrov)). * PromQL: `quantile` with a quantile level outside \[0, 1] now returns -Inf/+Inf/NaN like Prometheus instead of throwing an error, and `histogram_quantile` now ignores input series whose `le` label is missing or unparsable, like Prometheus. [#111871](https://github.com/ClickHouse/ClickHouse/pull/111871) ([valerypetrov](https://github.com/valerypetrov)). * PromQL: implemented functions `changes()` and `resets()`. [#112352](https://github.com/ClickHouse/ClickHouse/pull/112352) ([valerypetrov](https://github.com/valerypetrov)). * Correctly collect insert stats and queries through the Prometheus remote-write endpoint. [#112825](https://github.com/ClickHouse/ClickHouse/pull/112825) ([JTCunning](https://github.com/JTCunning)). * PromQL: preserve finite values when using modulo with an infinite divisor. [#113746](https://github.com/ClickHouse/ClickHouse/pull/113746) ([fallintoplace](https://github.com/fallintoplace)). * Reject literal line feeds in ordinary quoted PromQL strings. [#114381](https://github.com/ClickHouse/ClickHouse/pull/114381) ([fallintoplace](https://github.com/fallintoplace)). * A very large `distributed_plan_workers_num` no longer makes the server abort with a `std::length_error` when the experimental `make_distributed_plan` and `enable_cascades_optimizer` are enabled. Such a node count is now rejected with `INVALID_SETTING_VALUE`. [#115398](https://github.com/ClickHouse/ClickHouse/pull/115398) ([groeneai](https://github.com/groeneai)). * Implement the `/api/v1/series` endpoint of the Prometheus HTTP API for `TimeSeries` tables: `match[]` series selectors (repeated values are a union), an optional `start`/`end` time range, and an optional `limit` parameter. [#115676](https://github.com/ClickHouse/ClickHouse/pull/115676) ([nikitamikhaylov](https://github.com/nikitamikhaylov)).

Performance Improvements

JOIN and aggregation performance

* Enable the query condition cache for queries that go through TopK dynamic filtering (`ORDER BY ... LIMIT n`). The QCC entry is partitioned by the TopK plan parameters so the same query reuses cached WHERE filter results, while a different `LIMIT`, sort column, or sort direction produces a fresh entry. [#104478](https://github.com/ClickHouse/ClickHouse/pull/104478) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Push LIMIT into aggregation-in-order to enable early termination when GROUP BY key matches ORDER BY key and table sorting key, significantly reducing the number of rows read. [#104859](https://github.com/ClickHouse/ClickHouse/pull/104859) ([thevar1able](https://github.com/thevar1able)). * Enable `allow_aggregate_partitions_independently` by default. When a `GROUP BY` key suits the partition key, ClickHouse can aggregate each partition independently and skip the global merging step. Runtime heuristics automatically skip the optimization when the partition layout would make it unfavorable (too few partitions, too many partitions, or significantly skewed partition sizes). [#105128](https://github.com/ClickHouse/ClickHouse/pull/105128) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Enable `read_in_order_use_virtual_row` by default. When reading in order of the primary key (e.g. `ORDER BY primary_key LIMIT n`) over a table with many parts, only the parts that can actually contribute to the result are read (plus a read-ahead window of at most `max_threads` parts that keeps reading parallel), which significantly reduces peak memory consumption. [#106215](https://github.com/ClickHouse/ClickHouse/pull/106215) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Reduced memory usage and improved the performance of `JOIN`s. The right-hand side of a hash join now uses a compact 8-byte index-based row reference, so the hash-table entries of `ALL` joins are as small as those of `ANY` joins for every key type. Benchmarked on large `parallel_hash` join queries — `INNER` and `ANY` joins of 100–300 million row tables on `UInt64` and `String` keys, with both unique and duplicated keys: the median query became about 12% faster and used about 15% less peak memory, with no query becoming slower. `INNER` joins on `UInt64` keys gained the most (median about 21% faster with 38% less memory). [#107189](https://github.com/ClickHouse/ClickHouse/pull/107189) ([harikrishnan94](https://github.com/harikrishnan94)). * Added the dpsub join-order enumeration algorithm, providing optimal join plans with lower optimization overhead than dpsize and support for non-inner joins. [#107351](https://github.com/ClickHouse/ClickHouse/pull/107351) ([fkastrati](https://github.com/fkastrati)). * Avoid converting single-level aggregations with small memory usage to two-level aggregations by attaching a memory tracker for the aggregation state instead of using whole query memory. [#107490](https://github.com/ClickHouse/ClickHouse/pull/107490) ([m-selmi](https://github.com/m-selmi)). * Avoid building the full query AST during analysis of aggregating queries by computing the aggregation hash-table size-statistics cache key from the query plan instead of the AST. [#107643](https://github.com/ClickHouse/ClickHouse/pull/107643) ([novikd](https://github.com/novikd)). * Enabling switching for `hash` join in the first query run to `parallel_hash` in subsequent runs based on runtime statistics collected in the first run. [#108125](https://github.com/ClickHouse/ClickHouse/pull/108125) ([m-selmi](https://github.com/m-selmi)). * Use hash table size from runtime join statistics to optimize the size of join runtime filters in subsequent query executions. Can be controlled by the new setting `join_runtime_filter_size_from_hash_table_stats`. [#108313](https://github.com/ClickHouse/ClickHouse/pull/108313) ([m-selmi](https://github.com/m-selmi)). * Speed up and reduce the memory usage of `DISTINCT` queries on partitioned `MergeTree` tables by keeping each partition's rows within a single stream, so each stream's preliminary `DISTINCT` works on a disjoint set of keys, instead of the same set of keys appearing in multiple preliminary `DISTINCT`s. This applies when the partition expression is a deterministic function of the `DISTINCT` columns, and is gated by a cost heuristic on partition count and skew (which can be toggled by setting `force_distinct_partitions_independently`, disabled by default). The same per-partition disjointness is propagated through `Expression`, `Filter` and `ARRAY JOIN` steps to the final `DISTINCT`, `LIMIT BY` and `GROUP BY` consumers so they can skip their merge too. Controlled by the new settings `allow_distinct_partitions_independently` (enabled by default) and `max_number_of_partitions_for_independent_distinct` to tune the heuristic. [#108326](https://github.com/ClickHouse/ClickHouse/pull/108326) ([nihalzp](https://github.com/nihalzp)). * Runtime filters are now built on equality keys even when the `ON` clause also contains non-equality predicates, reducing rows entering the hash join for mixed equi + non-equi joins. `LEFT ANTI JOIN` is unaffected. [#108579](https://github.com/ClickHouse/ClickHouse/pull/108579) ([29antonioac](https://github.com/29antonioac)). * Lower the software-prefetch threshold for aggregation and join hash tables from `4 * L2 cache size` to `L2 cache size`, so prefetch is enabled once the hash table no longer fits in L2. This recovers a regression on `GROUP BY` and `JOIN` over medium-sized hash tables (\~1-8 MiB) on platforms whose reported L2 is large (e.g. aarch64 with a 2 MiB L2). [#108655](https://github.com/ClickHouse/ClickHouse/pull/108655) ([groeneai](https://github.com/groeneai)). * Improve window function performance. Window queries over wide tables that include `SELECT *` now use significantly less memory and run faster. Window aggregates, including `quantile*`, `uniq*`, and `groupArray`, are faster with `OVER ()` and `OVER (ORDER BY ... RANGE ...)`. The `cume_dist` function is also faster, and windows using `PARTITION BY` or `ORDER BY` now perform better on columns with many repeated values. [#108688](https://github.com/ClickHouse/ClickHouse/pull/108688) ([nihalzp](https://github.com/nihalzp)). * Aggregation queries without aggregate functions now use `HashSet`-based methods instead of `HashMap` (for supported key types). Speedups up to 1.8x times were observed. [#108862](https://github.com/ClickHouse/ClickHouse/pull/108862) ([nickitat](https://github.com/nickitat)). * Reduce the overhead coming from lock-contention in `parallel_hash` join algorithm. [#108938](https://github.com/ClickHouse/ClickHouse/pull/108938) ([m-selmi](https://github.com/m-selmi)). * Added a new `join_algorithm` value `parallel_full_sorting_merge`: for hash-compatible equality joins, it shards a full sorting merge join by the hash of the join keys into independent per-shard merge joins running on all threads. It keeps the low, streaming memory usage of a merge join while parallelizing it (in a benchmark, \~2.4x faster and \~3.3x less memory than `parallel_hash`). `ASOF` joins fall back to a single `full_sorting_merge`; hash-incompatible key types (floating-point, `JSON`, `Object`, `Dynamic`) skip only the hash-scatter rewrite and can still be sharded at the source by primary-key ranges when `query_plan_join_shard_by_pk_ranges` is enabled. The result is not ordered. [#109005](https://github.com/ClickHouse/ClickHouse/pull/109005) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Column statistics are now materialized on `INSERT` by default when the table's current active size plus the written block size is at most the new `materialize_statistics_on_insert_max_table_size` setting (default 25 GiB); the check is per written block, so a first bulk load into an empty table may still materialize statistics for each written block. This gives the cost-based join optimizer accurate estimates for freshly-loaded dimension tables and avoids pathological join orders (for example, TPC-H Q5 and Q8 no longer time out at scale factor 40), while large established fact tables keep materializing statistics during merges. [#109454](https://github.com/ClickHouse/ClickHouse/pull/109454) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Optimized aggregate function `uniqCombined` in case of aggregation without key. [#109794](https://github.com/ClickHouse/ClickHouse/pull/109794) ([CurtizJ](https://github.com/CurtizJ)). * Optimized aggregate functions `uniqHLL12` and `uniqCombined`. [#109831](https://github.com/ClickHouse/ClickHouse/pull/109831) ([CurtizJ](https://github.com/CurtizJ)). * Parallelize processing of non joined rows in `parallel_hash` join algorithm in two more cases: when there is a residual filter and in the case of small join keys (single-level hash map). [#110008](https://github.com/ClickHouse/ClickHouse/pull/110008) ([m-selmi](https://github.com/m-selmi)). * Push a filter below a window function when the predicate references only the window `PARTITION BY` columns. Such a predicate written after the window (an outer `WHERE` around a windowed subquery, or a conjunct in `QUALIFY`) now reaches storage and enables primary key / partition pruning, skip indexes and projections. For example `SELECT ... row_number() OVER (PARTITION BY key ORDER BY ts) AS rn ... QUALIFY rn = 1 AND key = 'x'` no longer reads and windows the whole table. [#110114](https://github.com/ClickHouse/ClickHouse/pull/110114) ([groeneai](https://github.com/groeneai)). * With `join_use_nulls = 1`, a null-rejecting `WHERE` that references columns from both sides of an outer join (e.g. `WHERE l.k = 42 AND r.k = 42`) now converts the join to `INNER` (or `LEFT`/`RIGHT`) and prunes the primary key, the same as with `join_use_nulls = 0`. Previously the conversion was skipped in this case and both tables were read in full. [#110121](https://github.com/ClickHouse/ClickHouse/pull/110121) ([groeneai](https://github.com/groeneai)). * Optimized functions `uniq`, `uniqExact`, and `uniqHLL12` in aggregation without key. [#110150](https://github.com/ClickHouse/ClickHouse/pull/110150) ([CurtizJ](https://github.com/CurtizJ)). * A `JOIN` whose `ON` condition is constant-false (e.g. `ON 1 = 2`, `ON NULL`, or a condition that folds to false such as `a.t = 'A' AND a.t = 'B'`) no longer reads the non-contributing side. A new query-plan optimization replaces the input sides that cannot contribute a row with an empty source. Controlled by the setting `query_plan_short_circuit_constant_false_join` (default enabled). [#110234](https://github.com/ClickHouse/ClickHouse/pull/110234) ([groeneai](https://github.com/groeneai)). * Parallelize the final merge of single-level aggregation hash tables: the key space is split into disjoint hash partitions that the threads merge independently (setting `enable_parallel_single_level_merge`, enabled by default). Speeds up `GROUP BY` queries whose per-thread hash tables stay below the two-level threshold but whose merge dominated the runtime. [#110395](https://github.com/ClickHouse/ClickHouse/pull/110395) ([nihalzp](https://github.com/nihalzp)). * Single-`String` aggregation keys now use a smaller hash-table representation, improving string-heavy `GROUP BY` workloads. The new `enable_packed_string_keys_in_aggregation` setting is enabled by default; disabling it, or setting `compatibility` below `26.8`, restores the legacy `StringHashTable` path, which can be faster for very low-cardinality keys longer than 11 bytes. [#110573](https://github.com/ClickHouse/ClickHouse/pull/110573) ([harikrishnan94](https://github.com/harikrishnan94)). [#112089](https://github.com/ClickHouse/ClickHouse/pull/112089) ([harikrishnan94](https://github.com/harikrishnan94)). * Improve full sorting merge join performance up to 1.5-3x times on sparse scenarios. [#111200](https://github.com/ClickHouse/ClickHouse/pull/111200) ([vdimir](https://github.com/vdimir)). * Enable `join_runtime_filter_from_fixed_hash_table` when the probe and build join key types differ (for example, a Nullable probe key with a non-Nullable build key). [#111244](https://github.com/ClickHouse/ClickHouse/pull/111244) ([m-selmi](https://github.com/m-selmi)). * Improve performance of `RIGHT` and `FULL` joins with the `parallel_hash` algorithm when the right side contains `NULL` join keys or rows filtered by the `ON` expression by avoiding null maps rebuilding. [#111258](https://github.com/ClickHouse/ClickHouse/pull/111258) ([m-selmi](https://github.com/m-selmi)). * New adaptive algorithm for parallel `GROUP BY` (controlled via setting `enable_adaptive_aggregator`, enabled by default): each thread aggregates into its own hash table until it holds `adaptive_aggregator_freeze_threshold` keys and then freezes it, so frequent keys keep updating the small cache-resident tables with no coordination, while rare keys are routed by their hash into per-bucket backlogs and aggregated exactly once, inside the bucket-parallel merge. [#111459](https://github.com/ClickHouse/ClickHouse/pull/111459) ([nihalzp](https://github.com/nihalzp)). * Sped up row-wise hashing of JSON/`Object` column values stored in shared data (used e.g. by `uniq`/`uniqExact` over a JSON column and by hashed `GROUP BY`/`DISTINCT` keys) by avoiding a per-value `ColumnDynamic` reconstruction. Hash values are unchanged. [#111501](https://github.com/ClickHouse/ClickHouse/pull/111501) ([valerypetrov](https://github.com/valerypetrov)). * Improve performance of RIGHT and FULL hash JOIN with multiple `OR` disjuncts in the ON section, and of RIGHT/FULL JOIN with residual (inequality) conditions. [#111590](https://github.com/ClickHouse/ClickHouse/pull/111590) ([m-selmi](https://github.com/m-selmi)). * Reading in order through a JOIN now also works when a hash join has an automatic spill-to-disk threshold configured (the default since 26.5). When this optimization applies, the join stays in memory so it preserves the left-side order; set `query_plan_read_in_order_through_spilling_join = 0` to restore the previous behavior, where such joins are not used for read-in-order and remain free to spill. As part of this, an `ORDER BY ... LIMIT` over a `LEFT JOIN` read in order no longer reports the whole table as the number of rows it intends to read, so it no longer trips `max_rows_to_read` with `read_overflow_mode = 'throw'` for a read that stops early. [#111973](https://github.com/ClickHouse/ClickHouse/pull/111973) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Re-enables the consecutive keys optimization for `TTL ... GROUP BY` aggregation during merges. The aggregator was constructed with `min_chunk_bytes_for_parallel_parsing` (a byte threshold, default 10 MiB) in the slot that expects `min_hit_rate_to_use_consecutive_keys_optimization` (a hit rate in `[0, 1]`, default 0.5), so the optimization was permanently disabled on that path. Results are not affected. [#112314](https://github.com/ClickHouse/ClickHouse/pull/112314) ([groeneai](https://github.com/groeneai)). * Lazy column replication for `ARRAY JOIN` is no longer lost when a query plan is serialized. With `serialize_query_plan = 1`, the node executing a remote plan fragment used to ignore `enable_lazy_columns_replication` and always materialize every replicated copy, even though the setting defaults to enabled. [#112330](https://github.com/ClickHouse/ClickHouse/pull/112330) ([groeneai](https://github.com/groeneai)). * A `WHERE` equality is now merged into the `JOIN` condition when its operands have different types but a common supertype, such as `Int32` and `Nullable(Int32)`. This allows a CROSS join to be converted to an INNER Join. [#112630](https://github.com/ClickHouse/ClickHouse/pull/112630) ([m-selmi](https://github.com/m-selmi)). * Fixed quadratic reallocation when replicating an `Array` whose elements hold `JSON` values in shared data, for example a constant `Array(JSON)` passed to an aggregate function. Such queries copied a number of bytes that grew quadratically with the row count. [#112897](https://github.com/ClickHouse/ClickHouse/pull/112897) ([groeneai](https://github.com/groeneai)). * Restores part and granule pruning when the bounds of a range predicate on a MergeTree primary or partition key are supplied through a `JOIN ON` clause against a single-row constant side, for example `SELECT count() FROM t JOIN bounds ON t.d >= bounds.lo AND t.d <= bounds.hi`. Such a query read every part since 25.2, and the `query_plan_use_logical_join_step` setting that used to restore pruning became obsolete in 26.5, leaving no way to recover it. [#112957](https://github.com/ClickHouse/ClickHouse/pull/112957) ([groeneai](https://github.com/groeneai)). * Fixed cubic complexity of query planning for a JOIN with a `Merge` table over many tables with different structures. Such queries could previously spend minutes in query planning without responding to `max_execution_time` or `KILL QUERY`. [#113140](https://github.com/ClickHouse/ClickHouse/pull/113140) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a performance regression where an aggregation without `GROUP BY` keys serialized its whole input plan subtree, including the full contents of constant-folded literals, on every execution to compute a hash-table-size cache key it can never use. A query such as `SELECT quantileMerge(arrayJoin(arrayMap(x -> state, range(5000000))))` spent about 63 ms per execution in query-plan optimization and now spends 0.15 ms. [#113333](https://github.com/ClickHouse/ClickHouse/pull/113333) ([groeneai](https://github.com/groeneai)). * Evaluate the PromQL aggregation operators `topk`, `bottomk` and `limitk` with a streaming plan whose selection state is `O(time_steps * k)` instead of collecting all series into a single row with `O(time_steps * series^2)` intermediate memory. Queries over high-cardinality metrics that previously failed with `MEMORY_LIMIT_EXCEEDED` now run in bounded memory, and the evaluation is parallelized instead of single-threaded. [#113656](https://github.com/ClickHouse/ClickHouse/pull/113656) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Replaced the per-bucket hash map inside the `timeSeries*ToGrid` aggregate functions with a flat sorted array of samples: sample ingestion becomes an O(1) append for in-order inputs (the overwhelmingly common case) and the per-bucket copy-and-sort at finalization is gone. [#113681](https://github.com/ClickHouse/ClickHouse/pull/113681) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Evaluate PromQL subexpressions shared by multiple plan steps (the `topk`/`bottomk`/`limitk` operand, the left side of `or`) once instead of twice: the SQL generated from PromQL now materializes shared subqueries. This removes a duplicated scan of the samples table; `topk` over `rate` gets \~2x faster cold, and kubernetes-mixin rule queries improve by 10–29%. [#113772](https://github.com/ClickHouse/ClickHouse/pull/113772) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * The [query condition cache](/operations/query-condition-cache) is now enabled by default for queries that use the `ORDER BY LIMIT n` (TopK) optimization. Set `use_query_condition_cache_for_top_k = 0` to opt out. [#111492](https://github.com/ClickHouse/ClickHouse/pull/111492) [#114539](https://github.com/ClickHouse/ClickHouse/pull/114539) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Parallelize the single-level to two-level conversion of per-thread aggregation states before merging. This fixes a 2-3x regression of short `GROUP BY` queries with heavy aggregate states (e.g. `COUNT(DISTINCT ...)`, ClickBench Q10/Q11) on machines with very many cores, introduced in 26.7 development builds when the aggregation memory accounting was made accurate. [#114691](https://github.com/ClickHouse/ClickHouse/pull/114691) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Speed up window functions on partitioned `MergeTree` tables by reading each partition through its own stream and evaluating the windows per partition, skipping the hash scatter that ordinarily reshuffles every row across threads before the window sort. This applies when the table partition expression is a deterministic function of the window `PARTITION BY` columns and is gated by the same cost heuristic as per-partition `GROUP BY` / `DISTINCT` (enough partitions relative to `max_threads`, no dominant partition). Controlled by the new settings `allow_window_partitions_independently` (enabled by default), `force_window_partitions_independently`, and `max_number_of_partitions_for_independent_window`. [#114783](https://github.com/ClickHouse/ClickHouse/pull/114783) ([nihalzp](https://github.com/nihalzp)). * Speed up the `timeSeries*ToGrid` aggregate functions: per-sample bucket math now avoids Int128 division, and consecutive samples in the same bucket skip the hash-map lookup. [#114889](https://github.com/ClickHouse/ClickHouse/pull/114889) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Process samples of the `timeSeries*ToGrid` aggregate functions in runs of consecutive samples falling into one grid bucket, speeding up the sample-ingestion stage of `timeSeriesRateToGrid` and related functions by 1.2-2x on sorted time series data. [#115041](https://github.com/ClickHouse/ClickHouse/pull/115041) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * An aggregation without `GROUP BY` keys no longer fans its single-row output out to `max_threads` streams, so queries whose plan continues after such an aggregation get a much smaller pipeline (fewer processors to create, schedule and profile) without losing any parallelism. [#115170](https://github.com/ClickHouse/ClickHouse/pull/115170) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Faster evaluation of PromQL instant queries: the `timeSeries*ToGrid` aggregate functions use the vectorized sample-classification path for single-point grids, and `timeSeriesIdToGroup` fills its result column directly. [#115267](https://github.com/ClickHouse/ClickHouse/pull/115267) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * A `JOIN` whose `ON` section has several `OR`-ed equalities no longer materializes the right-hand key columns that the query does not select. [#115465](https://github.com/ClickHouse/ClickHouse/pull/115465) ([nickitat](https://github.com/nickitat)). * Speeds up building the hash table of an `ANY`, `SEMI` or `ANTI` join when the right-hand side repeats the same join keys many times: the join no longer re-reads the `join_any_take_last_row` setting from memory for every right row. [#115820](https://github.com/ClickHouse/ClickHouse/pull/115820) ([nickitat](https://github.com/nickitat)). * Speeds up `GROUP BY` without aggregate functions over `String`, `FixedString` and `LowCardinality` keys: the query no longer rewrites an unused placeholder into the hash table for every row, only for the rows that add a new key. [#115834](https://github.com/ClickHouse/ClickHouse/pull/115834) ([nickitat](https://github.com/nickitat)). * Reduced memory usage and query-plan optimization time for keyed aggregations over large constant-folded literals. The aggregation hash-table-statistics cache key is now hashed as it is serialized instead of being accumulated in memory first, so a plan carrying a large literal no longer holds a copy of it during optimization. For an 80 MB literal, peak memory drops by about 250 MB and optimization time by about 2.5x. The computed key is unchanged. [#115847](https://github.com/ClickHouse/ClickHouse/pull/115847) ([groeneai](https://github.com/groeneai)). * Optimize `GROUP BY ... ORDER BY ... LIMIT` and `GROUP BY ... LIMIT` queries by maintaining a bounded heap during aggregation to prune groups that cannot appear in the result, significantly reducing memory usage and execution time for high-cardinality grouping. Controlled by the `enable_group_by_top_k_optimization` setting. Based on the initial implementation by Dmitriy Terenichev [#96630](https://github.com/ClickHouse/ClickHouse/pull/96630) ([thevar1able](https://github.com/thevar1able)).

Query optimization and execution

* Reading in-order with Parallel Replicas now uses the same logic of splitting the table into `max_threads` parts as the local reading for better parallelism. [#101434](https://github.com/ClickHouse/ClickHouse/pull/101434) ([nickitat](https://github.com/nickitat)). * Adds GeoParquet spatial pruning at row group and page levels to the Parquet reader. Row group pruning skips entire row groups whose bounding box doesn't overlap the query geometry. Page-level pruning uses the `covering.bbox` column index to skip irrelevant pages within row groups. Also generalizes spatial predicate pushdown through `IFunctionBase::isSpatialPredicate()` and adds a `GeoFilter` that evaluates spatial predicates during Parquet row reading. [#104435](https://github.com/ClickHouse/ClickHouse/pull/104435) ([bacek](https://github.com/bacek)). * Added the `use_constant_folding_in_index_analysis` setting (disabled by default). When enabled, `MergeTree` primary-key, `MinMax`, and skip-index analysis fold partition-level constants into the filter predicate separately for each part, improving pruning for filters whose branches depend on partition values, e.g. `(a = 1 AND b >= 1) OR (a = 2 AND b > 10)` with `PARTITION BY a`. [#104582](https://github.com/ClickHouse/ClickHouse/pull/104582) ([Michicosun](https://github.com/Michicosun)). * Apply exact row-positioning for vector search queries with `vector_search_with_rescoring` before exact distance evaluation in the query pipeline. [#105591](https://github.com/ClickHouse/ClickHouse/pull/105591) ([skuznetsov](https://github.com/skuznetsov)). * Added a query plan optimization that pushes volume-reducing functions (`length`, `lengthUTF8`, `empty`, `notEmpty`) below the `Sorting` and `Filter` steps, replacing the wide `String` / `FixedString` argument with the fixed-size result, so it is neither buffered by a sort nor copied by a filter. Controlled by the new setting `query_plan_push_down_volume_reducing_functions`, enabled by default. [#106199](https://github.com/ClickHouse/ClickHouse/pull/106199) ([fastio](https://github.com/fastio)). * Speed up filesystem cache loading on startup by encoding each cache file's size in its name (`_`), avoiding a `stat` per file. Loading of cache files written by older versions remains supported. [#107415](https://github.com/ClickHouse/ClickHouse/pull/107415) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Improved performance of text-search queries combined with a primary key filter. [#108114](https://github.com/ClickHouse/ClickHouse/pull/108114) ([CurtizJ](https://github.com/CurtizJ)). * Improve performance of the `pointInPolygon` function with a constant polygon. The preprocessed-polygon cache is now keyed on the raw constant arguments, so a constant polygon is parsed only once (on a cache miss) instead of being re-parsed on every input block. [#108184](https://github.com/ClickHouse/ClickHouse/pull/108184) ([groeneai](https://github.com/groeneai)). * Avoid maintaining `system.predicate_statistics_log` selectivity counters on the MergeTree read path when the feature is disabled (`predicate_statistics_sample_rate = 0`, the default). This removes a per-granule atomic update and an `O(rows)` filter popcount from every read, recovering a performance regression that was most visible on aarch64. [#108190](https://github.com/ClickHouse/ClickHouse/pull/108190) ([groeneai](https://github.com/groeneai)). * Add the `merge_use_batch_sorting_queueMergeTree` setting to optionally use the batch sorting queue for ordinary `MergeTree` merges, reducing CPU and wall-clock time for merge workloads where sorted merging is a significant cost. [#108468](https://github.com/ClickHouse/ClickHouse/pull/108468) ([rorylshanks](https://github.com/rorylshanks)). * Speed up query planning under the analyzer for expressions that reference the same `WITH` alias or repeat the same subexpression many times (for example deeply nested `if`/`multiIf` chains), by not rebuilding shared subexpressions when constructing the actions DAG. [#108523](https://github.com/ClickHouse/ClickHouse/pull/108523) ([novikd](https://github.com/novikd)). * Sped up query analysis for queries with many or very large `AND`-chains of comparisons: the `optimize_and_compare_chain` optimization is now bounded by a work budget (new setting `optimize_and_compare_chain_max_hash_work`) instead of hashing a large fraction of the query tree, and lambda resolution no longer recomputes lambda-body hashes for its recursion guard. [#108757](https://github.com/ClickHouse/ClickHouse/pull/108757) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Filter-only postprocessors (`if(..., '', token)`) now build the granule from raw tokens and drop the empties at build time, avoiding the general path's per-occurrence materialization and `ActionsDAG` run. [#109049](https://github.com/ClickHouse/ClickHouse/pull/109049) ([ahmadov](https://github.com/ahmadov)). * Remove a single-threaded bottleneck in `ShuffleSendStep` of the distributed query plan: each upstream stream now scatters its rows by destination bucket independently instead of funneling the whole fragment through one `ScatterByPartitionTransform`. [#109206](https://github.com/ClickHouse/ClickHouse/pull/109206) ([davenger](https://github.com/davenger)). * Reduce CPU overhead of expression evaluation for queries with a large number of columns (e.g. vector search over a `QBit` column with a small stride, where each vector expands into hundreds of bit-plane sub-columns that are all fed into a single `*DistanceTransposed` call). [#109380](https://github.com/ClickHouse/ClickHouse/pull/109380) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Push a filter below `LIMIT BY` when the predicate references only the `LIMIT BY` key columns and the clause keeps at least one row per non-empty group (`LIMIT n BY` with `n >= 1` and `OFFSET 0`). This lets such predicates (written as an outer `WHERE` around a `LIMIT BY` subquery) reach storage for primary key pruning, skip indexes and projections, instead of reading the whole table and applying the per-group top-N first, without changing exception semantics for `OFFSET`, `LIMIT 0 BY`, or negative `LIMIT BY` forms. [#110116](https://github.com/ClickHouse/ClickHouse/pull/110116) ([groeneai](https://github.com/groeneai)). * Avoid a heap allocation per hyperrectangle check in primary key index analysis, speeding up mark filtering by 5-26% depending on the query shape. [#110153](https://github.com/ClickHouse/ClickHouse/pull/110153) ([raimannma](https://github.com/raimannma)). * Parquet filter push-down (row-group pruning and column indexes) now supports conversions between `DateTime64` variants, `Decimal` variants, and `Float32`/`Float64`. [#110457](https://github.com/ClickHouse/ClickHouse/pull/110457) ([al13n321](https://github.com/al13n321)). * Optimized building of text and bloom filter based skip indexes with sparseGrams tokenizer [#110541](https://github.com/ClickHouse/ClickHouse/pull/110541) ([UnamedRus](https://github.com/UnamedRus)). * Combine I/O cost with selectivity in PREWHERE condition ordering. It fixes performance regression in PREWHERE execution in some cases introduced after [https://github.com/ClickHouse/ClickHouse/pull/101275](https://github.com/ClickHouse/ClickHouse/pull/101275). Part of [https://github.com/ClickHouse/ClickHouse/issues/110462](https://github.com/ClickHouse/ClickHouse/issues/110462) [#110695](https://github.com/ClickHouse/ClickHouse/pull/110695) ([Avogar](https://github.com/Avogar)). * Reduce hashing of tokens defined in the postprocessor `IN`/`NOT IN` filter. [#111690](https://github.com/ClickHouse/ClickHouse/pull/111690) ([ahmadov](https://github.com/ahmadov)). * Functions mapContainsKeyLike, mapContainsValueLike, mapExtractKeyLike and mapExtractValueLike with a non-constant pattern use less memory and time when enable\_lazy\_columns\_replication is enabled. The pattern is no longer eagerly copied per map entry at capture time; the copy is deferred to the execution of the internal lambda and is freed sooner, so one fewer expanded copy of the pattern exists at peak. [#111749](https://github.com/ClickHouse/ClickHouse/pull/111749) ([diegomestre2](https://github.com/diegomestre2)). * Enabled by default (and promoted to beta) two on-disk features: 1) `packed_skip_index_max_bytes = '1M'`: skip-index substreams up to 1 MiB are now written into a single `skp_idx.packed` archive per part, reducing object count and write requests on object storage. Compatibility: only newly written parts are affected and they stay fully readable on older versions; versions before 26.6 simply do not use the packed indices for pruning (they are treated as not materialized). Set `packed_skip_index_max_bytes = 0` to keep the standalone per-file layout. 2) `compute_exact_num_defaults_for_sparse_columns` plus `optimize_trivial_count_with_sparsity_filter`: exact per-column `num_defaults` counters are persisted and used to answer `SELECT count() FROM t WHERE ` without a data scan when the predicate splits rows into defaults vs non-defaults. Compatibility: the `exact_num_defaults` flag in `serialization.json` is ignored by older versions, so parts remain fully readable after a downgrade; the count rewrite only applies to parts that carry exact counters, so results stay correct on tables with mixed parts. [#111816](https://github.com/ClickHouse/ClickHouse/pull/111816) ([Algunenano](https://github.com/Algunenano)). * The transitive-conditions optimization now uses its additional filter only for index analysis when it cannot apply the whole filter. See the `optimize_and_compare_chain` setting. [#112076](https://github.com/ClickHouse/ClickHouse/pull/112076) ([yariks5s](https://github.com/yariks5s)). * The Prometheus query API (`/api/v1/query`, `/api/v1/query_range`) of the experimental TimeSeries engine now executes queries in parallel instead of on a single thread. [#112108](https://github.com/ClickHouse/ClickHouse/pull/112108) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * `randomHadamardTransform` of a constant vector is now evaluated once instead of once per row. This speeds up vector search queries that rotate the query vector with `randomHadamardTransform` before comparing it against a `QBit` column. [#112921](https://github.com/ClickHouse/ClickHouse/pull/112921) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Restores set sharing between the part tasks of one mutation filtering on `IN (subquery)` over a primary key column. `enable_sharing_sets_for_mutations` is on by default, but the set built for primary key analysis stopped participating in the shared cache, so a table with N parts materialized the same set N times. [#112941](https://github.com/ClickHouse/ClickHouse/pull/112941) ([groeneai](https://github.com/groeneai)). * In distributed query plans (`make_distributed_plan`), `ORDER BY ... LIMIT k` now uses lazy materialization on each replica: wide columns are read only for that replica's local top-k rows instead of for every candidate row. [#113111](https://github.com/ClickHouse/ClickHouse/pull/113111) ([shankar-iyer](https://github.com/shankar-iyer)). * Speed up query planning when `uniq_v2` column statistics are used. The estimated number of distinct values is now cached instead of being recomputed on every request, which query planning issues many times per query. [#113357](https://github.com/ClickHouse/ClickHouse/pull/113357) ([groeneai](https://github.com/groeneai)). * Filter pushdown now works for `Tuple` subcolumns in Parquet and ORC files. A predicate such as `WHERE tup.1 = 555555` over `file()`, `s3()` or `url()` now prunes row groups and row index strides using the tuple element's own statistics instead of reading the whole file. [#113383](https://github.com/ClickHouse/ClickHouse/pull/113383) ([groeneai](https://github.com/groeneai)). * Improve PromQL query performance over the TimeSeries engine: the internal selector SQL now reads the samples-table primary-key columns directly (without no-op casts) and checks the timestamp range before the series-id set, which makes cold selector-heavy queries 30-44% faster. [#113768](https://github.com/ClickHouse/ClickHouse/pull/113768) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * PromQL selectors that match all series of one metric now filter the samples table of a `TimeSeries` table with a continuous primary-key range on `id` during index analysis instead of a large `id IN ` condition, when the id layout is a two-component tuple with the canonical id generator. Removes the dominant single-threaded index-analysis cost of selector-heavy PromQL queries: up to −45% cold latency on dashboard and rule query shapes, −11% cold geomean over the full suite on a 62-billion-sample table. [#114131](https://github.com/ClickHouse/ClickHouse/pull/114131) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * The Prometheus HTTP API endpoints `/api/v1/query` and `/api/v1/query_range` now evaluate a PromQL subquery shared by several plan steps once, as the `prometheusQuery` and `prometheusQueryRange` table functions already did. Previously a request made with `enable_analyzer = 0`, or by a user whose profile sets it, rescanned the shared subquery once per referencing step. [#114261](https://github.com/ClickHouse/ClickHouse/pull/114261) ([groeneai](https://github.com/groeneai)). * Fixed a regression in 26.7: for a `MergeTree` table ordered by `toUnixTimestamp` (or another integer conversion) of a `DateTime64` column, a plain range filter on that column no longer used the primary key and read all granules of the matched parts. Additionally, a filter like `toInt64(ts) >= c` over a table ordered by the raw `DateTime64` column now uses the primary key. [#114413](https://github.com/ClickHouse/ClickHouse/pull/114413) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Speed up execution of `IN` over clustered (e.g. sorted by primary key) columns by reusing the previous row's result for equal consecutive rows. [#114853](https://github.com/ClickHouse/ClickHouse/pull/114853) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * Fixed quadratic time when the functions `h3kRing`, `h3ToChildren`, `h3PolygonToCells` and `h3PolygonToCellsWithContainment` build their result over many rows. A query over 196 000 rows spent 838 seconds copying memory in a CI stress run; it now takes about 5 seconds. [#115773](https://github.com/ClickHouse/ClickHouse/pull/115773) ([groeneai](https://github.com/groeneai)). * Optimize inverse dictionary lookups. A constant equality predicate such as `WHERE dictGet(dict, attr, key_expr) = value` is now constant-folded directly into a key filter (`key_expr = const`, `key_expr IN [...]`, or `WHERE 0`) instead of being rewritten to an `IN (SELECT ... FROM dictionary(...))` subquery, and the constant path of `dictGetKeys` now executes in parallel. [#91164](https://github.com/ClickHouse/ClickHouse/pull/91164) ([nihalzp](https://github.com/nihalzp)). * Add a new setting `merge_tree_generic_exclusion_search_max_steps` that limits the number of steps the generic exclusion search algorithm spends analyzing the primary key index of each data part. The budget is spent on the largest remaining key ranges first, so even a small budget prunes the bulk of the data; ranges that were not fully analyzed are read whole, so query results stay correct but more granules may be read. The limit is applied loosely: it may be exceeded by at most `merge_tree_coarse_index_granularity` steps, plus one step for each range the part is already divided into (for example, by the query condition cache). The default value 0 means unlimited steps. [#92779](https://github.com/ClickHouse/ClickHouse/pull/92779) ([EmeraldShift](https://github.com/EmeraldShift)). * Queries with `ORDER BY` and `LIMIT` on `VIEW`s over distributed tables now use the merge-sorted-streams optimization. The outer `ORDER BY`/`LIMIT` is pushed into a simple view's inner query so each shard sorts its data locally and the coordinator merges pre-sorted streams, instead of performing a full sort on the coordinator. [#94102](https://github.com/ClickHouse/ClickHouse/pull/94102) ([matanper](https://github.com/matanper)).

Storage, I/O, and data lake performance

* Use partition minmax index bounds to prune more granules during primary key analysis for `MergeTree` tables, when a primary key column is also an input column of the partition key. For example, in a `MergeTree` table with `ORDER BY (id, event_time)` and `PARTITION BY toYYYYMM(event_time)`, ClickHouse will use the partition minmax index on `event_time` during primary key index analysis to make more informed granule-pruning decisions. Controlled by the new setting `use_partition_minmax_for_primary_key_pruning` (enabled by default). [#103480](https://github.com/ClickHouse/ClickHouse/pull/103480) ([UnamedRus](https://github.com/UnamedRus)). * Speed up operators that scan a sorted stream for runs of equal key values — `DISTINCT` in order, `LIMIT BY` in order, negative `LIMIT BY` in order, `full_sorting_merge` and `partial_merge` joins. [#106502](https://github.com/ClickHouse/ClickHouse/pull/106502) ([nihalzp](https://github.com/nihalzp)). * The Parquet v3 reader can now skip whole row groups using the dictionary page (in addition to min/max statistics and bloom filters) for equality and `IN` conditions, when a column chunk is fully dictionary-encoded. Controlled by the new setting `input_format_parquet_dictionary_filter_push_down`. [#106952](https://github.com/ClickHouse/ClickHouse/pull/106952) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Use libdeflate for gzip/zlib/deflate compression and decompression, making it faster (compression \~1.15× with a better ratio, decompression \~1.4–1.5×) for `.gz`/HTTP/`url`/`s3` data and the Parquet GZIP codec. [#108074](https://github.com/ClickHouse/ClickHouse/pull/108074) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Speed up the default-whitespace `trimLeft`/`trimRight`/`trimBoth` functions (and their aliases `ltrim`/`rtrim`/`trim`): consecutive rows that need no trimming are copied in a single batch, and the per-row space scan runs only for rows that actually have a leading or trailing space. [#108177](https://github.com/ClickHouse/ClickHouse/pull/108177) ([groeneai](https://github.com/groeneai)). * Speed up reading of Iceberg tables by prefetching the next manifest file while the current manifest is being parsed, overlapping I/O with CPU work. [#108543](https://github.com/ClickHouse/ClickHouse/pull/108543) ([asya-ch](https://github.com/asya-ch)). * Vector search queries using rescoring now restrict exact distance calculation to vector-index candidate rows instead of brute-force rescoring neighbouring rows from the same `MergeTree` granule. [#108846](https://github.com/ClickHouse/ClickHouse/pull/108846) ([skuznetsov](https://github.com/skuznetsov)). * Added the `merge_tree_min_bytes_per_read_stream` setting to reduce excessive read streams and pipeline overhead for ordinary local unordered narrow-column `MergeTree` scans on high core-count servers. [#109035](https://github.com/ClickHouse/ClickHouse/pull/109035) ([jiebinn](https://github.com/jiebinn)). * Speed up serialization and merges of `JSON` columns whose shared data contains many sparse paths. When flattening shared data into per-path columns, `flattenAndBucketSharedDataPaths` no longer scans every accumulated path column on every row and no longer inserts a default one cell at a time; the per-row scan over all paths is removed and gaps are backfilled in bulk. [#109341](https://github.com/ClickHouse/ClickHouse/pull/109341) ([groeneai](https://github.com/groeneai)). * The native protocol now sends `String` columns as a separate stream of cumulative byte offsets (the same layout `Array` uses for its offsets) followed by the concatenated data, instead of a per-value varint length prefix, once both peers are on protocol revision 54489 or newer. A client can then read `String` columns with exact buffer preallocation and a single bulk copy, about 4 times faster than the per-value layout. Old clients and servers are unaffected: the layout is negotiated through the protocol revision. The same layout is available in the `Native` and `Buffers` formats through the new settings `output_format_native_write_string_with_size_stream` / `input_format_native_read_string_with_size_stream` (off by default, so the formats stay portable). [#110320](https://github.com/ClickHouse/ClickHouse/pull/110320) ([vahid-sohrabloo](https://github.com/vahid-sohrabloo)). * Queries with `FINAL` that read in the order of the primary key (`ORDER BY` a prefix of the sorting key) with a small `LIMIT` no longer use `split_intersecting_parts_ranges_into_layers_final`. [#110431](https://github.com/ClickHouse/ClickHouse/pull/110431) ([KochetovNicolai](https://github.com/KochetovNicolai)). * Lazy materialization (late materialization) for reading Parquet files from object storage, including Iceberg tables: for `ORDER BY ... LIMIT n` queries, the columns that are not needed for sorting and filtering are read only for the `n` rows that survive the `LIMIT`. On a 200 MB Parquet file on S3, `SELECT k, s ... ORDER BY k DESC LIMIT 10` reads 3.3 MB instead of 171 MB (51x less IO, 8x faster). Controlled by the new setting `query_plan_optimize_lazy_materialization_for_object_storage` (enabled by default, also requires `query_plan_optimize_lazy_materialization`). [#110970](https://github.com/ClickHouse/ClickHouse/pull/110970) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Queries on Iceberg tables with many delete files now start faster. ClickHouse reads and decodes the delete manifest files concurrently instead of one at a time, so their storage reads overlap. The new setting `iceberg_delete_manifest_decode_concurrency` (default `4`) controls how many are decoded at the same time. [#112679](https://github.com/ClickHouse/ClickHouse/pull/112679) ([asya-ch](https://github.com/asya-ch)). * Add opt-in Parquet serialization of UInt128/UInt256/Int128/Int256 as DECIMAL to enable row group and page pruning. [#113347](https://github.com/ClickHouse/ClickHouse/pull/113347) ([bobrik](https://github.com/bobrik)). * Fixed multiplied read requests on remote disks when a step of the readers chain reads over fragmented mark ranges, e.g., ranges filtered by primary or skip indexes. [#113584](https://github.com/ClickHouse/ClickHouse/pull/113584) ([CurtizJ](https://github.com/CurtizJ)). * Skip the redundant sortedness scan when writing `MergeTree` parts whose sorting keys are all constant. [#113899](https://github.com/ClickHouse/ClickHouse/pull/113899) ([perfloop-agent](https://github.com/perfloop-agent)). * Lazy materialization for `ORDER BY ... LIMIT n` queries now also applies to local Parquet files read with the `file` table function and the `File` table engine: the columns that are not needed for sorting and filtering are read only for the `n` rows that survive the `LIMIT`. The second read of a surviving file fails close with the new `FILE_CHANGED_DURING_READ` error if the file was modified between the two passes. Controlled by the new setting `query_plan_optimize_lazy_materialization_for_file` (enabled by default). Also fixes lazy materialization for object storage failing with `Not found column or subcolumn ... in block` when a requested subcolumn (e.g. of a `JSON` column) is deferred. [#114262](https://github.com/ClickHouse/ClickHouse/pull/114262) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Speed up set building for `IN (subquery)` on partitioned `MergeTree` tables by keeping each partition's rows within a single stream and deduplicating each stream independently, so the single set-filling transform — previously hashing every row serially — only sees unique rows. This applies when the partition expression is a deterministic function of the subquery's output columns. The optimization is not applied when the largest partition holds more than twice the rows of the average partition; the new setting `force_creating_set_partitions_independently` (disabled by default) bypasses this check. Controlled by the new setting `allow_creating_set_partitions_independently` (enabled by default). [#114645](https://github.com/ClickHouse/ClickHouse/pull/114645) ([nihalzp](https://github.com/nihalzp)). * Mark rotated non-replicated `MergeTree` system log tables without TTL as `table_readonly` to avoid unnecessary background operations. The `table_readonlyMergeTree` setting now also suppresses all background work on a plain `MergeTree` table — regular, TTL (`DELETE`/`MOVE`/recompression) and recompression merges, background mutations, and background part moves — and rejects the mutating partition commands (`ATTACH`/`MOVE`/`DROP`/`DROP DETACHED`/`FETCH`/`REPLACE PARTITION` and `MOVE PARTITION ... TO TABLE` into the table), in addition to the inserts, mutations, and `OPTIMIZE` it already rejected. As a result, a `table_readonly` table with a TTL no longer reclaims its expired data while the setting is enabled. [#95079](https://github.com/ClickHouse/ClickHouse/pull/95079) ([matt-metivier](https://github.com/matt-metivier)).

Memory and resource efficiency

* Reduce memory usage of `UNLOCK SNAPSHOT` by not loading per-data-part metadata that the unlock path does not use, avoiding out-of-memory for snapshots with a very large number of parts. [#111849](https://github.com/ClickHouse/ClickHouse/pull/111849) ([jkartseva](https://github.com/jkartseva)). * Reduce memory allocations when reading backup metadata: each file entry is now moved into the in-memory maps instead of being copied. [#111718](https://github.com/ClickHouse/ClickHouse/pull/111718) ([jkartseva](https://github.com/jkartseva)).

Function, type, and format performance

* Formatting `Decimal` values as text is much faster — up to 190 times for a `Decimal256` with a large scale, and about 1.6 times for a `Decimal64`. On processors with AVX-512 IFMA, converting 64-bit integers to text is about 1.4 times faster. Also fixed the rounding of `toDecimalString`, which dropped a carry out of the fractional part: `toDecimalString(toDecimal64('9.995', 3), 2)` returned `9.00` and now returns `10.00`. [#112457](https://github.com/ClickHouse/ClickHouse/pull/112457) ([thevar1able](https://github.com/thevar1able)). * Improved the performance of the `partial_merge` join algorithm on `FixedString` keys by comparing runs of values at once instead of one comparison call per value. [#105737](https://github.com/ClickHouse/ClickHouse/pull/105737) ([4ertus2](https://github.com/4ertus2)). * Single-column non-nullable string LowCardinality keys are natively supported now in `HashJoin`. [#107264](https://github.com/ClickHouse/ClickHouse/pull/107264) ([nickitat](https://github.com/nickitat)). * Improve the performance of `replaceAll` and `replaceRegexpAll` when the pattern is a single character and the replacement is a single character (for example `replaceRegexpAll(s, ' ', '_')`). Such replacements no longer change the string layout, so the column is now copied once and matching bytes are rewritten in place instead of running a per-match search loop. [#108178](https://github.com/ClickHouse/ClickHouse/pull/108178) ([groeneai](https://github.com/groeneai)). * Speed up parsing of the canonical `YYYY-MM-DD hh:mm:ss` date-time representation in best-effort mode (`date_time_input_format = 'best_effort'`, `cast_string_to_date_time_mode = 'best_effort'`), which is the default. This recovers a parsing performance regression introduced when those defaults were switched from `basic` to `best_effort`. [#108187](https://github.com/ClickHouse/ClickHouse/pull/108187) ([groeneai](https://github.com/groeneai)). * Parse deeply nested array and tuple literals in linear instead of quadratic time. [#108892](https://github.com/ClickHouse/ClickHouse/pull/108892) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Speed up function `toStartOfInterval` (aliases `time_bucket`, `date_bin`) by up to 5x for `SECOND`, `MINUTE` and `HOUR` intervals. [#109729](https://github.com/ClickHouse/ClickHouse/pull/109729) ([raimannma](https://github.com/raimannma)). * Functions `arraySort` and `arrayReverseSort` are several times faster over numeric arrays (including `Decimal` and `DateTime64`) when called without a lambda. [#109832](https://github.com/ClickHouse/ClickHouse/pull/109832) ([raimannma](https://github.com/raimannma)). * Speed up `addDays`, `addWeeks`, `subtractDays`, and `subtractWeeks` on `DateTime` and `DateTime64` values in fixed-offset time zones (such as UTC) by taking an arithmetic fast path. [#109836](https://github.com/ClickHouse/ClickHouse/pull/109836) ([raimannma](https://github.com/raimannma)). * Vectorize decompression of the `Delta` codec. Decoding is now 1.5–5 times faster for 8/16/32-bit data types, making scans of `Delta`-compressed columns up to 20% faster. [#110189](https://github.com/ClickHouse/ClickHouse/pull/110189) ([raimannma](https://github.com/raimannma)). * Speed up string search functions (`like`, `position`, `match`, `countSubstrings`, `hasToken`, etc.) over `Enum` columns with a constant needle by searching only the distinct enum names and mapping the results back per row, instead of searching every row. [#110325](https://github.com/ClickHouse/ClickHouse/pull/110325) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Improved the performance of substring search functions (`position`, `countSubstrings`, `multiSearch*`, `replaceAll`/`replaceOne`, and their case-insensitive and UTF-8 variants) with a non-constant needle by up to 1.5x, by using the SIMD-backed string searchers on the per-row path instead of a naive fallback. [#110580](https://github.com/ClickHouse/ClickHouse/pull/110580) ([Algunenano](https://github.com/Algunenano)). * Improved performance of squashing and merging `Dynamic` and `Object` columns with many dynamic variants or JSON paths. [#110583](https://github.com/ClickHouse/ClickHouse/pull/110583) ([cv4g](https://github.com/cv4g)). * Faster `estimateCompressionRatio` with the `NONE` codec [#111107](https://github.com/ClickHouse/ClickHouse/pull/111107) ([rienath](https://github.com/rienath)). * Speed up sorting and other comparison-based operations over `JSON` columns whose paths are stored in shared data. Previously each value comparison materialized a temporary `Dynamic` column and ran a full binary deserialization for both sides, making `ORDER BY` over such columns pathologically slow. [#111894](https://github.com/ClickHouse/ClickHouse/pull/111894) ([groeneai](https://github.com/groeneai)). * Division of 256-bit integers (`Int256`, `UInt256`, and `Decimal256`) now computes the quotient one 64-bit limb at a time, using a loop of 128 / 64 divisions when the divisor fits in 64 bits and Knuth's Algorithm D otherwise, instead of one bit at a time with shift-and-subtract long division. This affects `intDiv`, `modulo`, `Decimal256` arithmetic and conversion of these types to text. [#112477](https://github.com/ClickHouse/ClickHouse/pull/112477) ([thevar1able](https://github.com/thevar1able)). * Reading a `Nested` column from a binary-encoded type (`Native`, `RowBinary` with `input_format_binary_decode_types_in_binary_format`) no longer renders the type back into a type name and parses it again. Also, a malformed nested `Tuple` type name is now reported as a syntax error instead of taking time exponential in its nesting depth and failing with `TOO_SLOW_PARSING`. [#112560](https://github.com/ClickHouse/ClickHouse/pull/112560) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Speed up PromQL queries over TimeSeries tables: `timeSeriesIdToGroup` now reuses the previous row's group when consecutive rows carry the same series id, skipping the per-row id serialization and hash lookup. [#113580](https://github.com/ClickHouse/ClickHouse/pull/113580) ([nikitamikhaylov](https://github.com/nikitamikhaylov)).

Other performance improvements

* Implemented a native reader and writer for the `Arrow` and `ArrowStream` formats that does not use the Apache Arrow library, avoiding extra data copies and conversions. It is now the default (settings `input_format_arrow_use_native_reader` and `output_format_arrow_use_native_writer`) and is faster for both reading and writing. [#106522](https://github.com/ClickHouse/ClickHouse/pull/106522) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Optimize `nullIf(key, sentinel) = const` predicates to build key ranges and enable primary key, partition, and skip-index pruning. [#107308](https://github.com/ClickHouse/ClickHouse/pull/107308) ([adityaksolves](https://github.com/adityaksolves)). * Improve the performance of \`INTERSECT ALL\` and \`EXCEPT ALL\` (the default mode for \`INTERSECT\` and \`EXCEPT\`) by several times, by keying the multiset on the row value instead of hashing each row with SipHash. [#107649](https://github.com/ClickHouse/ClickHouse/pull/107649) ([Algunenano](https://github.com/Algunenano)). * Speed up the `match`, `extract`, `extractAll`, `replaceRegexpOne` and `replaceRegexpAll` functions for simple regular expressions by compiling them to native code with LLVM. Controlled by the new setting `compile_regular_expressions` (enabled by default); patterns outside the supported subset transparently fall back to the RE2 engine. [#108004](https://github.com/ClickHouse/ClickHouse/pull/108004) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Sped up ZSTD decompression on AArch64 (ARM) for columns with small match offsets, such as fixed-width integer columns, by vectorizing short-offset overlapping copies with NEON. For example, decompression of a `UInt64` column is up to \~2.3x faster on AWS Graviton 4. [#108049](https://github.com/ClickHouse/ClickHouse/pull/108049) ([alexey-milovidov](https://github.com/alexey-milovidov)). * LZ4 decompression speed was improved for fixed-size, low-cardinality columns. [#108175](https://github.com/ClickHouse/ClickHouse/pull/108175) ([nickitat](https://github.com/nickitat)). * Improved the performance of `levenshteinDistance` (and `editDistance`) for medium-length strings on ARM by avoiding per-row heap allocations in the blocked Myers code path. [#108185](https://github.com/ClickHouse/ClickHouse/pull/108185) ([groeneai](https://github.com/groeneai)). * Speed up parsing of floating-point numbers from text with `precise_float_parsing = 1`, making it as fast as or faster than the default parser on almost all inputs. Also bumps the bundled `fast_float` library to v8.2.10. [#108205](https://github.com/ClickHouse/ClickHouse/pull/108205) ([Algunenano](https://github.com/Algunenano)). * Base64 encoding and decoding functions now use `simdutf`, improving their performance. [#108333](https://github.com/ClickHouse/ClickHouse/pull/108333) ([thevar1able](https://github.com/thevar1able)). * Optimize `DISTINCT` for expensive high cardinality keys. [#108366](https://github.com/ClickHouse/ClickHouse/pull/108366) ([nihalzp](https://github.com/nihalzp)). * Improved math-function performance by enabling compiler optimizations that do not preserve `errno`. [#108628](https://github.com/ClickHouse/ClickHouse/pull/108628) ([nickitat](https://github.com/nickitat)). * Speed up `estimateCompressionRatio` for `T64`-encoded columns by computing the compressed size analytically instead of compressing. [#109054](https://github.com/ClickHouse/ClickHouse/pull/109054) ([rienath](https://github.com/rienath)). * JOINs can now use primary key index or skip index on the left hand side table to prune granules [#109085](https://github.com/ClickHouse/ClickHouse/pull/109085) ([shankar-iyer](https://github.com/shankar-iyer)). * Improve the performance of H3 geo functions (`h3ToGeoBoundary`, `h3ToGeo`, `h3CellAreaM2`/`h3CellAreaRads2`, `geoToH3`) by computing paired sine/cosine together and eliminating redundant trigonometric calls in the coordinate transforms. Results are unchanged (bit-for-bit identical). [#109399](https://github.com/ClickHouse/ClickHouse/pull/109399) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Improve performance of comparisons (`<`, `>`, `<=`, `>=`) of wide integer types (`Int128`, `UInt128`, `Int256`, `UInt256` and types based on them, such as `Decimal128`) by up to 7 times, and of converting signed integers to `Int256` (up to 7 times on mixed-sign data), by making the comparison and sign-extension code branchless. [#109474](https://github.com/ClickHouse/ClickHouse/pull/109474) ([raimannma](https://github.com/raimannma)). * Functions with a single non-const `Nullable` argument now share the argument's null map with the result instead of allocating and merging a new one. [#110151](https://github.com/ClickHouse/ClickHouse/pull/110151) ([raimannma](https://github.com/raimannma)). * Improve performance of functions `arrayMin` and `arrayMax` over numeric arrays by \~1.3-1.5x by using a vectorized reduction instead of a per-element comparison loop. [#110163](https://github.com/ClickHouse/ClickHouse/pull/110163) ([raimannma](https://github.com/raimannma)). * Improve performance of `DISTINCT` when the input stream is sorted by a prefix of the distinct columns, for example `SELECT DISTINCT a, b FROM table ORDER BY a`. Such queries run up to 2.4 times faster. [#110170](https://github.com/ClickHouse/ClickHouse/pull/110170) ([nihalzp](https://github.com/nihalzp)). * Vectorized batch paths for `groupBitOr`/`groupBitAnd`/`groupBitXor` and the variance family (`varPop`, `varSamp`, `stddevPop`, `stddevSamp`, `skewPop`, `skewSamp`, `kurtPop`, `kurtSamp`, `covarPop`, `covarSamp`, `corr`): up to 4x faster with the `-If` combinator on unpredictable conditions, and up to 3x faster for the variance family without it. [#110461](https://github.com/ClickHouse/ClickHouse/pull/110461) ([raimannma](https://github.com/raimannma)). * Optimized analysis of the text index. [#110530](https://github.com/ClickHouse/ClickHouse/pull/110530) ([CurtizJ](https://github.com/CurtizJ)). * Speed up parsing of `UUID` values from text (e.g. `JSONExtract` into `LowCardinality(UUID)`, `toUUID`, `CAST AS UUID`) by validating and converting hex digits in a single pass instead of two. [#110625](https://github.com/ClickHouse/ClickHouse/pull/110625) ([groeneai](https://github.com/groeneai)). * Sped up ZSTD decompression on x86\_64 for columns with small match offsets, such as fixed-width integer columns, by vectorizing short-offset overlapping copies with SSSE3 `pshufb` (the x86\_64 counterpart of the AArch64 NEON optimization from [#108049](https://github.com/ClickHouse/ClickHouse/pull/108049)). [#110909](https://github.com/ClickHouse/ClickHouse/pull/110909) ([thevar1able](https://github.com/thevar1able)). * Speed up `groupUniqArray` for numeric types by up to 3x by using CRC32 hashing (as in `uniqExact`) and inlining the per-row insertion. [#110917](https://github.com/ClickHouse/ClickHouse/pull/110917) ([raimannma](https://github.com/raimannma)). * Improve efficiency of minmax and basic column statistics for floating point columns. [#111221](https://github.com/ClickHouse/ClickHouse/pull/111221) ([cv4g](https://github.com/cv4g)). * Returns `COUNT()` queries directly from the text index cardinality metadata. [#111494](https://github.com/ClickHouse/ClickHouse/pull/111494) ([ahmadov](https://github.com/ahmadov)). * Speed up `min` and `max` on 128-bit and 256-bit types (`Int128`, `UInt128`, `Int256`, `UInt256`, `Decimal128`, `Decimal256`). Up to 1.8x faster for wide integers and up to 3.4x faster for wide decimals. [#111965](https://github.com/ClickHouse/ClickHouse/pull/111965) ([raimannma](https://github.com/raimannma)). * Fix a performance regression in `system.query_log` ingestion. Building the `Settings` and `asynchronous_read_counters` map columns was about 6 times slower than before, which could make `SYSTEM FLUSH LOGS query_log` exceed its 180 second timeout on servers that log many queries with many changed settings. [#112210](https://github.com/ClickHouse/ClickHouse/pull/112210) ([groeneai](https://github.com/groeneai)). * Add optional caching for tokens missing from text indexes. [#112742](https://github.com/ClickHouse/ClickHouse/pull/112742) ([rorylshanks](https://github.com/rorylshanks)). * Asynchronous logging now uses a bounded lock-free queue: threads writing log messages never block on a mutex or wake up the logging threads. [#112803](https://github.com/ClickHouse/ClickHouse/pull/112803) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Use the primary key index for `pointInPolygon` when the point argument is a whole key column of type `Point` (or another `Tuple` of two numeric elements), e.g. `pointInPolygon(coord, [...])` for a table ordered by `coord`. Previously only the `pointInPolygon((x, y), [...])` form with two scalar key columns was analyzed. [#112956](https://github.com/ClickHouse/ClickHouse/pull/112956) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `text_index_posting_list_apply_mode` now defaults to `lazy`, so text index queries decode posting lists on demand at packed-block granularity with a cursor instead of eagerly materializing them into bitmaps for text indexes with set `posting_list_codec`. [#113521](https://github.com/ClickHouse/ClickHouse/pull/113521) ([CurtizJ](https://github.com/CurtizJ)). * Speed up vectorized TimeSeries tag transformations by replacing hash-based remapping with direct indexing when tag-group IDs occupy a compact range. [#114244](https://github.com/ClickHouse/ClickHouse/pull/114244) ([fallintoplace](https://github.com/fallintoplace)). * Improved performance of merges of text indexes. [#114525](https://github.com/ClickHouse/ClickHouse/pull/114525) ([CurtizJ](https://github.com/CurtizJ)). * Use compact presence masks for PromQL set operators. [#115224](https://github.com/ClickHouse/ClickHouse/pull/115224) ([fallintoplace](https://github.com/fallintoplace)). * Attaching the system tables no longer copies each table's metadata, including its full column description, twice just to set the table comment. [#115490](https://github.com/ClickHouse/ClickHouse/pull/115490) ([thevar1able](https://github.com/thevar1able)). * Speed up duplicate-series checks for all-zero PromQL condition vectors. [#115822](https://github.com/ClickHouse/ClickHouse/pull/115822) ([fallintoplace](https://github.com/fallintoplace)). * Speed up `toUTCTimestamp`/`to_utc_timestamp` and `fromUTCTimestamp`/`from_utc_timestamp` on `DateTime64` arguments, restoring the throughput they had after earlier overflow corrections. Results are unchanged. [#115838](https://github.com/ClickHouse/ClickHouse/pull/115838) ([groeneai](https://github.com/groeneai)). * Reduced the fixed setup cost of random data generation (`rand`, `rand64`, `randomString`, `randomFixedString`, `generateUUIDv4`, `generateUUIDv7`) on x86-64. Noticeable when blocks are small; generated values are unchanged. [#116158](https://github.com/ClickHouse/ClickHouse/pull/116158) ([Algunenano](https://github.com/Algunenano)). * You can now enable early short-circuit folding for builtin `and` and `or` during analysis with the `enable_function_early_short_circuit` setting. This can skip executing dead single-row `count()` scalar subqueries while preserving normal type inference and semantic validation. [#83505](https://github.com/ClickHouse/ClickHouse/pull/83505) ([fhw12345](https://github.com/fhw12345)). * Use a single buffer IO call for trivially serializable vectors [#89842](https://github.com/ClickHouse/ClickHouse/pull/89842) ([Ergus](https://github.com/Ergus)). * Enable `optimize_or_like_chain` by default. [#94517](https://github.com/ClickHouse/ClickHouse/pull/94517) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Improves the performance of fetching a single user record from system.users when the server has many users. [#96699](https://github.com/ClickHouse/ClickHouse/pull/96699) ([alistairjevans](https://github.com/alistairjevans)).

Improvements

Query and SQL improvements

* Materialized CTEs referenced from multiple branches of a `UNION` query are now materialized once and shared across all branches. Previously each branch received its own copy of the CTE, which was inlined and evaluated separately. [#102107](https://github.com/ClickHouse/ClickHouse/pull/102107) ([novikd](https://github.com/novikd)). * Skip JOIN runtime filter when the probe side is estimated to be small. New setting `join_runtime_filter_min_probe_rows` (default `1000`) controls the threshold [#104860](https://github.com/ClickHouse/ClickHouse/pull/104860) ([vdimir](https://github.com/vdimir)). * Make `EXPLAIN [PLAN] actions=1, compact=1, pretty=1` the default. [#105036](https://github.com/ClickHouse/ClickHouse/pull/105036) ([Fgrtue](https://github.com/Fgrtue)). * Reduced cancellation latency for the positive forms of LIMIT BY (LimitByTransform, LimitBySortedStreamTransform): KILL QUERY and Ctrl+C now interrupt processing within a single chunk instead of waiting for the entire step to complete. [#106070](https://github.com/ClickHouse/ClickHouse/pull/106070) ([rvasin](https://github.com/rvasin)). * JOIN queries are supported with automatic parallel replicas. [#106073](https://github.com/ClickHouse/ClickHouse/pull/106073) ([nickitat](https://github.com/nickitat)). * Adds a new asynchronous metric UntrackedMemory (visible in `system.asynchronous_metrics`) that reports memory already allocated by threads but not yet accounted in the global memory tracking counter. Each thread accumulates small allocations locally and reports them in bulk. This helps explain discrepancies between `MemoryTracking` and the process's actual memory usage. `MEMORY_LIMIT_EXCEEDED` error messages now also include the amount of untracked memory, making it easier to understand why a query or the server hit its memory limit. [#106386](https://github.com/ClickHouse/ClickHouse/pull/106386) ([mstetsyuk](https://github.com/mstetsyuk)). * When an `INSERT` fails because a materialized view's `TO` target table rejects the write while the insert pipeline is being built (for example the target is a `View`, or it does not support transactions), the error message now names the materialized view and its target table instead of only reporting the bare storage error. [#107234](https://github.com/ClickHouse/ClickHouse/pull/107234) ([groeneai](https://github.com/groeneai)). * `ALTER TABLE ... MODIFY COLUMN Tuple(...)` on a named `Tuple` is now metadata-only when only adding subfields, matching the speed of top-level `ADD COLUMN`. Gated behind `SET allow_metadata_only_named_tuple_alter = 1`. [#107305](https://github.com/ClickHouse/ClickHouse/pull/107305) ([amosbird](https://github.com/amosbird)). * Added the create-time `materialized_postgresql_use_extended_date_and_time_types` setting for the `MaterializedPostgreSQL` database engine. By default (enabled), PostgreSQL `date`/`timestamp` columns are inferred as `Date32`/`DateTime64`; setting it to `0` at `CREATE DATABASE` time infers the narrower `Date`/`DateTime` types. The setting is not applicable to the `MaterializedPostgreSQL` table engine. [#107428](https://github.com/ClickHouse/ClickHouse/pull/107428) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `EXPLAIN PLAN` now marks a join's estimated result rows as approximate (`~~...`) when the estimate came from a hint or randomized fallback rather than table statistics, helping identify plans that lack reliable statistics for join reordering. [#107666](https://github.com/ClickHouse/ClickHouse/pull/107666) ([vdimir](https://github.com/vdimir)). * `INSERT` into a `MergeTree` table now honors query cancellation and `max_execution_time` while writing many parts, instead of potentially running long after being killed. [#107929](https://github.com/ClickHouse/ClickHouse/pull/107929) ([al13n321](https://github.com/al13n321)). * `arrayFold` now respects query cancellation and `max_execution_time`. Previously a fold over a very long array (for example `arrayFold(acc, x -> arrayPushFront(acc, x), range(number), emptyArrayUInt64())`) ran the whole fold inside a single function call and could not be interrupted, so `KILL QUERY` and time limits were ignored until the fold finished. [#108192](https://github.com/ClickHouse/ClickHouse/pull/108192) ([groeneai](https://github.com/groeneai)). * Add per-phase query pre-execution `ProfileEvents`: `QueryParseMicroseconds`, `QueryAnalysisMicroseconds`, `QueryPlanBuildMicroseconds` and `QueryPipelineBuildMicroseconds`. They expose where time is spent before query execution (parsing, analysis, query plan building, pipeline building) and are available in `system.query_log` and `system.events`. [#108282](https://github.com/ClickHouse/ClickHouse/pull/108282) ([jrdi](https://github.com/jrdi)). * `ALTER TABLE` operations that would produce table metadata exceeding `max_query_size` are now rejected upfront, preventing tables from becoming unloadable by components such as DDL distribution and replica recovery. [#108283](https://github.com/ClickHouse/ClickHouse/pull/108283) ([lockie](https://github.com/lockie)). * Added `ConstantJoin` for cartesian joins and analyzer-planned constant-predicate joins, so these queries no longer fail because of an incompatible `join_algorithm` setting when the analyzer is used. [#108289](https://github.com/ClickHouse/ClickHouse/pull/108289) ([antaljanosbenjamin](https://github.com/antaljanosbenjamin)). * Added a `source` column to `system.documentation` containing the path of the source file where each entity's documentation is defined. The table now also documents compression codecs, profile events, current metrics, asynchronous metrics, and the system tables themselves (with their columns), and the documentation of settings now includes their type and default value. [#108346](https://github.com/ClickHouse/ClickHouse/pull/108346) ([alexey-milovidov](https://github.com/alexey-milovidov)). * If you provide WHERE with `table` and `database` filters then `system.iceberg_history` will list only matched databases. It can save some time if you have many non-related remote databases on the server. [#108492](https://github.com/ClickHouse/ClickHouse/pull/108492) ([Diskein](https://github.com/Diskein)). * Query parameters (`{name:Type}` substitutions) can now be used as setting values, both in the `SETTINGS` clause of a query (such as `SELECT` or `INSERT`) and in standalone `SET` queries, e.g. `SELECT ... SETTINGS max_threads = {threads:UInt64}` and `SET max_threads = {threads:UInt64}`. [#108760](https://github.com/ClickHouse/ClickHouse/pull/108760) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allows to calculate the combined skip-index benefit in EXPLAIN WHATIF. It shows the data ratio, which is the intersection of the performance of all existing suitable hypothetical indices. [#108934](https://github.com/ClickHouse/ClickHouse/pull/108934) ([yariks5s](https://github.com/yariks5s)). * A query with a single CTE is now formatted with the same newline and indentation as a query with multiple CTEs. Previously `WITH a AS (...)` kept the CTE on the same line as `WITH`, while two or more CTEs put `WITH` on its own line with each CTE indented. [#109092](https://github.com/ClickHouse/ClickHouse/pull/109092) ([groeneai](https://github.com/groeneai)). * Support the Query Condition Cache for local Parquet files read via the `File` table engine (previously supported only for object storage and data lakes). [#109247](https://github.com/ClickHouse/ClickHouse/pull/109247) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Key the Query Condition Cache for remote (object storage) Parquet files by ETag in addition to the path, so that overwriting an object in place no longer serves stale cached results. [#109310](https://github.com/ClickHouse/ClickHouse/pull/109310) ([alexey-milovidov](https://github.com/alexey-milovidov)). * An unqualified reference to an `INNER JOIN` key that is equated in the `ON` condition (e.g. `SELECT id FROM a INNER JOIN b ON a.id = b.id`) is no longer reported as an ambiguous identifier, since both sides are guaranteed to carry the same value. [#109366](https://github.com/ClickHouse/ClickHouse/pull/109366) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The transposed distance functions over `QBit` (`L2DistanceTransposed`, `cosineDistanceTransposed`, `dotProductTransposed`, and their quantized counterparts `L2DistanceTransposedQuantized`, `cosineDistanceTransposedQuantized`, `dotProductTransposedQuantized`) now accept a reference vector longer than the number of dimensions being searched, ignoring the extra trailing elements. This allows reusing a full-size query vector for a reduced-dimension (Matryoshka) search without slicing it first. [#109388](https://github.com/ClickHouse/ClickHouse/pull/109388) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added a sanity check that a distributed query always carries a known client version, throwing a logical error instead of silently forwarding a zero version to remote shards. Server-initiated queries (background flushes, streaming consumers, dictionary reloads, asynchronous insert flushes) now report the server's own version as the initiator version. [#109408](https://github.com/ClickHouse/ClickHouse/pull/109408) ([alexey-milovidov](https://github.com/alexey-milovidov)). * This patch fixes the usage of qualified column names in the where clause of the mutation operation over the merge tree table engines. [#109491](https://github.com/ClickHouse/ClickHouse/pull/109491) ([Michicosun](https://github.com/Michicosun)). * Support non-partitioned window functions (`OVER (ORDER BY ...)`) in distributed query plans (`make_distributed_plan`). Previously any query with a window function failed because `WindowStep` could not be serialized for remote execution; windows with `PARTITION BY` are still rejected for now. [#109802](https://github.com/ClickHouse/ClickHouse/pull/109802) ([VighneshPath](https://github.com/VighneshPath)). * Now it's possible to alter some auth settings of one-lake catalog with `ALTER DATABASE ... MODIFY SETTING = 'xxx'`. [#110019](https://github.com/ClickHouse/ClickHouse/pull/110019) ([alesapin](https://github.com/alesapin)). * A subquery on the right side of `IN` whose single column is an array one dimension deeper than the left argument is now interpreted as the set of the array's elements (like an array literal or an array-returning function), instead of failing with a confusing type-mismatch error. For example, `x IN (SELECT groupArray(x) FROM ...)` now works. [#110169](https://github.com/ClickHouse/ClickHouse/pull/110169) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Queries accelerated by TopK dynamic filtering (`ORDER BY ... LIMIT k`) now make fuller use of the query condition cache: the cache is populated even when lazy materialization is applied, and such queries can reuse entries previously written by an ordinary query with the same `WHERE` predicate. [#110507](https://github.com/ClickHouse/ClickHouse/pull/110507) ([shankar-iyer](https://github.com/shankar-iyer)). * `SHOW CREATE TABLE` (and `SHOW CREATE VIEW` / `SHOW CREATE DICTIONARY`) now suggests a similarly-named table in the error message when the requested table does not exist, the same way `SELECT` queries do. [#110633](https://github.com/ClickHouse/ClickHouse/pull/110633) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Add STREAM BOUNDED modifier, which read only the first snapshot of a streaming query, then finish instead of subscribing for updates. [#110653](https://github.com/ClickHouse/ClickHouse/pull/110653) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Fix the query result cache for PromQL queries. The `promql` dialect bypassed the non-deterministic-function check and could serve stale results anchored at `now()`; the Prometheus HTTP API (`/api/v1/query`, `/api/v1/query_range`) never stored cache entries at all. [#110887](https://github.com/ClickHouse/ClickHouse/pull/110887) ([nikitamikhaylov](https://github.com/nikitamikhaylov)). * `EXPLAIN ANALYZE` now reports per-side join row counts, match rates, fanout, and algorithm-specific state such as hash-table memory and spilling. Use `matches = 1` to collect match metrics that need additional bookkeeping. [#110892](https://github.com/ClickHouse/ClickHouse/pull/110892) ([Fgrtue](https://github.com/Fgrtue)). * Fixed a heap-use-after-free in the pipeline executor that could crash the server when a query removes processors at runtime (for example `FINAL` with lazy reads or external sort), found by the AST fuzzer. [#111017](https://github.com/ClickHouse/ClickHouse/pull/111017) ([groeneai](https://github.com/groeneai)). * Fix a use-after-free (segmentation fault) when a query with a shared storage snapshot (setting `enable_shared_storage_snapshot_in_query`) strips data parts from the snapshot — e.g. a streaming subquery reading the same `MergeTree` table that another part of the query reads concurrently. [#111192](https://github.com/ClickHouse/ClickHouse/pull/111192) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Support distributed query plans (`make_distributed_plan`) for queries whose source provably produces no rows at planning time, such as a read from an empty `MergeTree` table. Such queries previously failed with `SUPPORT_IS_DISABLED`. [#111941](https://github.com/ClickHouse/ClickHouse/pull/111941) ([groeneai](https://github.com/groeneai)). * Fixed the *query condition cache* not being populated for granules eliminated by any but the last conjunct of a `WHERE` clause with multiple `AND` conditions. [#112083](https://github.com/ClickHouse/ClickHouse/pull/112083) ([shankar-iyer](https://github.com/shankar-iyer)). * Reduced peak memory when writing JSON columns with bucketed shared data serialization (map\_with\_buckets and advanced) by splitting the shared data into buckets one at a time on write instead of materializing all buckets simultaneously. This is mostly beneficial for Array(JSON) columns: for scalar JSON the shared data is split per granule (bounded by index\_granularity\_bytes), so the saving is small, whereas a single Array(JSON) granule can hold many nested rows and thus a large amount of shared data, where the one-bucket-at-a-time split significantly lowers peak memory [#112172](https://github.com/ClickHouse/ClickHouse/pull/112172) ([Avogar](https://github.com/Avogar)). * Memory reservation scheduling (experimental) no longer evicts an allocation when a pending memory release would free the room needed by an over-limit reservation increase: the eviction decision now waits for in-flight decreases to be applied, avoiding unnecessary query kills under memory pressure. [#112299](https://github.com/ClickHouse/ClickHouse/pull/112299) ([serxa](https://github.com/serxa)). * `SYSTEM DROP FILESYSTEM CACHE` now removes cache metadata keys in parallel, making large filesystem-cache clears faster. The new file-cache `drop_cache_threads` setting controls the number of removal workers and defaults to 16; a value of 1 performs removal in the query thread. [#112532](https://github.com/ClickHouse/ClickHouse/pull/112532) ([kssenii](https://github.com/kssenii)). * The modifiers of a column declaration - `COMMENT`, `CODEC`, `STATISTICS`, `TTL`, `COLLATE`, `PRIMARY KEY` and per-column `SETTINGS` - can now be written in any order in `CREATE TABLE`, and each of them at most once. Previously only one fixed order was accepted, and, for example, `x UInt64 CODEC(ZSTD) COMMENT 'text'` was a syntax error. In `ALTER TABLE ... ADD COLUMN` / `MODIFY COLUMN`, the supported modifiers - `COMMENT`, `CODEC`, `STATISTICS`, `TTL` and per-column `SETTINGS` - can also be written in any order; per-column `SETTINGS` in `ADD COLUMN` and a declared `STATISTICS` in `ADD COLUMN` / `MODIFY COLUMN` are now applied instead of being silently dropped, and `COLLATE` and `PRIMARY KEY` in these `ALTER` commands now throw an exception instead of being silently ignored. [#112788](https://github.com/ClickHouse/ClickHouse/pull/112788) ([alexey-milovidov](https://github.com/alexey-milovidov)). * A query over a `Merge` table (or the `merge` table function) that matches many tables now reacts to `KILL QUERY` and `max_execution_time` while the per-table query plans are being built, instead of only after the planning of all children has finished. [#113415](https://github.com/ClickHouse/ClickHouse/pull/113415) ([alexey-milovidov](https://github.com/alexey-milovidov)). * A `SELECT` from a `Merge` table that exceeds `max_execution_time` with `timeout_overflow_mode = 'break'` now stops building query plans for the remaining matched tables, instead of running full query analysis for every matched table before stopping. The query still finishes without an error, as that mode documents. [#113642](https://github.com/ClickHouse/ClickHouse/pull/113642) ([groeneai](https://github.com/groeneai)). * When a column's declared type and its data diverge during a `MergeTree` read (mixed type provenance, e.g. after an `ALTER TABLE ... MODIFY COLUMN` whose mutation has not finished), the server now reports a clear exception naming the column and both structures, instead of an unchecked cast: debug and sanitizer builds previously aborted with a bare `Bad cast from type A to B` naming no column, and release builds walked mismatched memory silently. [#114087](https://github.com/ClickHouse/ClickHouse/pull/114087) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Enabled `ie_join` by default: the default value of `join_algorithm` is now `direct,parallel_hash,hash,ie_join`. A `JOIN` whose `ON` section has only inequality conditions (two comparisons `<`, `<=`, `>`, `>=` between expressions of the joined tables) is now executed with the sort-based IEJoin algorithm instead of a `CROSS JOIN` with a filter, and `LEFT`/`RIGHT`/`FULL`/`SEMI`/`ANTI` joins with such conditions are supported. Since `ie_join` is last in the list, it is used only when the other algorithms do not apply. [#114327](https://github.com/ClickHouse/ClickHouse/pull/114327) ([vdimir](https://github.com/vdimir)). * Propagate the OpenTelemetry trace context into the fibers used for asynchronous connection establishment and remote query execution (hedged requests, `async_socket_for_remote`, `async_query_sending_for_remote`). Spans of remote queries in a distributed query are now correctly parented under the initiator's `Connection::sendQuery` span instead of being attached directly to the client-supplied trace context, and each asynchronous task execution produces its own span. [#114813](https://github.com/ClickHouse/ClickHouse/pull/114813) ([diegomestre2](https://github.com/diegomestre2)). * `DROP DATABASE` now drops tables in reverse topological order of both loading and referential dependencies (previously only loading dependencies), so a crash in the middle of `DROP DATABASE` cannot leave a table whose dependencies were already dropped. [#114952](https://github.com/ClickHouse/ClickHouse/pull/114952) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Support PostgreSQL-style regular expression match operators: `~` (an alias for the `match` function), `~*` (case-insensitive match), `!~` and `!~*` (negations). The `\d`, `\dt`, `\dv` commands of `psql` now work when connected to ClickHouse over the PostgreSQL compatibility protocol, and a failed query no longer terminates the connection. [#115066](https://github.com/ClickHouse/ClickHouse/pull/115066) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `EXPLAIN WHATIF` now says why the empirical estimate was skipped, in a new `empirical_reason` line shown when `empirical_status` is `unsupported`. [#115140](https://github.com/ClickHouse/ClickHouse/pull/115140) ([yariks5s](https://github.com/yariks5s)). * When no join algorithm enabled by the `join_algorithm` setting can execute a `JOIN`, the error message now names the algorithms that were tried and the strictness and kind of the JOIN that failed, instead of reporting only that none of them worked. [#115226](https://github.com/ClickHouse/ClickHouse/pull/115226) ([elkinal](https://github.com/elkinal)). * Vector search queries can now use the vector similarity index when the reference vector is an integer array literal, e.g. `ORDER BY L2Distance(vec, [1, 2])`. Previously, such queries silently fell back to a brute-force scan. [#115255](https://github.com/ClickHouse/ClickHouse/pull/115255) ([rschu1ze](https://github.com/rschu1ze)). * `BACKUP` no longer leaves a lock file behind in its destination when the attempt that created it fails before writing anything -- whether it fails while claiming the destination or while opening an archive. No later backup could match such a lock, so every retry to the same destination failed until the file was deleted by hand. Backups to `S3` and `Azure` destinations also create the lock exclusively, and a lock that belongs to another backup is reported as a concurrent backup rather than as whatever error the storage returned. [#115387](https://github.com/ClickHouse/ClickHouse/pull/115387) ([jkartseva](https://github.com/jkartseva)). * A trace started by sampling (`opentelemetry_start_trace_probability`) is now written back into the query's client info, so remote and distributed secondary queries, and DDL entries, join the same trace instead of starting disjoint ones. [#115619](https://github.com/ClickHouse/ClickHouse/pull/115619) ([diegomestre2](https://github.com/diegomestre2)). * Fixed `CREATE TABLE` for Iceberg tables backed by Amazon S3 Tables, which manages table locations itself. [#115652](https://github.com/ClickHouse/ClickHouse/pull/115652) ([scanhex12](https://github.com/scanhex12)). * The adaptive `GROUP BY` aggregation (`enable_adaptive_aggregator`) now measures whether deferring rare keys pays off and returns to the ordinary algorithm when their keys or aggregate arguments are too wide to copy profitably, so aggregations over heavy string columns (e.g. `min(URL)` per user) run at the ordinary algorithm's speed and memory. Adds `adaptive_aggregator_freeze_threshold_bytes` to bound each thread's frozen table in bytes (whichever of it and the key-count threshold is reached first freezes the table). [#115755](https://github.com/ClickHouse/ClickHouse/pull/115755) ([nihalzp](https://github.com/nihalzp)). * Adds setting `query_plan_aggregation_bucket_top_k` (enabled by default) to toggle the plan optimization that materializes only each two-level bucket's best `n` groups when a final aggregation feeds `ORDER BY` over its outputs with `LIMIT n`. The optimization is now also visible in `EXPLAIN` output. [#116124](https://github.com/ClickHouse/ClickHouse/pull/116124) ([nihalzp](https://github.com/nihalzp)). * Improve `DESCRIBE` query support for parameterized views. Support parameterized views in scalar expressions in the new analyzer. [#68978](https://github.com/ClickHouse/ClickHouse/pull/68978) ([novikd](https://github.com/novikd)). * Queries like `SELECT * FROM t WHERE id` will now use index skipping on the `id` column. [#89603](https://github.com/ClickHouse/ClickHouse/pull/89603) ([adityachopra29](https://github.com/adityachopra29)). * Added the `query_plan_merge_expression_into_join` setting to allow merging Expression steps into JOIN steps during join reordering optimization. This enables join reordering across subqueries that wrap joins (e.g., when a JOIN is inside a subquery with computed columns), leading to better optimization of complex join trees. [#98533](https://github.com/ClickHouse/ClickHouse/pull/98533) ([vdimir](https://github.com/vdimir)). * Added an `is_wildcard` column to the `system.grants` table that indicates whether a grant uses wildcard prefix matching (e.g. `GRANT SELECT ON db*.*`). Previously, wildcard and exact grants on the same name were indistinguishable in `system.grants`. [#98577](https://github.com/ClickHouse/ClickHouse/pull/98577) ([il9ue](https://github.com/il9ue)).

Functions and data types

* Added a new setting `allow_lossy_numeric_supertype` (disabled by default). When enabled, `if`, `multiIf`, `coalesce`, `ifNull`, `array` and `map` over numeric arguments that have no lossless common type (for example a `Decimal` and a `Float64`, or an `Int64` and a `Float64`) resolve to the numeric supertype `Float64` instead of a `Variant`, so the result can be used with aggregate functions like `sum`, `avg`, `min` and `max`. The setting name is now mentioned in the relevant aggregate function error messages. [#107236](https://github.com/ClickHouse/ClickHouse/pull/107236) ([groeneai](https://github.com/groeneai)). * The `SOME` / `ALL` array quantifier (`expr OP SOME(array)` / `expr OP ALL(array)`) now also supports the keyword comparison predicates `IS DISTINCT FROM` and `IS NOT DISTINCT FROM`, and the string-search predicates `LIKE`, `ILIKE`, `NOT LIKE`, `NOT ILIKE`, and `REGEXP`, rewritten to `arrayExists` / `arrayAll`. [#107454](https://github.com/ClickHouse/ClickHouse/pull/107454) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Add `input_format_json_max_object_size` setting to limit JSON object size on parsing. [#107669](https://github.com/ClickHouse/ClickHouse/pull/107669) ([Avogar](https://github.com/Avogar)). * Fixed a memory leak that occurred when opening a SQLite database failed (for example, when the `sqlite` table function or the `SQLite` database engine is given a path that cannot be opened). [#107807](https://github.com/ClickHouse/ClickHouse/pull/107807) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Support the `length` function for the `QBit` data type. It returns the dimension of the vector as a constant. [#108071](https://github.com/ClickHouse/ClickHouse/pull/108071) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Support `CAST` from a `QBit` to an `Array`, reconstructing the original vector. This is the inverse of the existing `Array` to `QBit` conversion. [#108072](https://github.com/ClickHouse/ClickHouse/pull/108072) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The `MySQL` database engine, table engine and table function now map MySQL's spatial column types (`LINESTRING`, `POLYGON`, `MULTILINESTRING`, `MULTIPOLYGON`, and the generic `GEOMETRY`) to the corresponding ClickHouse geometric types instead of `String`. This is controlled by the new `geometry` flag of the `mysql_datatypes_support_level` setting, enabled by default. `POINT` is still always converted to `Point`. The generic `GEOMETRY` column maps to the umbrella `Geometry` type; reading a value whose subtype has no ClickHouse counterpart (`MULTIPOINT`, `GEOMETRYCOLLECTION`) throws an exception at read time. [#108944](https://github.com/ClickHouse/ClickHouse/pull/108944) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Use the precise (closest-representable) float parsing algorithm by default and apply the `precise_float_parsing` setting to input formats (CSV, TSV, JSON, VALUES, ...) and numeric literals, not just `toFloat*`/`CAST`. Set `precise_float_parsing = 0` for the previous, faster in some cases but less accurate, behavior. [#109086](https://github.com/ClickHouse/ClickHouse/pull/109086) ([Algunenano](https://github.com/Algunenano)). * Support `reinterpret` of an `Array` of fixed-size elements as a `String`, the inverse of the existing `reinterpret` of a `String`/`FixedString` as an `Array`. [#109383](https://github.com/ClickHouse/ClickHouse/pull/109383) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allow `CAST` between `QBit` types that differ in the element type and/or the stride, as long as the dimension stays the same (for example `CAST(x AS QBit(Float64, N))` from a `QBit(Float32, N)`). Stride-only changes are lossless; element-type changes follow the corresponding `Array` conversion semantics (for example `Float32` to `BFloat16` may lose precision, and `accurateCast` / `accurateCastOrNull` reject rows that are not exactly representable). [#109387](https://github.com/ClickHouse/ClickHouse/pull/109387) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Functions `quantizeBFloat16ToInt8` and `dequantizeInt8ToBFloat16` now also accept `Array` and `QBit` arguments, applying the Lloyd-Max codec to the whole vector (returning `Array`/`QBit` of the corresponding element type), in addition to the existing scalar overloads. [#109398](https://github.com/ClickHouse/ClickHouse/pull/109398) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The MySQL-style format specifier `%f` in `parseDateTime` / `parseDateTime64` (and their `OrZero` / `OrNull` variants) now accepts between 1 and 6 fractional digits, interpreted as left-aligned microseconds like MySQL's `STR_TO_DATE`, instead of requiring exactly 6. Also fixed misaligned `PrettyCompact` tables in the built-in function documentation examples. [#109421](https://github.com/ClickHouse/ClickHouse/pull/109421) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The `basic` statistics `NullCount` sub-statistic is renamed to `DefaultCount`: instead of counting `NULL` rows of a `Nullable` column, `build` now counts rows equal to the column type's intrinsic default via `IColumn::getNumberOfDefaultRows`. [#109977](https://github.com/ClickHouse/ClickHouse/pull/109977) ([hanfei1991](https://github.com/hanfei1991)). * Reduced the binary size by \~10.5 MB by executing comparison and arithmetic operations on rarely used mixed type pairs (`Decimal` vs integer of a different width, and pairs involving `Int128`/`UInt128`/`Int256`/`UInt256`) via a conversion to a common type instead of a dedicated compiled kernel for every combination of types. Same-type pairs, commonly used pairs, and the memory-bound `plus`/`minus`/`multiply` keep their dedicated kernels; results, result types and exceptional cases are unchanged. [#110131](https://github.com/ClickHouse/ClickHouse/pull/110131) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Reading from a `SQLite` table engine or `sqlite` table function no longer busy-spins a full CPU core when the SQLite database is locked by another connection. The read now idles while waiting for the lock and stays cancellable. [#110248](https://github.com/ClickHouse/ClickHouse/pull/110248) ([groeneai](https://github.com/groeneai)). * Round elapsed time, rate, and ratio values in log and exception messages to three digits, so numbers like `1.345844286 sec.` are no longer printed at full precision. [#110277](https://github.com/ClickHouse/ClickHouse/pull/110277) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `MergeTree` reads of `String` columns with a separate size stream after leading rows are skipped, which could return wrong data or read past bounds from the substreams cache. Also fixed possible wrong data when concurrent or repeated reads access a `JSON` sub-object stored with map shared-data serialization. [#110471](https://github.com/ClickHouse/ClickHouse/pull/110471) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Support the `SETTINGS` clause for the `PostgreSQL` table engine and the `postgresql` table function (for example `SETTINGS postgresql_connection_pool_size = 50`), bringing feature parity with the `MySQL` engine. [#110614](https://github.com/ClickHouse/ClickHouse/pull/110614) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Functions `bitmaskToArray` and `bitmaskToList` now support `(U)Int128` and `(U)Int256` arguments. Function `bitPositionsToArray` now runs in time proportional to the number of set bits for big integer arguments, instead of the position of the highest set bit. [#110743](https://github.com/ClickHouse/ClickHouse/pull/110743) ([raimannma](https://github.com/raimannma)). * Fixed GCD codec handling of signed fixed-width values with negative numbers. Previously it computed the divisor from signed values directly, which could miss the common divisor and significantly reduce compression efficiency for signed integer and decimal columns. It now computes the divisor from magnitudes for signed types while preserving existing behavior for unsigned types and decompression. [#113798](https://github.com/ClickHouse/ClickHouse/pull/113798) ([kshcherbatov](https://github.com/kshcherbatov)). * A window function with `PARTITION BY` now works under `make_distributed_plan`. When the plan shape allows, the window runs in parallel across buckets that each receive complete partitions. [#114111](https://github.com/ClickHouse/ClickHouse/pull/114111) ([VighneshPath](https://github.com/VighneshPath)). * 64-bit hash function is used now for external nullable fixed-width aggregation methods (avoids collision disaster for very high cardinality aggregation). [#114210](https://github.com/ClickHouse/ClickHouse/pull/114210) ([nickitat](https://github.com/nickitat)). * `x IN (subquery)` over a `LowCardinality` column now returns `LowCardinality(UInt8)` — the same type as `x IN (literal list)`. Previously the two forms returned different types, and distributed plans with such an `IN` needed a special case in the plan type check. [#114229](https://github.com/ClickHouse/ClickHouse/pull/114229) ([davenger](https://github.com/davenger)). * Support parsing integers that exceed the 64-bit range in the `JSON` data type, `JSONExtract`, and `isValidJSON`. Such values are read into wide integer types such as `Int128`/`UInt256` instead of causing the whole document to be rejected. [#114379](https://github.com/ClickHouse/ClickHouse/pull/114379) ([Avogar](https://github.com/Avogar)). * Added `keyword` as an alias for text index tokenizer `array` for improved compatibilty with OpenSearch, Elasticsearch, SolR, and Lucene. [#115507](https://github.com/ClickHouse/ClickHouse/pull/115507) ([rschu1ze](https://github.com/rschu1ze)). * Functions `toDateOrNull`, `toDateTimeOrNull` and `toDateTime64OrNull` now accept integer arguments of all native integer types (interpreted the same way as by `toDate`, `toDateTime` and `toDateTime64`, with an optional timezone argument), returning `NULL` for values out of range of the result type. For example, `toDateTimeOrNull(1583851242, 'Asia/Shanghai')` returns `2020-03-10 22:40:42` and `toDateTimeOrNull(4294967296)` returns `NULL`. [#79791](https://github.com/ClickHouse/ClickHouse/pull/79791) ([jitendra1411](https://github.com/jitendra1411)).

Storage, data lakes, and object storage

* Added opt-in Prometheus metrics for filesystem cache eviction activity (`filesystem_cache_evictions_total`, `filesystem_cache_evicted_bytes_total`, `filesystem_cache_evicted_segment_hits`, `filesystem_cache_evicted_segment_size_bytes`, and their per-user variants labeled with `user_id`), exposed per cache via `system.dimensional_metrics`, `system.histogram_metrics`, and the Prometheus endpoint. They are controlled by the cache disk-config settings `expose_prometheus_eviction_metrics` and `expose_prometheus_eviction_metrics_per_user` (both off by default), which can be toggled at runtime via `SYSTEM RELOAD CONFIG`. [#105020](https://github.com/ClickHouse/ClickHouse/pull/105020) ([sacheendra](https://github.com/sacheendra)). * `Alias` engine now supports parallel replicas read when the target table is a `MergeTree` family engine. [#107830](https://github.com/ClickHouse/ClickHouse/pull/107830) ([nauu](https://github.com/nauu)). * Limit the number of concurrently staged parts in Azure Blob Storage read-then-write copy (used by backups when native copy is unavailable) to max\_inflight\_parts\_for\_one\_file, preventing excessive memory usage when many large files are copied in parallel. [#108232](https://github.com/ClickHouse/ClickHouse/pull/108232) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Support `_etag` virtual column for HDFS storage [#108255](https://github.com/ClickHouse/ClickHouse/pull/108255) ([zhangyifan27](https://github.com/zhangyifan27)). * Added `settings` and `engine_settings` columns to `system.backups` and `system.backup_log`. `settings` exposes the backup/restore-specific settings requested for an operation (e.g. `allow_s3_native_copy`, `deduplicate_files`, `structure_only`), and `engine_settings` exposes the settings effectively used by the backup engine's reader/writer (e.g. the S3 request settings such as `allow_native_copy`, which may differ from what was requested after merging endpoint configuration). This makes it possible to see which settings a `BACKUP`/`RESTORE` operation actually ran with. [#108334](https://github.com/ClickHouse/ClickHouse/pull/108334) ([jkartseva](https://github.com/jkartseva)). * Added `system.s3_queue_metadata` and `system.azure_queue_metadata` for inspecting the Keeper state of registered `S3Queue` and `AzureQueue` tables, including processed, processing, and failed-file counts. [#108522](https://github.com/ClickHouse/ClickHouse/pull/108522) ([kssenii](https://github.com/kssenii)). * Reduced the memory usage of the text index header cache by 2-2.5 times. More headers of data parts fit into the cache (setting `text_index_header_cache_size`), reducing disk reads for text search queries. [#109332](https://github.com/ClickHouse/ClickHouse/pull/109332) ([CurtizJ](https://github.com/CurtizJ)). * Added MySQL-compatibility columns to `INFORMATION_SCHEMA` views: `SCHEMATA.DEFAULT_COLLATION_NAME` and `DEFAULT_ENCRYPTION`; `TABLES.ENGINE` and related MySQL columns; and `COLUMNS.COLUMN_KEY`, `PRIVILEGES`, `GENERATION_EXPRESSION`, `SRS_ID`. This lets MySQL-aware clients run their catalog introspection queries without hitting `UNKNOWN_IDENTIFIER`. [#109351](https://github.com/ClickHouse/ClickHouse/pull/109351) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Reduced memory usage of `MergeTree` tables with many data parts by sharing the schema-derived per-part metadata (column list, column descriptions, serializations, and columns substreams) across parts of a table instead of copying it into every part. The effect is largest for wide tables with many small parts. [#109816](https://github.com/ClickHouse/ClickHouse/pull/109816) ([Algunenano](https://github.com/Algunenano)). * Added a new setting `use_projection_index_in_read_pools` (disabled by default). When enabled, mark ranges that are fully filtered out by a projection index are dropped inside `MergeTree` read pools before read tasks are created for them, instead of being skipped granule by granule during reading. This avoids creating read tasks and setting up readers for data that will not be read, and keeps read task sizes measured in surviving marks. [#110291](https://github.com/ClickHouse/ClickHouse/pull/110291) ([KochetovNicolai](https://github.com/KochetovNicolai)). * Reduced the verbosity of the per-part primary-key binary-search diagnostics in `MergeTree` by lowering them from trace logging to test logging. [#110324](https://github.com/ClickHouse/ClickHouse/pull/110324) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed spurious reconnections of pooled HTTP keep-alive connections over TLS (HTTPS, S3). The stale-connection check used `poll` on the socket, which misreports a live secure connection carrying an unread TLS post-handshake record (a session ticket or `KeyUpdate`) as closed; it now uses a non-blocking `MSG_PEEK` on plain sockets and `SSL_peek`/`SSL_has_pending` on TLS sockets, which also correctly detects an orderly TLS shutdown and a TLS record that has only partially arrived. [#110402](https://github.com/ClickHouse/ClickHouse/pull/110402) ([alexey-milovidov](https://github.com/alexey-milovidov)). * OneLake catalogs can now use OAuth refresh tokens to renew access tokens. [#110413](https://github.com/ClickHouse/ClickHouse/pull/110413) ([alesapin](https://github.com/alesapin)). * Added the `DuplicationDataHashComputations` ProfileEvent, which counts the column-wise data-hash computations performed while deduplicating INSERTed blocks to `*MergeTree` tables. [#111173](https://github.com/ClickHouse/ClickHouse/pull/111173) ([valerypetrov](https://github.com/valerypetrov)). * Write the UNIQUE KEY index SST directly into the part through the storage abstraction, instead of staging it in a local temporary file. [#111189](https://github.com/ClickHouse/ClickHouse/pull/111189) ([JingYanchao](https://github.com/JingYanchao)). * Support concurrent writes to Iceberg tables on a local disk: `LocalObjectStorage` now implements conditional writes, so the compare-and-swap that publishes a new snapshot through `version-hint.text` works there and concurrent writers cannot lose each other's updates. [#112556](https://github.com/ClickHouse/ClickHouse/pull/112556) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The bucket-region cache of the `S3` client now also works for data lake catalogs; previously every request to them resolved the region again. [#113330](https://github.com/ClickHouse/ClickHouse/pull/113330) ([scanhex12](https://github.com/scanhex12)). * Azure client-side network failures (connection reset/refused, DNS and TLS errors) are now retried as transport errors instead of surfacing as a synthetic `500`, matching the S3 client. [#115269](https://github.com/ClickHouse/ClickHouse/pull/115269) ([arsenmuk](https://github.com/arsenmuk)).

Formats and ingestion

* Allow snappy compression in the HTTP interface (`Accept-Encoding: snappy`) and add the `snappy_mode` setting to choose the Hadoop snappy block format or the snappy framing format for generic `file`/`url` snappy I/O. [#100752](https://github.com/ClickHouse/ClickHouse/pull/100752) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Added a setting `input_format_csv_missing_nullable_as_empty_string` (disabled by default). When enabled, a missing value of `Nullable(String)` in CSV input is read as an empty `String` instead of `NULL`, regardless of `input_format_csv_empty_as_default`. [#107577](https://github.com/ClickHouse/ClickHouse/pull/107577) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Processing protobuf messages with `input_format_protobuf_oneof_presence` we do not require exact match of tags to allow schema changes. [#109174](https://github.com/ClickHouse/ClickHouse/pull/109174) ([ilejn](https://github.com/ilejn)). * Removed the legacy Apache Arrow-based ORC reader. The ORC input format now always uses the native ClickHouse decoder (previously the default). The `input_format_orc_use_fast_decoder` setting is obsolete (still accepted, but has no effect). [#110074](https://github.com/ClickHouse/ClickHouse/pull/110074) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The Hive engine now reads ORC file metadata (min/max indexes, row counts) with ClickHouse's native ORC reader instead of the Apache Arrow ORC adapter. [#110086](https://github.com/ClickHouse/ClickHouse/pull/110086) ([alexey-milovidov](https://github.com/alexey-milovidov)).

Security, access, and observability

* Added a new column `skipping_indices_types` to the `system.tables` table. It contains the distinct types of data-skipping indices defined for each table. [#106388](https://github.com/ClickHouse/ClickHouse/pull/106388) ([CurtizJ](https://github.com/CurtizJ)). * The automatic value of `max_threads` and similar settings is now shown in `system.settings` as `auto(8)` instead of `'auto(8)'`; the surrounding single quotes were a long-standing artifact baked into the value. Cross-version compatibility is preserved: the legacy quoted form is still accepted when parsing settings received from older servers. [#108657](https://github.com/ClickHouse/ClickHouse/pull/108657) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Render an empty setting default value as *empty string* in `system.documentation` instead of empty backticks. [#108708](https://github.com/ClickHouse/ClickHouse/pull/108708) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The documentation of a setting in `system.documentation` — shown by the built-in `/docs` page and by the `help` command — now includes the history of the changes of its default value: the version in which the setting was introduced and every later change of its default, with the previous value, the new value and the reason for the change. [#112177](https://github.com/ClickHouse/ClickHouse/pull/112177) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The documentation of all SQL statements (e.g. name, syntax, description, examples) can now be retrieved from system table `system.statements`. [#115341](https://github.com/ClickHouse/ClickHouse/pull/115341) ([rschu1ze](https://github.com/rschu1ze)).

Settings and configuration

* Added a new server setting `memory_worker_rss_speculative_reserve_ratio` (default `1.0`, or `0` in sanitizer builds) which makes the global memory tracker speculatively reserve `ratio * min(resident - previous_resident, resident - tracked)` on top of the observed RSS each `MemoryWorker` tick. This biases the tracker upward when RSS growth is outpacing the tracker bookkeeping between samples, so allocations get `MEMORY_LIMIT_EXCEEDED` earlier and the kernel OOM-killer is less likely to fire first. Set the ratio to `0` to disable speculation. Sanitizer builds default to `0` because the `resident - tracked` gap is dominated by sanitizer shadow memory there. [#104976](https://github.com/ClickHouse/ClickHouse/pull/104976) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The text index lazy posting-list apply mode is no longer experimental and can be selected with `text_index_posting_list_apply_mode = 'lazy'` without `allow_experimental_text_index_lazy_apply`. The density-threshold setting was renamed from `text_index_density_threshold` to `text_index_lazy_intersection_density_threshold`. [#108814](https://github.com/ClickHouse/ClickHouse/pull/108814) ([CurtizJ](https://github.com/CurtizJ)). * The `merge_selector_enable_heuristic_to_lower_max_parts_to_merge_at_once` setting is now enabled by default: the merge selector automatically lowers the maximum number of parts to merge at once based on how full the partition is. [#110726](https://github.com/ClickHouse/ClickHouse/pull/110726) ([Michicosun](https://github.com/Michicosun)). * Accept PostgreSQL cleanup commands `RESET`, `UNLISTEN`, and `DISCARD` as no-ops in the PostgreSQL wire protocol instead of failing them with a syntax error. This improves compatibility with drivers such as Skunk that send `RESET ALL` and `UNLISTEN *` during connection setup and cleanup. [#110780](https://github.com/ClickHouse/ClickHouse/pull/110780) ([alexey-milovidov](https://github.com/alexey-milovidov)). * AI text functions (`aiGenerate`, `aiClassify`, `aiExtract`, `aiTranslate`) now reject truncated or otherwise incomplete provider responses (e.g. when the model hits the `max_tokens` limit) instead of silently returning partial output. Behavior follows the `ai_function_throw_on_error` setting. [#111830](https://github.com/ClickHouse/ClickHouse/pull/111830) ([george-larionov](https://github.com/george-larionov)). * Improve reduced-precision `L2DistanceTransposedQuantized`, `cosineDistanceTransposedQuantized`, and `dotProductTransposedQuantized` by reconstructing `p < 8QBit(Int8)` codes with Gaussian conditional-mean prefix centroids. Full precision (`p = 8`) remains bit-exact; reduced-precision approximate distances may change, and recall/latency gains are workload-dependent. [#111867](https://github.com/ClickHouse/ClickHouse/pull/111867) ([skuznetsov](https://github.com/skuznetsov)). * Make `write_snapshot_version` setting hot re-loadable. [#112150](https://github.com/ClickHouse/ClickHouse/pull/112150) ([alesapin](https://github.com/alesapin)). * Added observability for the stacks of fibers, which are used for asynchronous communication with remote replicas: profile events `FiberStackAllocs`, `FiberStackAllocBytes`, `FiberStackAllocNanoseconds`, `FiberStackFreeNanoseconds`, and metrics `FiberStacks`, `FiberStackBytes`. [#114541](https://github.com/ClickHouse/ClickHouse/pull/114541) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The Prometheus remote-write protocol now supports asynchronous inserts: data from multiple concurrent remote-write requests is batched before forming parts, following the `async_insert` setting (enabled by default; add `async_insert=0` to the handler URL to keep the synchronous behavior). A request is acknowledged only after the data is flushed to all inner tables of the target `TimeSeries` table. [#115688](https://github.com/ClickHouse/ClickHouse/pull/115688) ([nikitamikhaylov](https://github.com/nikitamikhaylov)).

Other improvements

* Added two settings, `shrink_over_allocated_columns_min_waste_ratio` and `shrink_over_allocated_columns_min_waste_bytes`, that make INSERTs shrink over-allocated columns (whose reserved memory exceeds used memory due to power-of-two growth of variable-length columns) to fit before materialization and part writing, reducing peak memory usage. Disabled by default.. [#112161](https://github.com/ClickHouse/ClickHouse/pull/112161) ([Avogar](https://github.com/Avogar)). * Skip data reading step entirely for `TTLDrop` merges to avoid excessive memory usage, especially on read/prefetch buffers allocations. [#105859](https://github.com/ClickHouse/ClickHouse/pull/105859) ([Avogar](https://github.com/Avogar)). * Added support for `BFloat16` in `dotProduct` and improved dot product performance by batching the SIMD path. [#106569](https://github.com/ClickHouse/ClickHouse/pull/106569) ([nickitat](https://github.com/nickitat)). * Made `max_named_collection_num_to_throw`, `max_table_num_to_throw`, `max_replicated_table_num_to_throw`, `max_view_num_to_throw`, `max_dictionary_num_to_throw`, and `max_database_num_to_throw` changeable without a server restart. [#106821](https://github.com/ClickHouse/ClickHouse/pull/106821) ([UberDever](https://github.com/UberDever)). * Writing to `DeltaLake` tables, controlled by `allow_experimental_delta_lake_writes`, was promoted to Beta. [#107034](https://github.com/ClickHouse/ClickHouse/pull/107034) ([kssenii](https://github.com/kssenii)). * Added the `uniq_v2` column statistics type, a lightweight alternative to `uniq` based on the `uniqCombined64` sketch. [#107863](https://github.com/ClickHouse/ClickHouse/pull/107863) ([hanfei1991](https://github.com/hanfei1991)). * Added ProfileEvents `DistributedPlanRemoteTasks`, `DistributedPlanLocalExecution`, and `DistributedPlanHostsUsed` to observe make\_distributed\_plan execution. [#107985](https://github.com/ClickHouse/ClickHouse/pull/107985) ([shankar-iyer](https://github.com/shankar-iyer)). * Requests to REST data lake catalogs such as OneLake now include a ClickHouse `User-Agent` header. [#108117](https://github.com/ClickHouse/ClickHouse/pull/108117) ([scanhex12](https://github.com/scanhex12)). * `MVTEncodeGeom` now snaps geometry to the integer pixel grid before clipping and clips polygons with `wagyu`, so clipped output is valid (self-intersecting rings are repaired) and edge-aligned, matching PostGIS `ST_AsMVTGeom`. [#108248](https://github.com/ClickHouse/ClickHouse/pull/108248) ([saarthak2002](https://github.com/saarthak2002)). * Enabled all three text index caches globally - previously, they were only enabled within queries. Also, the posting lists cache size is now zero (which effectively disables it again). This is because posting lists are large and caching them is too costly. [#108274](https://github.com/ClickHouse/ClickHouse/pull/108274) ([rschu1ze](https://github.com/rschu1ze)). * Start the Prometheus endpoint (for metrics-only configurations) and asynchronous metrics collection before tables are loaded, so that metrics are observable during the potentially long metadata loading phase. [#108402](https://github.com/ClickHouse/ClickHouse/pull/108402) ([cwurm](https://github.com/cwurm)). * Support BFloat16 in binary math functions [#108442](https://github.com/ClickHouse/ClickHouse/pull/108442) ([zhangyifan27](https://github.com/zhangyifan27)). * The transposed vector distance functions `L2DistanceTransposed`, `cosineDistanceTransposed` and `dotProductTransposed` now apply the partial bit-plane read optimization to `Nullable(QBit)` columns, reading only the requested bit planes instead of the whole column. [#109358](https://github.com/ClickHouse/ClickHouse/pull/109358) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Support `arraySum`, `arrayAvg`, and `arrayProduct` for arrays of `BFloat16`. [#109420](https://github.com/ClickHouse/ClickHouse/pull/109420) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Functions `L1Normalize`, `L2Normalize`, `LinfNormalize` and `LpNormalize` (and their aliases) now work for arrays, not only for tuples, consistently with `L2Norm` and the distance functions. [#110052](https://github.com/ClickHouse/ClickHouse/pull/110052) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Pooled connections are no longer pinged before each use. This removes a `Ping`-`Pong` round trip that was added to every reused connection, reducing the latency of distributed queries and of `clickhouse-benchmark`. A stale pooled connection is detected with a zero-timeout poll (a non-blocking check that adds no round trip) and recovered by reconnecting. [#110068](https://github.com/ClickHouse/ClickHouse/pull/110068) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Send big list parts for distributed index analysis as scalars (avoids possible failures due to max\_ast\_elements/max\_query\_size and removes AST parsing overhead) [#110419](https://github.com/ClickHouse/ClickHouse/pull/110419) ([azat](https://github.com/azat)). * Clicking the cloud logo in the Playground now opens the Clickhouse website in a new tab so you don't loose progress while queries are running. [#110422](https://github.com/ClickHouse/ClickHouse/pull/110422) ([williamhatcher](https://github.com/williamhatcher)). * Change default `auto_statistics_types` from `basic, uniq` to `basic, uniq_v2` [#110878](https://github.com/ClickHouse/ClickHouse/pull/110878) ([hanfei1991](https://github.com/hanfei1991)). * When an `RWLockImpl::getLock()` re-entrancy assertion fires (`RWLock is already locked in exclusive mode` / `Cannot acquire exclusive lock while RWLock is already locked`), the error message now names the other `query_id`s currently owning the lock and how many locks the requesting `query_id` already holds, to make such lock-ordering errors diagnosable. [#110971](https://github.com/ClickHouse/ClickHouse/pull/110971) ([groeneai](https://github.com/groeneai)). * `randomHadamardTransform` now computes an exact, length-preserving transform for any vector length whose largest odd factor is at most 64 (for example `3584 = 512 * 7`, common in embedding models), extending the previous `2^N`, `2^k * {12, 20}`, and `2^k * 9` families. A full transform of a length that cannot be represented exactly now raises an exception instead of silently zero-padding to a longer vector; pass `output_dims` to compute a truncated projection of an arbitrary length. [#111006](https://github.com/ClickHouse/ClickHouse/pull/111006) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Return HTTP code `403 Forbidden` (instead of `500 Internal Server Error`) for `ACCESS_DENIED` exceptions over the HTTP interface. [#111043](https://github.com/ClickHouse/ClickHouse/pull/111043) ([Schum-io](https://github.com/Schum-io)). * `UTMToGeo` now accepts the MGRS latitude band letter returned by `geoToUTM` as its fourth argument (in addition to the integer hemisphere flag), so a `geoToUTM` result round-trips through `UTMToGeo` directly. [#111521](https://github.com/ClickHouse/ClickHouse/pull/111521) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `MaterializedPostgreSQL` now preserves unchanged PostgreSQL `TOAST` values during updates instead of replacing them with default values. [#111552](https://github.com/ClickHouse/ClickHouse/pull/111552) ([OrpheusAgent](https://github.com/OrpheusAgent)). * Added the `text_index_max_processed_tokens_before_flush` and `text_index_max_memory_usage_before_flush` settings to control when text index builders flush temporary segments. [#111573](https://github.com/ClickHouse/ClickHouse/pull/111573) ([rorylshanks](https://github.com/rorylshanks)). * Add STREAM UNORDERED modifier: skip the per-snapshot commit-order sort [#111794](https://github.com/ClickHouse/ClickHouse/pull/111794) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Now `make_distributed_plan=1` will disable not yet supported features.. [#112463](https://github.com/ClickHouse/ClickHouse/pull/112463) ([alesapin](https://github.com/alesapin)). * Do not write ANSI escape sequences into INTO OUTFILE when stdout is a tty [#113252](https://github.com/ClickHouse/ClickHouse/pull/113252) ([azat](https://github.com/azat)). * Renamed `allow_experimental_delta_kernel_rs` to `allow_delta_kernel_rs`. The old name is kept as an alias. [#113476](https://github.com/ClickHouse/ClickHouse/pull/113476) ([kssenii](https://github.com/kssenii)). * Add settings `enable_alp_codec`, `enable_sz3_codec`, `enable_zxc_codec`, and `enable_quantized_codec` to enable each experimental compression codec individually. [#113824](https://github.com/ClickHouse/ClickHouse/pull/113824) ([rienath](https://github.com/rienath)). * Report a stack overflow instead of dying silently: the deadly signal handlers now run on the alternative signal stack, so `SIGSEGV`, `SIGABRT`, `SIGILL`, `SIGBUS`, `SIGSYS`, `SIGFPE` and `SIGTRAP` produce a stack trace even when the faulting thread has exhausted its stack. [#113927](https://github.com/ClickHouse/ClickHouse/pull/113927) ([groeneai](https://github.com/groeneai)). * Support the Prometheus HTTP API lookback\_delta parameter for instant and range queries. [#113971](https://github.com/ClickHouse/ClickHouse/pull/113971) ([fallintoplace](https://github.com/fallintoplace)). * QueryRunner tables now start worker threads on demand and release them once idle, instead of occupying threads for the table's whole lifetime. [#114522](https://github.com/ClickHouse/ClickHouse/pull/114522) ([mstetsyuk](https://github.com/mstetsyuk)). * With this change, while using the WHATIF indices, the projection-served baseline reports per-candidate `not_applicable` instead of failing the statement. [#114846](https://github.com/ClickHouse/ClickHouse/pull/114846) ([yariks5s](https://github.com/yariks5s)). * Optimize AND chains with multiple comparison conditions on the same expression: detect contradictions (e.g., `a < 3 AND a > 5` → `false`), prune redundant conditions (e.g., `a = 3 AND a < 5` → `a = 3`). [#99736](https://github.com/ClickHouse/ClickHouse/pull/99736) ([wudidapaopao](https://github.com/wudidapaopao)).

Bug Fixes

MergeTree, replication, and storage fixes

* Fixed a race between `ALTER TABLE ... RENAME COLUMN` and a concurrent `OPTIMIZE TABLE ... FINAL` (or any background merge) in `MergeTree` that could silently replace the renamed column's data with default values. The race window between publishing the new in-memory metadata and registering the rename mutation in `current_mutations_by_version` is now closed by holding `currently_processing_in_background_mutex` across both operations. [#104822](https://github.com/ClickHouse/ClickHouse/pull/104822) ([groeneai](https://github.com/groeneai)). * Fixed incorrect query results when using GROUP BY with ORDER BY and `arrayJoin` in the projection. [#101775](https://github.com/ClickHouse/ClickHouse/pull/101775) ([Onyx2406](https://github.com/Onyx2406)). * Invalid Azure upload-size settings (for example `azure_min_upload_part_size = 0` or `azure_max_blocks_in_multipart_upload = 0`) are now rejected early with a clear error instead of triggering an internal exception (`second_size > 0`) deep in the write path. This applies to both query settings and Azure disk configuration. [#103394](https://github.com/ClickHouse/ClickHouse/pull/103394) ([statxc](https://github.com/statxc)). * Throw `BAD_ARGUMENTS` instead of `LOGICAL_ERROR` when `ALTER TABLE ... MOVE/REPLACE/ATTACH PARTITION` is run between two `MergeTree` tables with incompatible granularity settings (one adaptive, one non-adaptive). [#103881](https://github.com/ClickHouse/ClickHouse/pull/103881) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` "Expected one block from input stream" thrown by `KILL QUERY` / `KILL MUTATION` / `KILL PART_MOVE_TO_SHARD` / `KILL TRANSACTION` when their `WHERE` clause contains a per-row subquery, or when `max_block_size` is small enough that the internal `SELECT` over the relevant `system.*` table emits more than one block. [#104927](https://github.com/ClickHouse/ClickHouse/pull/104927) ([groeneai](https://github.com/groeneai)). * Fix a leak in Refreshable Materialized Views (`MATERIALIZED VIEW ... REFRESH ...`) where the temporary inner table that the refresh task rotates out via `EXCHANGE TABLES` could not be dropped if it was larger than the server's `max_table_size_to_drop`. Every subsequent refresh would create yet another temporary inner table while still failing to clean up the previous ones, growing the view's data directory until the disk was exhausted. The refresh task's internal drop now bypasses `max_table_size_to_drop` and `max_partition_size_to_drop`; those safety nets still apply to user-issued `DROP TABLE` of the view itself. [#105106](https://github.com/ClickHouse/ClickHouse/pull/105106) ([groeneai](https://github.com/groeneai)). * Fix an unexpected `DATA_TYPE_CANNOT_BE_USED_IN_KEY` error on `ALTER` queries that cannot change the sorting key (settings, comments, codecs, adding a non-key column, etc.) for `MergeTree` tables that have a `SimpleAggregateFunction` (or another type allowed only with `allow_suspicious_primary_key = 1`) in the sorting key. [#105111](https://github.com/ClickHouse/ClickHouse/pull/105111) ([groeneai](https://github.com/groeneai)). * `OPTIMIZE ... DRY RUN` interrupted by a query timeout (`max_execution_time` with `timeout_overflow_mode = 'break'`) now returns `TIMEOUT_EXCEEDED` instead of a logical error about `rows_sources` (which aborted the server in debug and sanitizer builds). [#107114](https://github.com/ClickHouse/ClickHouse/pull/107114) ([groeneai](https://github.com/groeneai)). * Fixed `NOT_FOUND_COLUMN_IN_BLOCK` when reading hive-partitioned files with `use_hive_partitioning=1` and a `WHERE`/`PREWHERE` clause that filters a real (non-virtual) column while a hive partition column is also selected. [#107505](https://github.com/ClickHouse/ClickHouse/pull/107505) ([groeneai](https://github.com/groeneai)). * Fix wrong query results from incorrect primary key and partition pruning when `toStartOfDay`, `toStartOfISOYear`, `toDaysSinceYearZero`, `toRelativeSecondNum`, `toRelativeMinuteNum`, `toRelativeHourNum`, `toRelativeWeekNum`, `toRelativeDayNum`, `toMonthNumSinceEpoch`, or `toYearNumSinceEpoch` is applied to a `Date32` key column containing dates outside the range the function can represent (in particular dates before `1970-01-01`). These functions report themselves as monotonic to the primary index, but previously wrapped around for such arguments, so the index could prune granules that actually contained matching rows. They now saturate at the bounds of their result type and stay monotonic over the whole `Date32` range, so pruning is correct. [#108018](https://github.com/ClickHouse/ClickHouse/pull/108018) ([nihalzp](https://github.com/nihalzp)). * Fixed `CREATE TABLE dst AS src SETTINGS ...` (and similar variants with `ORDER BY`/`PARTITION BY` but without an explicit `ENGINE`) silently dropping the source table's engine, storage clauses (such as table `TTL` and keys) and settings. The engine and the non-overridden storage clauses are now inherited from the source table, and the specified settings are merged on top of the source table's settings. [#108085](https://github.com/ClickHouse/ClickHouse/pull/108085) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `INSERT` into `MergeTree` tables failing with `filesystem error: in rename: Permission denied` on filesystems backed by Windows (WSL, CIFS/SMB, Docker Desktop bind mounts), caused by the part writer leaving file descriptors open across the part-directory rename. [#108089](https://github.com/ClickHouse/ClickHouse/pull/108089) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a wedge on a MergeTree table attached with a legacy `hypothesis` skip index (a removed index type kept only for `ATTACH` compatibility). Previously a lightweight `DELETE`, `OPTIMIZE`, a plain `INSERT`, or a filtered `SELECT` on such a table failed with `ILLEGAL_INDEX` ("Index of type 'hypothesis' is no longer supported"), and the `DELETE` mutation retried forever. The dead index is now treated as inert: it is carried forward untouched during merge/mutation, skipped on insert and query planning, and can still be dropped with `ALTER TABLE ... DROP INDEX`. [#108217](https://github.com/ClickHouse/ClickHouse/pull/108217) ([groeneai](https://github.com/groeneai)). * Fix `NOT_FOUND_COLUMN_IN_BLOCK` error (e.g. `Column _part not found`) when selecting virtual columns from a `MergeTree` table under parallel replicas, including via `SELECT *` with `asterisk_include_virtual_columns = 1`. [#108451](https://github.com/ClickHouse/ClickHouse/pull/108451) ([groeneai](https://github.com/groeneai)). * Fix `ATTACH PARTITION ... FROM` rejecting tables whose primary keys are equivalent but declared differently (one explicit `PRIMARY KEY`, the other implicit from `ORDER BY`). [#108590](https://github.com/ClickHouse/ClickHouse/pull/108590) ([jordiori](https://github.com/jordiori)). * `join_any_take_last_row` is now respected by all supported hash-based join paths, including joins that use automatic spilling to disk. [#108936](https://github.com/ClickHouse/ClickHouse/pull/108936) ([antaljanosbenjamin](https://github.com/antaljanosbenjamin)). * Fix `NOT_FOUND_COLUMN_IN_BLOCK` error when `query_plan_optimize_lazy_final` is enabled together with a `WHERE` filter partially pushed to `PREWHERE` (via `optimize_move_to_prewhere_if_final`). The lazy FINAL optimization no longer requests prewhere-computed predicate columns from storage. [#109073](https://github.com/ClickHouse/ClickHouse/pull/109073) ([groeneai](https://github.com/groeneai)). * Fixed `ALTER TABLE ... ADD PROJECTION` over a table-level `ALIAS` column leaving the table unusable (subsequent `INSERT` failed with `UNKNOWN_IDENTIFIER`, "Missing columns") when the session had `optimize_respect_aliases = 0`. [#109091](https://github.com/ClickHouse/ClickHouse/pull/109091) ([groeneai](https://github.com/groeneai)). * Fix a null-pointer assertion (`px != 0` in `TreeRewriter::analyzeSelect`, aborting the server in debug and sanitizer builds) when running `ORDER BY ALL` (or `GROUP BY ALL`) over a `Merge`-engine table joined with another table under the old analyzer (`allow_experimental_analyzer = 0`). [#109139](https://github.com/ClickHouse/ClickHouse/pull/109139) ([groeneai](https://github.com/groeneai)). * Fixed a wrong result of correlated `EXISTS` and scalar subqueries when the correlation appears only in the subquery projection together with a non-correlated `WHERE` clause (the subquery could evaluate to false / `NULL` for every row). Fixes [https://github.com/ClickHouse/ClickHouse/issues/105760](https://github.com/ClickHouse/ClickHouse/issues/105760). [#109186](https://github.com/ClickHouse/ClickHouse/pull/109186) ([novikd](https://github.com/novikd)). * Fixed a bug where merging a `MergeTree` table that has projections while the `enable_block_number_column` or `enable_block_offset_column` setting is enabled produced projection parts containing a spurious `_block_number`/`_block_offset` column. The merged projection part then no longer matched the projection definition and the insert-produced projection parts, so `CHECK TABLE` and `OPTIMIZE ... DRY RUN` reported it as corrupted (`CORRUPTED_DATA`). [#109284](https://github.com/ClickHouse/ClickHouse/pull/109284) ([groeneai](https://github.com/groeneai)). * Fix `UNKNOWN_IDENTIFIER` error when running `ALTER TABLE ... RENAME/ADD/MODIFY COLUMN` on a MergeTree table that has an implicit min-max index over the persistent virtual columns `_block_number` or `_block_offset` (enabled by `add_minmax_index_for_block_number_column` / `add_minmax_index_for_block_offset_column`). [#109428](https://github.com/ClickHouse/ClickHouse/pull/109428) ([groeneai](https://github.com/groeneai)). * Fix skip indices being dropped from a data part (or `CHECK TABLE` failing with `UNEXPECTED_FILE_IN_DATA_PART`) after a full-part-rewrite mutation, such as `ALTER TABLE ... DROP COLUMN` of a `MATERIALIZED` column, on a `Wide` part. [#109616](https://github.com/ClickHouse/ClickHouse/pull/109616) ([groeneai](https://github.com/groeneai)). * Fixed stale skip indices (`text`, `bloom_filter`, etc.) and projections after materializing lightweight updates with `ALTER TABLE ... APPLY PATCHES`. Previously the index and projection files that depend on a patch-updated column were left unchanged, so queries using them could return wrong results (e.g. `hasToken` missing an updated row, or a projection returning stale aggregates once the spent patch part was removed). [#109709](https://github.com/ClickHouse/ClickHouse/pull/109709) ([groeneai](https://github.com/groeneai)). * Fixes a mutation that references the `_sample_factor` virtual column failing mid-execution with `Unexpected const virtual column: _sample_factor` and leaving the mutation stuck. A mutation now reads it as `1`, the same value a query without `SAMPLE` returns. [#109813](https://github.com/ClickHouse/ClickHouse/pull/109813) ([groeneai](https://github.com/groeneai)). * Fixed two errors raised when a query with `GROUP BY GROUPING SETS` (or a scalar or mutation subquery containing one) runs with `force_aggregation_in_order = 1`: `Trying to get name of not a column: ExpressionList` under the old analyzer (`enable_analyzer = 0`), and `Memory bound merging of aggregated results is not supported for grouping sets.` for a distributed query with `enable_memory_bound_merging_of_aggregation_results = 1` under the analyzer. [#109862](https://github.com/ClickHouse/ClickHouse/pull/109862) ([groeneai](https://github.com/groeneai)). * Fix three issues with `use_constant_folding_in_index_analysis`: a logical error `Invalid partition key size` that could abort a `SELECT` using a normal projection when the table sets `part_minmax_index_columns = 'with_block_number_offset'`; wrong results (silently dropped rows) for a filter on a modulo partition key such as `PARTITION BY id % 200`; and a crash (data race) when a `text` index with a `sparseGrams` tokenizer is queried with a `LIKE` predicate over many partitions with `max_threads > 1`. [#109896](https://github.com/ClickHouse/ClickHouse/pull/109896) ([groeneai](https://github.com/groeneai)). * Fix wrong results when a `minmax` skip index is built on a `LowCardinality(Nullable(...))` column: `WHERE`/`PREWHERE`/`HAVING ... IS NULL` predicates pushed to storage previously pruned every granule and returned 0 rows even though the column contained NULLs. [#110061](https://github.com/ClickHouse/ClickHouse/pull/110061) ([groeneai](https://github.com/groeneai)). * Fixed slow `ORDER BY ... LIMIT` queries on `Distributed` tables when `prefer_localhost_replica` selects a local replica. [#110136](https://github.com/ClickHouse/ClickHouse/pull/110136) ([EmeraldShift](https://github.com/EmeraldShift)). * Fixed an exception (`UNION mode UNION_DEFAULT must be normalized`) when a `DELETE` or `ALTER UPDATE` mutation used a set operation (`UNION`/`UNION ALL`/`UNION DISTINCT`/`EXCEPT`/`INTERSECT`) inside a subquery in its `WHERE` condition or in an `UPDATE` assignment. [#110196](https://github.com/ClickHouse/ClickHouse/pull/110196) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed slow server shutdown and unresponsive `KILL MUTATION` when a mutation with `x IN (subquery)` was building the subquery set during primary-key analysis; the set build is now cancelled promptly. [#110198](https://github.com/ClickHouse/ClickHouse/pull/110198) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a race between `KILL TRANSACTION` and `ALTER TABLE ... UPDATE/DELETE` executed inside a transaction: a rollback landing between the mutation's registration in the transaction and its registration in the table left an orphaned mutation entry, which raised a logical error (`Cannot find transaction ... that has started mutation ...`) in background jobs and blocked all subsequent mutations of the affected parts until server restart. [#110226](https://github.com/ClickHouse/ClickHouse/pull/110226) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed `system.tables` for a `DataLakeCatalog` (Iceberg/Glue) database aborting the whole query, or silently dropping tables, when a single table's metadata is unresolvable. Such a table now stays listed by name with default/NULL values for the columns that need the opened storage object (`engine`, `total_rows`, etc.), regardless of `database_datalake_require_metadata_access`. Direct access to the broken table still reports the error. [#110242](https://github.com/ClickHouse/ClickHouse/pull/110242) ([groeneai](https://github.com/groeneai)). * Join-order cardinality estimation now composes column statistics over the parts surviving partition/PK pruning instead of all active parts. Previously a query pruned to a small partition was planned against table-wide statistics (observed 2500x row overestimation flipping the hash-join build side), and disabling statistics paradoxically produced a better plan. Also fixed: lazy FINAL was silently disabled for MergeTree relations under a JOIN because the join-order optimizer's memoized index-analysis result was mistaken for an applied projection. [#110283](https://github.com/ClickHouse/ClickHouse/pull/110283) ([skuznetsov-clickhouse](https://github.com/skuznetsov-clickhouse)). * `ALTER TABLE ... MOVE/REPLACE PARTITION` between two MergeTree tables with incompatible granularity (one adaptive, one non-adaptive) now throws `BAD_ARGUMENTS` instead of `LOGICAL_ERROR`. This is a user-reachable condition, so it should not surface as an internal error (which aborts the server in debug/sanitizer builds). [#110535](https://github.com/ClickHouse/ClickHouse/pull/110535) ([groeneai](https://github.com/groeneai)). * Fix a `TTL ... DELETE WHERE ` never removing rows that started matching `` because of an `ALTER TABLE ... UPDATE` on a column referenced in the `WHERE`. The mutation now recalculates the row TTL, so the newly matching expired rows are deleted instead of being silently retained. [#110648](https://github.com/ClickHouse/ClickHouse/pull/110648) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` "Invalid partition key size: 0" that could abort an `INSERT` into a partitioned `MergeTree` table with `non_replicated_deduplication_window` enabled, when an async insert batch was only partially deduplicated (some `insert_deduplication_token`s were duplicates and some were new). [#110651](https://github.com/ClickHouse/ClickHouse/pull/110651) ([groeneai](https://github.com/groeneai)). * Fix constant folding under the new analyzer silently changing the type of a constant `Geometry` value. `Geometry` is a named `Variant` whose alternatives can share the same storage layout (`LineString`/`Ring`, `MultiLineString`/`Polygon`, and empties), so two folded constants that differ only in their alternative collapsed into one, e.g. a constant `MULTIPOLYGON EMPTY` came out as `LINESTRING()`. [#110709](https://github.com/ClickHouse/ClickHouse/pull/110709) ([groeneai](https://github.com/groeneai)). * Fix a wrong result (mis-ordered merge) for read-in-order queries where `distinct-in-order` or `aggregation-in-order` widens the read to a longer sort-key prefix than the one `ORDER BY` set up the read-in-order virtual row for. The extra sort columns of the virtual row were default-filled, which could mis-order the `MergingSortedTransform` (and abort with a `LOGICAL_ERROR` in debug/sanitizer builds). [#110725](https://github.com/ClickHouse/ClickHouse/pull/110725) ([groeneai](https://github.com/groeneai)). * Propagate a query's active roles (SET ROLE / role=) to remote nodes on parallel-replica and Distributed reads over an interserver-secret cluster; previously they fell back to the user's default roles, evaluating row policies inconsistently with the initiator. [#110867](https://github.com/ClickHouse/ClickHouse/pull/110867) ([evillique](https://github.com/evillique)). * Fixed a rare race where a comment-only or settings-only `ALTER` of a `ReplicatedMergeTree` table could silently drop a column added by a concurrent `ALTER ... ADD COLUMN`, leaving a replica on an outdated table structure. [#111029](https://github.com/ClickHouse/ClickHouse/pull/111029) ([tiandiwonder](https://github.com/tiandiwonder)). * Fix `NUMBER_OF_COLUMNS_DOESNT_MATCH` error for a `GROUP BY` with an aggregate over a `Merge` table wrapping a `Distributed` table, when the grouping key is a function of a column and an aggregate reads the same column (e.g. `SELECT toString(g), min(g) FROM merge_over_distributed GROUP BY toString(g)`). [#111334](https://github.com/ClickHouse/ClickHouse/pull/111334) ([groeneai](https://github.com/groeneai)). * Fix wrong result of a `FULL JOIN` on a `Join` engine table (`StorageJoin`) with `USING` when the left key is `Nullable` and the storage key is not: an unmatched left row with a `NULL` key returned `0` instead of `NULL`. [#111371](https://github.com/ClickHouse/ClickHouse/pull/111371) ([groeneai](https://github.com/groeneai)). * `ALTER TABLE ... DETACH PARTITION/PART` now fsyncs the clone it creates under `detached/` (the part subtree plus the directory chain up to the disk root) when the table's `fsync_part_directory` setting is enabled, for both Full (directory) and Packed part storage. Previously the clone was never fsynced while the empty covering part that removes the rows from the active set was, so a power loss right after the acknowledgement could destroy the detached data on btrfs (the un-synced clone directory entries roll back, leaving `detached/` empty and nothing to `ATTACH`). Gated on the existing `fsync_part_directory` setting; default behavior is unchanged. [#111426](https://github.com/ClickHouse/ClickHouse/pull/111426) ([groeneai](https://github.com/groeneai)). * Reject a `Nullable(Nothing)` (or any other type not allowed in tables) column at `CREATE TABLE` time when the schema is inferred from the storage (e.g. a columnless `remote()`/`Remote`/`Distributed` table over a view that does `SELECT NULL`), instead of persisting metadata that fails to load on the next server restart and blocks loading of unrelated tables. [#111487](https://github.com/ClickHouse/ClickHouse/pull/111487) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` (`Got read request from replica N for unknown stream ...`) that could abort the server when reading with parallel replicas and the query is fully answered by a projection optimization (exact-count, minmax-count, or a stored aggregate/normal projection selecting no ranges) on the initiator. [#111689](https://github.com/ClickHouse/ClickHouse/pull/111689) ([groeneai](https://github.com/groeneai)). * Fixed DeltaLake partitioned `INSERT` (`allow_experimental_delta_lake_writes=1`) corrupting partition values that contain path characters and rejecting `NULL`. A value like `a/b` was silently committed as `a`, `%` produced a data file that could not be read back, and a `NULL` partition value failed with `NOT_IMPLEMENTED`. Partition values are now percent-encoded into a single path segment, the true value is stored in `partitionValues` instead of being parsed back out of the path, and a `NULL` (or empty-string) partition value is written as `__HIVE_DEFAULT_PARTITION__` with a null `partitionValues` entry. [#111793](https://github.com/ClickHouse/ClickHouse/pull/111793) ([groeneai](https://github.com/groeneai)). * Fixed `Expected UInt64 column for __grouping_set, got BLOB` and `ColumnBLOB should be converted to a regular column before usage` logical errors on a distributed query executed on a shard with `enable_parallel_blocks_marshalling` (enabled by default), for example with `GROUP BY GROUPING SETS` over a cluster that has a local replica, or when reading a `Merge` table that spans a local and a distributed table. `BlocksMarshallingStep` is no longer added to plans whose blocks are consumed in the same process. [#111997](https://github.com/ClickHouse/ClickHouse/pull/111997) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` (`Type mismatch when building statistics for column`) during a `MergeTree` mutation of a table whose column type was changed by a metadata-only `ALTER MODIFY COLUMN` (for example `Enum8` to `Int8` or `DateTime` to `UInt32`) while the column carries statistics. The mutation now recomputes only the statistics of columns it actually rewrites and carries the rest over unchanged, and a column rewritten by `MATERIALIZE COLUMN`, by an `UPDATE` of a column a `MATERIALIZED` column depends on, or by `CLEAR COLUMN` now gets its statistics recomputed instead of keeping stale content. [#112009](https://github.com/ClickHouse/ClickHouse/pull/112009) ([groeneai](https://github.com/groeneai)). * Fixes silently missing rows when a query filters on `intDiv(constant, key)` or `divide(constant, key)`. Primary-key and partition analysis reports the wrong direction of monotonicity for a constant divided by a key column, so matching granules are pruned away and the query returns fewer rows with no error. Also fixes `intDiv` with an unsigned constant dividend whose high bit is set, stops claiming monotonicity when a `Decimal` operand makes the division compute in the decimal's native width, and stops key analysis from raising `Cannot compare DB::IPv4 with long` for a valid query that divides an `IPv4`/`IPv6` constant by a key column. [#112156](https://github.com/ClickHouse/ClickHouse/pull/112156) ([groeneai](https://github.com/groeneai)). * Fixes a parameterized view whose `SELECT` body uses `INTERSECT`/`EXCEPT`, or a `UNION` chain that mixes `UNION DISTINCT` with `UNION ALL`, being classified as an ordinary view. Such a view was created successfully but its metadata could not be read back: loading it threw `Invalid storage definition in metadata file`, which made the view permanently unloadable and aborted a debug or sanitizer server on startup. `system.tables.parameterized_view_parameters` also reported no parameters for those views. [#112211](https://github.com/ClickHouse/ClickHouse/pull/112211) ([groeneai](https://github.com/groeneai)). * Fixes `max_execution_time` and `KILL QUERY` being ignored while a query evaluates `geohashesInBox`. Such a query expanded every row of a block before it could be interrupted, so cancellation took effect only minutes later; the function now checks the query's time limit as it walks the rows. Evaluations inside a persisted table expression, such as a sorting key, a skip index or a `PARTITION BY`, are not covered and are tracked separately. [#112273](https://github.com/ClickHouse/ClickHouse/pull/112273) ([groeneai](https://github.com/groeneai)). * Fixes a logical error when PREWHERE is used on a Merge table over another Merge table or a materialized view whose column type differs. Also fixes a row policy being silently ignored on tables that read from remote servers (a Distributed table, or a table wrapping one), which could expose rows the policy was meant to hide; such queries are now rejected with an ILLEGAL\_PREWHERE error. Define these policies on the underlying local tables on each remote server instead; note that such a policy is still not applied to reads shipped with `serialize_query_plan = 1`, which is a separate, pre-existing bug. [#112326](https://github.com/ClickHouse/ClickHouse/pull/112326) ([PedroTadim](https://github.com/PedroTadim)). * Fixes reading a stale secondary (skip) index after an `ALTER TABLE ... MODIFY COLUMN` type change whose mutation never ran, for example because `KILL MUTATION` removed it: the granules on disk were written with the old type but decoded with the new one, raising `LOGICAL_ERROR`, requesting multi-exabyte allocations, or silently returning a wrong result. Also fixes a wrong result from an index over an expression whose meaning changes while every stored byte stays identical, so no mutation is created at all: a `MODIFY COLUMN` altering only a `DateTime` timezone, or only a custom type name such as `UInt8` to `Bool`. Such an index is now skipped for the affected part. [#112484](https://github.com/ClickHouse/ClickHouse/pull/112484) ([groeneai](https://github.com/groeneai)). * Fixes a `LOGICAL_ERROR` (`Cannot add transform Resize to Pipes because it has no outputs`) when a query aggregating over a cluster runs with `serialize_query_plan = 1` and plan-based parallel replicas, caused by the merge step resizing the pipeline to zero streams. [#112506](https://github.com/ClickHouse/ClickHouse/pull/112506) ([groeneai](https://github.com/groeneai)). * Fixes a `LOGICAL_ERROR` ("Part ... contains column ... that is absent in table ...") when a `ReplicatedMergeTree` replica mutates a part it fetched from another replica before applying its own pending `ALTER ... ADD COLUMN`. Such a mutation is now postponed until the metadata change is applied. [#112510](https://github.com/ClickHouse/ClickHouse/pull/112510) ([groeneai](https://github.com/groeneai)). * Fixes the read counters (`read_rows`, `read_bytes`, `ProfileEvents['SelectedRows']`, `ProfileEvents['SelectedBytes']`) and the `READ_ROWS` and `READ_BYTES` quotas being over-charged for queries whose `ORDER BY` spills to disk. Rows re-read from the external sort temporary files were counted again as source reads, so a spilling query over-reported what it read and could be rejected with `QUOTA_EXCEEDED` even though it stayed within its quota. [#112645](https://github.com/ClickHouse/ClickHouse/pull/112645) ([groeneai](https://github.com/groeneai)). * Fixes `ProfileEvents['JoinResultRowCount']` under-reporting the result size of a `JOIN` that spills to disk. Rows emitted from delayed buckets were counted only in `JoinDelayedJoinedTransformRowCount` and were missing from the total. Query results were always correct; only the profile event was wrong. [#112673](https://github.com/ClickHouse/ClickHouse/pull/112673) ([groeneai](https://github.com/groeneai)). * Fixes `max_execution_time` and `KILL QUERY` being silently ineffective for a distributed query that is establishing connections to replicas. Such a query now stops at the next connection attempt instead of working through every remaining replica and retry. [#112689](https://github.com/ClickHouse/ClickHouse/pull/112689) ([groeneai](https://github.com/groeneai)). * Fixed `ALTER` on `MergeTree` tables being applied while an incomplete `RENAME COLUMN` alter-mutation is still in progress. This could update the storage metadata incompatibly and block later mutations completely. [#112783](https://github.com/ClickHouse/ClickHouse/pull/112783) ([Michicosun](https://github.com/Michicosun)). * Fix wrong results of `ORDER BY` when reading from a `Merge` table over `Distributed` tables: rows could be returned out of order, and rows belonging in the result could be missing when `LIMIT` was used. [#112931](https://github.com/ClickHouse/ClickHouse/pull/112931) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed wrong results when filesystem read prefetch is enabled and a `Nested` column is read beside a subcolumn that was dropped and re-added, on tables with `share_nested_offsets` enabled (the default). Such a query could return default values for a physically present column that the `ALTER`s never touched. This needs no non-default settings on object storage, where `remote_filesystem_read_prefetch` defaults to `1`; on local disks it needs `local_filesystem_read_prefetch = 1`. A separate, prefetch-independent defect still corrupts such reads when a range spans several blocks; that one is not covered here. [#112997](https://github.com/ClickHouse/ClickHouse/pull/112997) ([groeneai](https://github.com/groeneai)). * Fixed a `Pipeline stuck` logical error in queries that scatter data by partition, such as a window function with `PARTITION BY` or a join with `join_algorithm = 'parallel_full_sorting_merge'`, when one shard's downstream finished while a block was only partially distributed. [#113190](https://github.com/ClickHouse/ClickHouse/pull/113190) ([groeneai](https://github.com/groeneai)). * Fixed a hang of `ALTER`, `OPTIMIZE`, `TRUNCATE` and partition manipulation statements with `alter_sync = 2` on `ReplicatedMergeTree` when the table starts shutting down while the statement waits for another replica, for example because of a concurrent `DROP DATABASE ... SYNC`. Such a statement now fails with `UNFINISHED` and also honours `max_execution_time` and `KILL QUERY`. [#113228](https://github.com/ClickHouse/ClickHouse/pull/113228) ([groeneai](https://github.com/groeneai)). * Fixed `Cannot insert element into Set` (`NOT_IMPLEMENTED`) when `read_in_order_use_virtual_row` is enabled and a `MergeTree` read-in-order query combines `WITH FILL ... INTERPOLATE` with an `IN` or `has` filter. [#113264](https://github.com/ClickHouse/ClickHouse/pull/113264) ([groeneai](https://github.com/groeneai)). * Fix a logical error (exception) `Chunk info was not set for chunk in MergingAggregatedTransform` when the setting `inject_random_order_for_select_without_order_by` is enabled and an aggregation query reads from a `Merge` table containing a `Distributed` child: the random `ORDER BY rand()` wrapper is no longer injected into queries planned only up to an intermediate stage. [#113266](https://github.com/ClickHouse/ClickHouse/pull/113266) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed rows never being deleted when a `MergeTree` table has two or more `TTL ... DELETE WHERE` rules whose time expression is identical and which differ only in the `WHERE` condition. After a TTL delete merge the part's `rows_where_ttl_info` was reset to zero, so no further TTL merge was ever scheduled for it and the expired rows were retained indefinitely with no error or log message. [#113384](https://github.com/ClickHouse/ClickHouse/pull/113384) ([groeneai](https://github.com/groeneai)). * Fixed `max_execution_time` and `KILL QUERY` being ignored while a query evaluates `geohashesInBox`, `arrayFold`, `sleep`, `base58Decode`, `h3PolygonToCells` or `h3PolygonToCellsWithContainment` inside an expression stored in table metadata, such as a sorting key, a skip index or a `PARTITION BY` key. [#113456](https://github.com/ClickHouse/ClickHouse/pull/113456) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` (`Reading from materialized CTE '...' before its materialization completed - DelayedPortsProcessor gate is missing in the query plan`) raised when a `Merge` table with more than one child reads a materialized CTE that the outer query references. [#113558](https://github.com/ClickHouse/ClickHouse/pull/113558) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `Logical error: 'Required output position N is out of range for pass-through inputs'` in `EXPLAIN ANALYZE` over a `Merge` table with several children, a bare-column `PREWHERE` and a separate `WHERE`. [#113725](https://github.com/ClickHouse/ClickHouse/pull/113725) ([groeneai](https://github.com/groeneai)). * Fix a logical error (`Unexpected return type`) when a filter over a view whose column type differs from the underlying storage (e.g. `information_schema.tables`, where `engine` is `Nullable(String)`, over `system.tables`, where it is `String`) is pushed down to the storage. [#113984](https://github.com/ClickHouse/ClickHouse/pull/113984) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a `Not-ready Set is passed as the second argument for function 'in'` error when a mutation whose predicate contains `IN (subquery)` is cancelled, for example by `KILL MUTATION` or `DETACH DATABASE`, while the subquery's set is still being built.. [#114463](https://github.com/ClickHouse/ClickHouse/pull/114463) ([groeneai](https://github.com/groeneai)). * `ALTER TABLE ... DROP COLUMN` of a column that is used in the sorting, primary or partition key now fails with `ALTER_OF_COLUMN_IS_FORBIDDEN` and an explanation, the same as `ALTER TABLE ... CLEAR COLUMN` does. Previously it failed with a confusing `UNKNOWN_IDENTIFIER: Missing columns` error coming from the recalculation of the key expressions. The same applies to dropping or clearing a whole `Nested` group by its common prefix when a column of the group is used in a key; previously that could silently destroy the group's data or leave a mutation that never finishes. Dropping a column that is used only in a `TTL` expression now also reports which TTL expression it breaks [#114468](https://github.com/ClickHouse/ClickHouse/pull/114468) ([m7kss1](https://github.com/m7kss1)). * Fixed a column that was renamed, dropped and added again in one `ALTER` returning the dropped column's values instead of its own default while the mutation was still pending. [#114601](https://github.com/ClickHouse/ClickHouse/pull/114601) ([tiandiwonder](https://github.com/tiandiwonder)). * Fix reading a subcolumn (`.size0`, `.null`, a tuple element, `String.size`) through a `Merge` table when the underlying table declares the parent column as `ALIAS`. Such a read returned the type default, and when the parent was selected in the same query it could return the value of an unrelated column. [#114673](https://github.com/ClickHouse/ClickHouse/pull/114673) ([groeneai](https://github.com/groeneai)). * Fixed rows missing from the result of a `SELECT` filtered by `pointInPolygon` over a MergeTree primary key when the polygon argument is an invalid constant literal and `validate_polygons = 0`. Primary key analysis derived granule pruning from a polygon it never validated, and dropped granules holding matching rows. [#114710](https://github.com/ClickHouse/ClickHouse/pull/114710) ([groeneai](https://github.com/groeneai)). * Fixed the formatting of `ALTER TABLE ... MATERIALIZE INDEX`, `MATERIALIZE STATISTICS` and `MATERIALIZE PROJECTION`, which dropped the `IF EXISTS` clause, so the formatted query was not the query that was parsed. Also rejected four `ALTER ... STATISTICS` AST JSON payloads that the SQL parser cannot produce and that formatted into a different command. [#115099](https://github.com/ClickHouse/ClickHouse/pull/115099) ([groeneai](https://github.com/groeneai)). * Fixed an out-of-bounds read of the cached dictionary hashes when aggregating or joining on a `LowCardinality(String)` key whose dictionary grows while it is being aggregated, reachable from `TTL ... GROUP BY`. It could yield a wrong hash, and a reported heap-buffer-overflow in sanitizer builds. [#115604](https://github.com/ClickHouse/ClickHouse/pull/115604) ([groeneai](https://github.com/groeneai)). * `CREATE TABLE` and `ALTER TABLE` now reject a lossy codec such as `SZ3` on a column used in the partition key. [#115626](https://github.com/ClickHouse/ClickHouse/pull/115626) ([groeneai](https://github.com/groeneai)). * Reading through a `Merge` table that matches an `Alias` table whose target is a parameterized view no longer raises `LOGICAL_ERROR: Table has no columns.` (which aborts the server in debug and sanitizer builds). Such a table now reports `STORAGE_REQUIRES_PARAMETER`, the same error the parameterized view itself reports. Any other matched table that cannot supply columns now reports `UNSUPPORTED_METHOD` instead of an internal error. [#115896](https://github.com/ClickHouse/ClickHouse/pull/115896) ([groeneai](https://github.com/groeneai)). * Fixed the on-the-fly preview of a pending mutation whose expression references `_sample_factor`. With `apply_mutations_on_fly = 1` and a `SAMPLE` clause, the expression was evaluated with the reading query's sample factor instead of the value the background materialization persists, so previewed rows disagreed with the eventual materialized result, and a pending `ALTER TABLE t DELETE WHERE _sample_factor > 1` could make a `SAMPLE` read return no rows at all. The query's own `_sample_factor` column is unchanged. [#115906](https://github.com/ClickHouse/ClickHouse/pull/115906) ([groeneai](https://github.com/groeneai)). * Statistics files with format version `V3` are now read instead of rejected with `ILLEGAL_STATISTICS`. Such files were produced only by builds of `master` from a narrow window, but parts containing them were completely unreadable, and on a readonly disk `ALTER TABLE ... MATERIALIZE STATISTICS` cannot regenerate them. [#116030](https://github.com/ClickHouse/ClickHouse/pull/116030) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `Invalid number of columns in chunk pushed to OutputPort` (`LOGICAL_ERROR`) when a `SELECT ... FROM t STREAM ... ORDER BY ... LIMIT n` query read a table with a `PROJECTION`. Lazy materialization is no longer applied to a `STREAM` read. [#116187](https://github.com/ClickHouse/ClickHouse/pull/116187) ([groeneai](https://github.com/groeneai)). * Do not replicate `ALTER TABLE ATTACH PARTITION FROM` for replicated db. [#94835](https://github.com/ClickHouse/ClickHouse/pull/94835) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). * Fixed a server abort (in sanitizer builds) and silent data loss during table startup when a rolled-back transactional MergeTree part intersects a committed part. `PartLoadingTree::add` now correctly identifies which of the two intersecting parts was rolled back, evicts it, and re-parents its orphaned children under the committed part in the correct order. [#100992](https://github.com/ClickHouse/ClickHouse/pull/100992) ([tuanpach](https://github.com/tuanpach)). * Fixed `indexOfAssumeSorted` returning incorrect results for `Array(LowCardinality(String))` columns in MergeTree tables. [#101771](https://github.com/ClickHouse/ClickHouse/pull/101771) ([Onyx2406](https://github.com/Onyx2406)). * Fix `LOGICAL_ERROR` exceptions when reading `Iceberg` or `DeltaLake` data lake tables through paths that can reach the read pipeline without a pinned `datalake_table_state`, such as concurrent `Iceberg` metadata updates or `merge` reads over `DeltaLake` tables. [#102033](https://github.com/ClickHouse/ClickHouse/pull/102033) ([groeneai](https://github.com/groeneai)). * Fixed `Logical error: Parsed partition value: ... doesn't match partition value for an existing part with the same partition ID: ...` thrown by `OPTIMIZE TABLE ... PARTITION ...` (and other queries that resolve a partition value through `convertFieldToTypeImpl`) on tables with a `Time`-typed partition key. [#106202](https://github.com/ClickHouse/ClickHouse/pull/106202) ([groeneai](https://github.com/groeneai)). * Fixes a data part being incorrectly marked as having a broken projection after a lightweight delete leaves no projection part behind — either when the projection is rebuilt with zero output rows or when it is dropped (`lightweight_mutation_projection_mode = 'rebuild'`/`'drop'`). The broken state previously disabled projection optimization for queries on that part. [#106273](https://github.com/ClickHouse/ClickHouse/pull/106273) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed memory/CPU/mutation overload warnings in `system.warnings` lagging one asynchronous-metrics cycle behind (and being absent on the first cycle). [#108035](https://github.com/ClickHouse/ClickHouse/pull/108035) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a `LOGICAL_ERROR` exception (`ScatterExchangeStep should have one source shard, got 8`) and duplicated rows when querying a `Merge` table over `Distributed` tables or views with the experimental setting `make_distributed_plan = 1`. [#108401](https://github.com/ClickHouse/ClickHouse/pull/108401) ([groeneai](https://github.com/groeneai)). * Fixed a server abort (`Sort order of blocks violated`) during a merge, and incorrect reads, when a non-nullable `LowCardinality` element of a `Nullable(Tuple(...))` column is used as a subcolumn (for example as a sort key). [#108679](https://github.com/ClickHouse/ClickHouse/pull/108679) ([groeneai](https://github.com/groeneai)). * Fix a crash in the vendored Azure SDK `Base64Decode` reachable via `azureBlobStorage()` (and related Azure table functions/storage) when the account key is not valid Base64. A byte `>= 0x80` caused an out-of-bounds read of the decode table, and any invalid byte caused an undefined left shift of a negative value (abort under sanitizers). Invalid keys are now rejected cleanly. [#108716](https://github.com/ClickHouse/ClickHouse/pull/108716) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` ("New empty part is about to materialize but the directory already exist") that could abort the server in debug and sanitizer builds when a `DROP`/`DETACH`/`MOVE`/`REPLACE PARTITION` on a MergeTree table ran after a previous such operation was interrupted (for example a rolled-back transaction or a crash) and left a stale `tmp_empty_` directory behind. The stale directory is now reclaimed instead of failing. [#108879](https://github.com/ClickHouse/ClickHouse/pull/108879) ([groeneai](https://github.com/groeneai)). * Fix a projection returning a column's type default (e.g. `0`) instead of its DDL `DEFAULT` (e.g. `-1`) after the column's `TTL` expired on a wide part. [#109013](https://github.com/ClickHouse/ClickHouse/pull/109013) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed `ALTER` queries failing for `MergeTree` tables created with a custom `disk` setting (`SETTINGS disk = disk(...)`). [#109199](https://github.com/ClickHouse/ClickHouse/pull/109199) ([Michicosun](https://github.com/Michicosun)). * Fix stale and deleted rows appearing when reading tables of a `MaterializedPostgreSQL` database through a `Merge` table. [#109339](https://github.com/ClickHouse/ClickHouse/pull/109339) ([vdimir](https://github.com/vdimir)). * Fixed reading Paimon tables partitioned by a `BIGINT` column whose value does not fit into `Int32`. Such partition values were truncated (for example `9223372036854775807` became `-1`), which produced a wrong partition path and failed with a filesystem error. [#109510](https://github.com/ClickHouse/ClickHouse/pull/109510) ([groeneai](https://github.com/groeneai)). * Fix a logical error ("Cannot determine row level filter; 0 columns deleted, 0 columns added") when reading from a `Merge` table whose child table has a row policy whose filter is a bare existing column (for example `CREATE ROW POLICY ... USING flag`). [#109843](https://github.com/ClickHouse/ClickHouse/pull/109843) ([groeneai](https://github.com/groeneai)). * IcebergLocal was reporting Azure as the storage type. [#109872](https://github.com/ClickHouse/ClickHouse/pull/109872) ([PedroTadim](https://github.com/PedroTadim)). * `CHECK TABLE` now reports the actual corruption for a projection part that failed to load its metadata, instead of a misleading `Columns doesn't match ... Expected: 0 columns` error. [#110262](https://github.com/ClickHouse/ClickHouse/pull/110262) ([Algunenano](https://github.com/Algunenano)). * Fixed server startup failure for `MergeTree` tables on object storage disks when a leftover `txn_version.txt.tmp` file has broken disk-level metadata. [#110519](https://github.com/ClickHouse/ClickHouse/pull/110519) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a MergeTree table with a `_part_offset` or commit-order projection becoming permanently unattachable after `allow_part_offset_column_in_projections` or `allow_commit_order_projection` was disabled and the table was detached or the server restarted. These two settings are CREATE-time gates, so they are no longer re-checked on ATTACH. [#110570](https://github.com/ClickHouse/ClickHouse/pull/110570) ([groeneai](https://github.com/groeneai)). * Fix a rare `LOGICAL_ERROR` ("Unexpected number of parts to remove from parts\_queue") that could abort the server on a `ReplicatedMergeTree` when two partition operations (for example two `MOVE PARTITION` / `REPLACE PARTITION` queries, or one racing a background `DROP_RANGE`) cancelled the background part check over overlapping ranges at the same time. [#110738](https://github.com/ClickHouse/ClickHouse/pull/110738) ([groeneai](https://github.com/groeneai)). * Fixed a `Hash is not set for serialization` logical error that could occur when a column with a `Quantized(...)` codec was nested inside a poolable serialization, for example a `Merge` table over sources whose same-named column has different types (merged into a `Variant`). [#110776](https://github.com/ClickHouse/ClickHouse/pull/110776) ([groeneai](https://github.com/groeneai)). * Fixed a `std::length_error` (`LOGICAL_ERROR`) when reading a MergeTree table with a pathological `max_streams_for_merge_tree_reading` value (near the maximum of `UInt64`) together with `allow_asynchronous_read_from_io_pool_for_merge_tree = 1`. The setting is now clamped to the same ceiling as `max_threads`. [#110936](https://github.com/ClickHouse/ClickHouse/pull/110936) ([groeneai](https://github.com/groeneai)). * Fix a logical error `Cannot calculate columns sizes when columns or checksums are not initialized` during a merge of a single-column `MergeTree` table whose only column has a fully-expiring column `TTL` (with `min_bytes_for_wide_part = 1`). [#111169](https://github.com/ClickHouse/ClickHouse/pull/111169) ([groeneai](https://github.com/groeneai)). * Fsync the text index files of a merged or materialized part when `min_rows_to_fsync_after_merge` / `min_compressed_bytes_to_fsync_after_merge` / `fsync_part_directory` are set. Previously these files were the only ones in the part left unsynced, so a power loss right after a merge could leave a committed but broken part and cause data loss. [#111335](https://github.com/ClickHouse/ClickHouse/pull/111335) ([groeneai](https://github.com/groeneai)). * Fix `OPTIMIZE TABLE ... MANIFEST` for `Iceberg` tables when manifest partition tuples do not match their partition spec [#111368](https://github.com/ClickHouse/ClickHouse/pull/111368) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Fix an out-of-bounds read (release) / logical error (debug) in the lazy materialization optimization, and a logical error in `Pipe::unitePipes` for `Merge` tables, when the direct-read-from-text-index optimization leaves an unused index virtual column in the read header of a `MergeTree` table with multiple `text` indexes. [#111467](https://github.com/ClickHouse/ClickHouse/pull/111467) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` ("Part level Min-Max index was constructed from unexpected columns set") that could abort a background merge in debug/sanitizer builds after `part_minmax_index_columns` was lowered (for example from `with_block_number_offset` back to `partition_key_only`). [#111511](https://github.com/ClickHouse/ClickHouse/pull/111511) ([groeneai](https://github.com/groeneai)). * Fixed a crash when running a `STREAM` read (`enable_streaming_queries = 1`) over a table with more than one partition and a row policy defined on it. In debug and sanitizer builds it failed with the `Filter column ... not found in DAG outputs` logical error; in release builds it could segfault. [#111572](https://github.com/ClickHouse/ClickHouse/pull/111572) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` (`Cannot create part with type Compact and storage type Full because table does not support polymorphic parts`) raised when a `MergeTree` table loads a pre-existing `Compact` part while its current settings disable polymorphic parts (non-adaptive granularity). This could happen, for example, when a read-only table shares an on-disk path with another table and reloads its parts via `SYSTEM RESTART DISK` or on server startup. In debug and sanitizer builds the exception aborts the server. [#111626](https://github.com/ClickHouse/ClickHouse/pull/111626) ([groeneai](https://github.com/groeneai)). * With `fsync_part_directory = 1`, the rename of a part directory from its temporary to its final name is now itself made durable by fsyncing the parent directory (and, for moves across parents, the source parent too). Previously only the moved directory was fsynced, so a power loss right after a part was committed could leave the part missing on disk even though its files were already written, losing rows despite the setting being enabled. [#111861](https://github.com/ClickHouse/ClickHouse/pull/111861) ([groeneai](https://github.com/groeneai)). * Fixed wrong results and a `LOGICAL_ERROR` when a `Nullable(Tuple(...))` column is selected together with one of its null-carrying element subcolumns (`Variant`, `Dynamic` or `LowCardinality`) from a `MergeTree` table. Reading the subcolumn marked the rows that are `NULL` in the parent tuple as `NULL` in a column that had already been shared through the substreams cache, so the read of the whole column saw shifted values. [#111949](https://github.com/ClickHouse/ClickHouse/pull/111949) ([groeneai](https://github.com/groeneai)). * Fixed hash joins losing their post-build optimizations (conversion to a fixed hash table, the shared runtime filter, right table reranging) whenever an automatic spill-to-disk threshold was configured, which is the default since 26.5. [#111972](https://github.com/ClickHouse/ClickHouse/pull/111972) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes wrong results (silently missing rows) for `hasAllTokens` and other multi-token `text` index searches on builds that do not use `simdcomp` (every target except x86\_64 Linux, e.g. aarch64) when `text_index_posting_list_apply_mode = 'lazy'`. The portable bitpacking decoder did not write the decoded zeros for a block whose bit width is zero, so a posting living in a single-posting segment was dropped. The on-disk format is unchanged and existing data reads correctly after the fix. [#112178](https://github.com/ClickHouse/ClickHouse/pull/112178) ([groeneai](https://github.com/groeneai)). * Operations now fail when corrupt MergeTree statistics are loaded instead of ignoring the error. [#112201](https://github.com/ClickHouse/ClickHouse/pull/112201) ([skuznetsov-clickhouse](https://github.com/skuznetsov-clickhouse)). * Fixes silent loss of acknowledged Iceberg writes on HDFS: an Iceberg commit publishes each new metadata file with a conditional write, but `HDFSObjectStorage` dropped the condition and performed a plain overwrite, so concurrent writers were all acknowledged while one writer's snapshot became unreachable. HDFS now refuses a conditional write instead, so storage-managed Iceberg writes on HDFS, `CREATE TABLE ... ENGINE = IcebergHDFS(...)` with an explicit column list included, now fail. Reads are unaffected. [#112437](https://github.com/ClickHouse/ClickHouse/pull/112437) ([groeneai](https://github.com/groeneai)). * Fix `DELETE` on `KeeperMap` tables deleting only the first `max_block_size` matched rows while reporting the mutation as successfully completed. Also fix `DELETE` with `keeper_map_strict_mode = 1` not being atomic: it could fall back to unversioned removals and delete rows that were concurrently updated, and it applied the matched rows block by block so a conflict could leave the delete partially applied. [#112777](https://github.com/ClickHouse/ClickHouse/pull/112777) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a transient Azure Blob Storage error on the read or merge path — an RBAC permission-propagation `403 Forbidden`, a credential authentication failure, or a connect timeout surfaced as `408` — being misclassified as non-retryable and reported as `POTENTIALLY_BROKEN_DATA_PART` (Code 740), raising a false critical alert for a healthy part. These transient failures are now treated as retryable, and a genuinely persistent one surfaces as a plain Azure error instead of a phantom broken data part. [#112871](https://github.com/ClickHouse/ClickHouse/pull/112871) ([arsenmuk](https://github.com/arsenmuk)). * Fixed `listObjects` on a prefixed-endpoint Azure Blob Storage disk returning wrong blob names on every page after the first, because pagination bypassed the client wrapper that strips the endpoint prefix. Pagination now re-enters the wrapper via the continuation token so every page's blob names are correct. [#112872](https://github.com/ClickHouse/ClickHouse/pull/112872) ([arsenmuk](https://github.com/arsenmuk)). * Fix the per-part minmax index over `_block_number` and `_block_offset` being lost after a table reload when the part was produced by a mutation that does not rewrite the whole part. [#112878](https://github.com/ClickHouse/ClickHouse/pull/112878) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed an exception (in release builds - an error that prevents the server from starting) when a mutation predicate uses the `merge` table function without an explicit database name. The database of the mutated table is now used, the same way as it is done for table names. [#113185](https://github.com/ClickHouse/ClickHouse/pull/113185) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix wrong results and a `LOGICAL_ERROR` when reading a subcolumn of a dropped-and-re-added `Nested` member together with a present member of the same group, on a `MergeTree` table with `share_nested_offsets = 1`. The present members returned values from a later block whenever the read spanned more than one block. [#113225](https://github.com/ClickHouse/ClickHouse/pull/113225) ([groeneai](https://github.com/groeneai)). * Fix an arithmetic exception in debug and sanitizer builds when reading from a `Merge` table with a very large `max_streams_to_max_threads_ratio`: the number of streams was truncated to 32 bits, so multiples of `2^32` became zero. Also throw `PARAMETER_OUT_OF_BOUND` instead of invoking undefined behavior when multiplying stream counts for a `Merge` table exceeds `size_t`, avoid an overflow when limiting `MergeTree` streams to the amount of data, and reject a `Merge` table read whose aggregate source count would exceed 65536. [#113382](https://github.com/ClickHouse/ClickHouse/pull/113382) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed writes to the `hdfs` disk: they failed with `Parent directory doesn't exist` because the generated object keys contain a nested directory prefix, but unlike blob storages HDFS has real directories and does not create the missing parent on file creation.. [#113396](https://github.com/ClickHouse/ClickHouse/pull/113396) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed the error reported when the `remote`/`remoteSecure` table functions or the `Remote`/`RemoteSecure` table engines are given a named collection that does not exist together with a `key = value` override. Such a call was silently reparsed as a positional call, so the override was analyzed as a data expression and the reported error named something unrelated (for example Function with name `merge` does not exist for `remote(collection, database = merge(...))`). The missing named collection is now reported. [#113510](https://github.com/ClickHouse/ClickHouse/pull/113510) ([groeneai](https://github.com/groeneai)). * Fixed an endless loop in background merges and `OPTIMIZE` when `merge_max_block_size_bytes` was set below the `byteSize()` of an empty column. An empty `LowCardinality` column still carries its dictionary, so the byte limit could be reached before a single row was merged; the merge then produced empty blocks forever, burning 100% of a CPU core and never releasing its background pool slot. [#113585](https://github.com/ClickHouse/ClickHouse/pull/113585) ([groeneai](https://github.com/groeneai)). * Fixed a mutation of a Compact part producing a different result depending on whether the part's serialization info was still in memory or had been reloaded from `serialization.json`, when the table uses `serialization_info_version = 'basic'`. On `ReplicatedMergeTree` this made two replicas holding a byte-identical source part write different mutated parts, and the mutation failed with `CHECKSUM_DOESNT_MATCH`. [#113588](https://github.com/ClickHouse/ClickHouse/pull/113588) ([groeneai](https://github.com/groeneai)). * Backups now report the underlying lock-file access error instead of incorrectly reporting a concurrent backup when the destination storage cannot read the .lock file. [#113637](https://github.com/ClickHouse/ClickHouse/pull/113637) ([kirillshokhin](https://github.com/kirillshokhin)). * Fixed a data race between the background tasks of a `MergeTree` table (statistics refresh, parts refresh, outdated/unexpected parts loading) and the destruction of the table. [#113722](https://github.com/ClickHouse/ClickHouse/pull/113722) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes `UPDATE` and lightweight `DELETE` reading the wrong tables when the database of the updated table differs from the database of the session. The expression of the mutation is now canonicalized against the database of the updated table, both for the `merge` table function, which uses the current database implicitly, and for unqualified table identifiers. Previously such names were resolved against the database of the session, so the statement silently updated or deleted rows selected from a different table. [#113748](https://github.com/ClickHouse/ClickHouse/pull/113748) ([groeneai](https://github.com/groeneai)). * Fixed an exception when reading a `MergeTree` table by primary-key range layers when a layer produces an empty pipe. [#114177](https://github.com/ClickHouse/ClickHouse/pull/114177) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `BACKUP`, `RESTORE` and `ENGINE = Backup` now authorize the backup location against the user's `SOURCES` grants, the same way `s3()`, `file()` and `azureBlobStorage()` do. Previously a user holding `BACKUP` but not `SOURCES` could write a backup to, and read one from, any location the server could reach. Writing a backup now requires the `WRITE` direction on the destination's source (`WRITE ON S3`, `WRITE ON AZURE`, `WRITE ON FILE`) and reading one requires `READ`; `Disk(...)` destinations stay restricted by `backups.allowed_disk` only. [#114405](https://github.com/ClickHouse/ClickHouse/pull/114405) ([groeneai](https://github.com/groeneai)). * Fixed reading a table while a mutation is in progress: a `MATERIALIZED` column could return a stale value, and selecting one together with an updated column could fail with an error. [#114561](https://github.com/ClickHouse/ClickHouse/pull/114561) ([tiandiwonder](https://github.com/tiandiwonder)). * Fix a logical error (`Block structure mismatch`) when reading a `Merge` table with `make_distributed_plan` enabled over children with internal set operations. [#114753](https://github.com/ClickHouse/ClickHouse/pull/114753) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `OPTIMIZE TABLE` on an object-storage table whose engine does not implement compaction reported success while doing nothing. It now raises an error, like every other engine that does not support the statement. This affects `DeltaLake`, `Paimon`, `Hudi` and plain object-storage engines (`S3`, `GCS`, `COSN`, `OSS`, `AzureBlobStorage`, `HDFS`, and object-storage-backed `URL`); `Iceberg`, which does implement compaction, is unchanged. [#114805](https://github.com/ClickHouse/ClickHouse/pull/114805) ([groeneai](https://github.com/groeneai)). * Fix `BAD_DATA_PART_NAME` ("Trying to get part name in new format for old format version") on a `ReplicatedMergeTree` created with the deprecated positional syntax under `allow_deprecated_syntax_for_merge_tree`. A mutation could be reported as done on a replica that had not applied it, and the table could stay readonly after a restart. [#114980](https://github.com/ClickHouse/ClickHouse/pull/114980) ([groeneai](https://github.com/groeneai)). * Fixed `Found patch part ... that intersects mutation with version ...` (`LOGICAL_ERROR`) on tables with lightweight updates. A merge of patch parts could produce a patch whose range of data versions spans the data version of an existing part; that patch can then be neither applied nor skipped, so every later operation on the partition failed and the replication queue stalled. Such merges are no longer assigned. [#115064](https://github.com/ClickHouse/ClickHouse/pull/115064) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed the server failing to start when a `Backup` database engine refers to a backup that has become unavailable (for example, when its files were deleted or the underlying storage is inaccessible). The database is now loaded without tables instead of preventing the whole server from starting. [#83188](https://github.com/ClickHouse/ClickHouse/pull/83188) ([orloffv](https://github.com/orloffv)). * Fix logging recovery after disk full error. Previously, when the log disk became full, ClickHouse would enter a failed state and continuously spam error messages to syslog (potentially writing 100+ GB/hour), never recovering even after disk space was freed. Now the logging system automatically recovers once disk space becomes available, without requiring a server restart. [#93127](https://github.com/ClickHouse/ClickHouse/pull/93127) ([jaehanbyun](https://github.com/jaehanbyun)). * Fix crash in schema visitor in storage DeltaLake. [#97112](https://github.com/ClickHouse/ClickHouse/pull/97112) ([kssenii](https://github.com/kssenii)).

Data lake and external-storage fixes

* Fixed a transient `Code: 499 ... InvalidPart` (`S3_ERROR`) failure on S3 multipart uploads (for example `INSERT INTO ... hits_s3`). MinIO can briefly report a just-uploaded part as missing on `CompleteMultipartUpload`; this error is now retried like the already-handled `NoSuchKey`, bounded by `s3_max_unexpected_write_error_retries`. [#109364](https://github.com/ClickHouse/ClickHouse/pull/109364) ([groeneai](https://github.com/groeneai)). * Reject `ALTER TABLE ... DELETE` and `ALTER TABLE ... UPDATE` on Iceberg tables whose data file format is not `Parquet` with a clear `NOT_IMPLEMENTED` error instead of crashing the server or silently corrupting the table. [#105893](https://github.com/ClickHouse/ClickHouse/pull/105893) ([groeneai](https://github.com/groeneai)). * Fixed the error `Cannot add column ...: column with this name already exists` on `INSERT SELECT` from a table function such as `file`, `s3` or `input`, when the same source column is selected more than once. [#105982](https://github.com/ClickHouse/ClickHouse/pull/105982) ([crakjie](https://github.com/crakjie)). * Sensitive values passed as HTTP query-string parameters (for example the `param_*` query-parameter binding such as `param_secret_key`/`param_aws_secret_access_key`, the `password` parameter, or S3-style `signature` parameters) are no longer written verbatim to `system.text_log` or OpenTelemetry spans. The `Request URI` log line now redacts the values of parameters whose names look sensitive, replacing them with `[HIDDEN]`. [#108475](https://github.com/ClickHouse/ClickHouse/pull/108475) ([groeneai](https://github.com/groeneai)). * Fixed the `_etag` virtual column for the `S3Queue` and `AzureQueue` table engines: it was declared but never populated, so `SELECT _etag` always returned an empty string. It now returns the object ETag, like the `S3` engine and the `s3()` table function. [#108625](https://github.com/ClickHouse/ClickHouse/pull/108625) ([groeneai](https://github.com/groeneai)). * A query reading from or writing to S3 that is cancelled (e.g. by `KILL QUERY`) while an S3 request is in flight is now reported as cancelled instead of with a misleading network/S3 error. In particular, a cancelled backup or restore to S3 now reports `BACKUP_CANCELLED`/`RESTORE_CANCELLED` in `system.backups`. [#108673](https://github.com/ClickHouse/ClickHouse/pull/108673) ([jkartseva](https://github.com/jkartseva)). * Fixed the `_tags` virtual column for the `S3Queue` table engine: it was declared but never populated, so `SELECT _tags` always returned an empty map. It now returns the object tags, like the `S3` engine and the `s3()` table function. [#108676](https://github.com/ClickHouse/ClickHouse/pull/108676) ([groeneai](https://github.com/groeneai)). * Fixed SHOW TABLES and SELECT ... FROM system.tables returning different results for data lake catalog databases: tables that ClickHouse cannot read are now consistently hidden from both. [#109273](https://github.com/ClickHouse/ClickHouse/pull/109273) ([SmitaRKulkarni](https://github.com/SmitaRKulkarni)). * Fixed a server abort (or, in release builds, a silently unfinalized data file) when an `INSERT` into a Delta Lake table (`DeltaLakeLocal` / `deltaLake`) fails after the sink has already started writing, for example when the source query throws mid-insert or the query is cancelled. [#109328](https://github.com/ClickHouse/ClickHouse/pull/109328) ([groeneai](https://github.com/groeneai)). * Fix `Not found column or subcolumn c.null in block` (and `Bad cast from ColumnConst to ColumnNullable` in debug/sanitizer builds) when running `WHERE col IS NULL` / `IS NOT NULL` on a schema-evolved Iceberg column that was added after older data files were written, with `optimize_functions_to_subcolumns=1` (the default). [#109515](https://github.com/ClickHouse/ClickHouse/pull/109515) ([groeneai](https://github.com/groeneai)). * Fix `Host is empty in S3 URI` error when a scalar subquery is used as the URL argument of the `s3` table function (or as an argument of any other table function) in `CREATE TABLE ... AS SELECT` queries: scalar subqueries in table function arguments are now evaluated during query analysis. [#110254](https://github.com/ClickHouse/ClickHouse/pull/110254) ([CheSema](https://github.com/CheSema)). * Fix a logical error (server abort in debug/sanitizer builds) when a multi-command `ALTER` such as `UPDATE ..., DELETE ...` was issued on an Iceberg table. Such mutations are now rejected with a clear `NOT_IMPLEMENTED` error. [#110347](https://github.com/ClickHouse/ClickHouse/pull/110347) ([groeneai](https://github.com/groeneai)). * Fixed Iceberg tables producing duplicate field IDs after `ALTER TABLE ... ADD COLUMN` when the initial schema contains nested fields (`Tuple`/`Array`/`Map`). `last-column-id` now records the maximum assigned field ID including nested children, so a subsequently added column no longer reuses a nested field ID and reads no longer fail with `ICEBERG_SPECIFICATION_VIOLATION` (`Duplicate field id`). [#110884](https://github.com/ClickHouse/ClickHouse/pull/110884) ([groeneai](https://github.com/groeneai)). * Fix `DELETE`/`UPDATE` on an Iceberg table silently removing the wrong rows (or throwing `NOT_FOUND_COLUMN_IN_BLOCK`) when the table contains data files written both before and after `ALTER TABLE ... ADD COLUMN` and `optimize_move_to_prewhere` is enabled. [#111693](https://github.com/ClickHouse/ClickHouse/pull/111693) ([groeneai](https://github.com/groeneai)). * Fixed `made_current_at` in `system.iceberg_history` becoming `1970-01-01` for a retained snapshot after `ALTER TABLE ... EXECUTE expire_snapshots` on an Iceberg table. Also fixed `system.iceberg_history` returning no rows for a table whose metadata has no `snapshot-log`. [#111781](https://github.com/ClickHouse/ClickHouse/pull/111781) ([groeneai](https://github.com/groeneai)). * Fixed data corruption when inserting into an Iceberg table reached through a `MATERIALIZED VIEW`. The data file was written against the target's `CREATE`-time column types instead of its live Iceberg schema, so a `DateTime` value was stored as a bare Avro `int` and read back as microseconds, and with `DateTime64(0)` or `DateTime64(9)` the file could not be read back at all. As a consequence, `INSERT INTO ` against a target whose Iceberg schema is wider than the view's declared columns now raises `TYPE_MISMATCH` instead of writing a corrupt row. [#112359](https://github.com/ClickHouse/ClickHouse/pull/112359) ([groeneai](https://github.com/groeneai)). * Fixes writing Iceberg tables whose schema contains a tuple nested inside a tuple. The published table metadata replaced the inner tuple's field names with positional names `1`, `2`, ..., so with the default `Parquet` format the `INSERT` failed with `INCORRECT_DATA` and with `Avro` the column could not be read back. [#112480](https://github.com/ClickHouse/ClickHouse/pull/112480) ([groeneai](https://github.com/groeneai)). * Fixes a deadlock that permanently wedges a `Filesystem` or `HDFS` database when two queries resolve the same table name at the same time. Every later query against that database blocks forever and cannot be interrupted, so the server needs a restart to recover. [#112495](https://github.com/ClickHouse/ClickHouse/pull/112495) ([groeneai](https://github.com/groeneai)). * Fixes `DeltaLake` tables returning too few rows when the `WHERE` predicate contains a sub-expression the delta-kernel predicate translator cannot handle, positioned under a `NOT`. Such a sub-expression is now reported to delta-kernel as an explicit unknown predicate instead of as the end of the child iterator, which used to silently truncate the enclosing conjunction. [#112652](https://github.com/ClickHouse/ClickHouse/pull/112652) ([groeneai](https://github.com/groeneai)). * Fixed `ALTER TABLE ... EXECUTE remove_orphan_files` deleting a live `metadata/version-hint.text` of an Iceberg table. Its reachable-set entry was built by string concatenation assuming a trailing slash on a value which never has one, so the real hint file was never recognised as reachable and was removed once older than the `older_than` threshold. With `iceberg_use_version_hint = 1` the table then failed to read at all. [#113548](https://github.com/ClickHouse/ClickHouse/pull/113548) ([groeneai](https://github.com/groeneai)). * Fix `LOGICAL_ERROR` when filtering an Iceberg column that `ALTER TABLE ... MODIFY COLUMN` made `Nullable`. [#114521](https://github.com/ClickHouse/ClickHouse/pull/114521) ([groeneai](https://github.com/groeneai)). * Fixed a case where `ATTACH TABLE` carrying an `ENGINE = URL(...)` definition did not check the `TABLE ENGINE` grant of the engine the URL scheme dispatches to. A user holding only `TABLE ENGINE ON URL` could attach and then read a `File`, `S3`, `AzureBlobStorage` or `HDFS` backed table, which `CREATE TABLE` correctly rejects. [#115914](https://github.com/ClickHouse/ClickHouse/pull/115914) ([groeneai](https://github.com/groeneai)). * Fix outbound HTTP requests (e.g. \`aiGenerate\`, \`url\` table function, S3) failing with \`No route to host\` on hosts that advertise both IPv4 and IPv6 when only one address family is routable. The HTTP connection pool now falls back to the next resolved address when the first one fails with a network error, instead of propagating the error on the very first request. [#103786](https://github.com/ClickHouse/ClickHouse/pull/103786) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allow reading an Avro `array` of two-field `{key, value}` records into a ClickHouse `Map` column. This is the encoding Iceberg and Spark produce for a `MAP` with a non-string key (Avro native maps only support string keys). Previously such a file could be read as `Array(Tuple(key, value))` but not as `Map(K, V)`, which failed with `Type Map(...) is not compatible with Avro array`. [#109289](https://github.com/ClickHouse/ClickHouse/pull/109289) ([groeneai](https://github.com/groeneai)). * Fixed reading an Iceberg table with a metadata file whose version number is all digits but exceeds the 32-bit integer range (for example `v99999999999999999999.metadata.json`). Such a name previously produced an opaque `std::out_of_range` (`STD_EXCEPTION`) instead of a clear `BAD_ARGUMENTS` error. [#109619](https://github.com/ClickHouse/ClickHouse/pull/109619) ([groeneai](https://github.com/groeneai)). * Fixed the file name of gzip-compressed Iceberg metadata files. ClickHouse wrote them as `v{N}.gzip.metadata.json` (the HTTP `Content-Encoding` token), while the Iceberg spec expects the `gz` extension `v{N}.gz.metadata.json`. As a result Spark and other Hadoop-catalog readers could not find the metadata written by ClickHouse. ClickHouse now writes `v{N}.gz.metadata.json` and still reads the legacy `gzip` name for backward compatibility. [#109812](https://github.com/ClickHouse/ClickHouse/pull/109812) ([groeneai](https://github.com/groeneai)). * Fixes an Iceberg table whose write format is `Avro` serializing a field declared `"required": false` whose type is a `list`, `map` or `struct` as required, with no `["null", T]` union, so that the data file's own schema disagreed with the table metadata and other Iceberg readers saw a required field. Such a field is now written as the `["null", T]` union the Iceberg spec uses for an optional one. [#112648](https://github.com/ClickHouse/ClickHouse/pull/112648) ([groeneai](https://github.com/groeneai)). * Fixed `made_current_at` in `system.iceberg_history` reporting the earliest time a snapshot became current instead of the latest one, when the same `snapshot-id` appears more than once in the Iceberg `snapshot-log` (as a rollback leaves it). [#113598](https://github.com/ClickHouse/ClickHouse/pull/113598) ([groeneai](https://github.com/groeneai)). * Fixed a hang of up to one hour in `BACKUP`/`RESTORE ... TO S3`, GCS disks and the `s3` table function when the named collection sets `http_client = 'gcp_oauth'` and the Google OAuth2 token endpoint accepts the connection but never answers. The bearer token request no longer inherits the caller's data transfer request timeout; its response wait is bounded at 10 seconds, matching the STS `AssumeRole` request cap. [#115531](https://github.com/ClickHouse/ClickHouse/pull/115531) ([groeneai](https://github.com/groeneai)). * Fix a logical error in the filesystem cache (`Expected file ... not to exist`) that could occur when a background cache download failed with a non-`ErrnoException` (for example while `OPTIMIZE`-ing an Iceberg table), leaving an empty orphan cache file behind. [#110549](https://github.com/ClickHouse/ClickHouse/pull/110549) ([groeneai](https://github.com/groeneai)). * Fixed caches configured by a maximum number of entries (`SLRU` policy) silently stopping to admit new entries once that many entries had been accessed at least twice. Affected caches include the Iceberg and Paimon metadata file caches, the Parquet metadata cache, the text index caches, the vector similarity index cache, the compiled expression cache and the ObjectStorageQueue file status cache. [#115517](https://github.com/ClickHouse/ClickHouse/pull/115517) ([groeneai](https://github.com/groeneai)). * Fixed Paimon table reads that could fail while the mutable `snapshot/LATEST` hint was being replaced concurrently. Invalid or stale hints now safely fall back to snapshot listing. [#112010](https://github.com/ClickHouse/ClickHouse/pull/112010) ([JiaQiTang98](https://github.com/JiaQiTang98)). * Surface the real underlying exception when a zip archive cannot be unpacked because the input buffer (e.g. an S3 read buffer with `restricted_seek = true`) refuses a seek or read requested by minizip. Previously the actual error was hidden behind the generic `Couldn't unpack zip archive: Code = -100` / `Couldn't open zip archive` message. [#105103](https://github.com/ClickHouse/ClickHouse/pull/105103) ([groeneai](https://github.com/groeneai)). * Fix iceberg write for metadata without refs [#106896](https://github.com/ClickHouse/ClickHouse/pull/106896) ([AVMusorin](https://github.com/AVMusorin)). * Fixes an exception when inserting into an Iceberg table whose metadata was created by an external engine and omits the optional `snapshots`, `metadata-log`, or `snapshot-log` arrays. Such inserts now succeed instead of failing. [#107473](https://github.com/ClickHouse/ClickHouse/pull/107473) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed a `ReadBuffer is canceled. Can't read from it.` error that could occur when an upload to S3 (for example a backup) is retried after a transient read error. [#108573](https://github.com/ClickHouse/ClickHouse/pull/108573) ([groeneai](https://github.com/groeneai)). * Fix a `std::future_error` ("The associated promise has been destructed prior to the associated state becoming ready") that could surface, and abort the server in debug/sanitizer builds, when scheduling the final asynchronous S3/Azure multipart-upload completion task failed (for example under thread-pool exhaustion). The real scheduling error is now reported instead. [#108730](https://github.com/ClickHouse/ClickHouse/pull/108730) ([groeneai](https://github.com/groeneai)). * Fixed reading ORC timestamps beyond \~year 2262 into a `DateTime64` column with a scale coarser than 9. The native ORC reader used to convert every timestamp through a fixed `DateTime64(9)` intermediate, which overflows `Int64` at nanosecond scale and rejected such values with `VALUE_IS_OUT_OF_RANGE_OF_DATA_TYPE`, even when the requested `DateTime64` scale (for example `DateTime64(6)`, as produced by Iceberg) can represent the value. Timestamps are now read directly at the requested scale, and `date_time_overflow_behavior` is honored when even the target scale cannot hold the value. [#109302](https://github.com/ClickHouse/ClickHouse/pull/109302) ([groeneai](https://github.com/groeneai)). * Reading a Paimon table whose schema contains an unsupported nested type (for example a `ROW` field) now reports a clear `BAD_ARGUMENTS` error naming the unsupported type, instead of a bare `DB::Exception. (OK)` with error code 0 and no message. [#109762](https://github.com/ClickHouse/ClickHouse/pull/109762) ([groeneai](https://github.com/groeneai)). * Fix reading Iceberg tables partitioned by the same source column more than once (for example `PARTITIONED BY (hours(ts), ts)`). Such tables previously failed with `Cannot add column ...: column with this name already exists (ILLEGAL_COLUMN)`. [#109895](https://github.com/ClickHouse/ClickHouse/pull/109895) ([groeneai](https://github.com/groeneai)). * Data files written by ClickHouse into Iceberg tables now embed Iceberg field IDs for the Avro (`field-id`) and ORC (`iceberg.id`/`iceberg.required` type attributes) formats, and ORC `String` columns are written as ORC `string` instead of binary. Previously only Parquet embedded field IDs, so spec-compliant readers (e.g. Spark/iceberg-java) silently returned NULLs from ClickHouse-written Avro rows after a column rename, and could not read ClickHouse-written ORC files at all. [#109994](https://github.com/ClickHouse/ClickHouse/pull/109994) ([groeneai](https://github.com/groeneai)). * Fixed a logical error `ChunkInfoRowNumbers does not exist` on `OPTIMIZE TABLE` of an Iceberg table containing a data file in a non-Parquet format (e.g. `ORC`) newer than all position delete files. [#110107](https://github.com/ClickHouse/ClickHouse/pull/110107) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed reading large `Parquet` files from uncompressed `STORED` zip entries via `s3` and `file`. [#110495](https://github.com/ClickHouse/ClickHouse/pull/110495) ([Gudauu](https://github.com/Gudauu)). * Failing to start a background schedule pool (for example the `iceberg` pool, under global thread pool exhaustion) no longer aborts the server: the error is now a recoverable exception. Reading `system.background_schedule_pool` also no longer creates those pools as a side effect. [#111209](https://github.com/ClickHouse/ClickHouse/pull/111209) ([groeneai](https://github.com/groeneai)). * Fix DLF authentication in the Paimon REST catalog: the request signature was computed once and reused, so every catalog request after the first one failed with HTTP 401 and SHOW TABLES returned an empty list. Also fix HTTP 404 detection in table existence checks. [#112015](https://github.com/ClickHouse/ClickHouse/pull/112015) ([JiaQiTang98](https://github.com/JiaQiTang98)). * Fix a `LOGICAL_ERROR` (`partitions_count > 0`) when partitioning `Iceberg` data chunks emptied by position deletes, e.g. during compaction of a partitioned `Iceberg` table with fully deleted data files. [#112432](https://github.com/ClickHouse/ClickHouse/pull/112432) ([PedroTadim](https://github.com/PedroTadim)). * Fixed Azure batch object deletion recording no `system.blob_storage_log` Delete events when the batch request itself failed, leaving the whole batch unlogged. A Delete event is now recorded for each object on a batch-level failure. [#112873](https://github.com/ClickHouse/ClickHouse/pull/112873) ([arsenmuk](https://github.com/arsenmuk)). * Fixed reading `Paimon` tables partitioned by a `TIMESTAMP` column with a precision higher than milliseconds, which failed with `scale 6 is not supported, only support scale <= 3`. [#113401](https://github.com/ClickHouse/ClickHouse/pull/113401) ([JiaQiTang98](https://github.com/JiaQiTang98)). * Fixes a crash when reading a data lake table whose schema declares a column with an empty name. Malformed Iceberg metadata is rejected with `ICEBERG_SPECIFICATION_VIOLATION`; a schema supplied by a catalog, Delta Lake or Paimon is rejected with `AMBIGUOUS_COLUMN_NAME`. The check is unconditional, so an Iceberg table that merely retains an unused historical schema with an empty field name also becomes unreadable instead of aborting once that schema is read. [#114394](https://github.com/ClickHouse/ClickHouse/pull/114394) ([groeneai](https://github.com/groeneai)). * Reading an Iceberg table whose metadata describes a schema evolution that the Iceberg specification forbids no longer aborts the server. Four spec violations in `IcebergSchemaProcessor` were reported as `LOGICAL_ERROR`, which is treated as a failed assertion, or reached a fatal assertion unchecked; they now raise `ICEBERG_SPECIFICATION_VIOLATION`. [#114518](https://github.com/ClickHouse/ClickHouse/pull/114518) ([groeneai](https://github.com/groeneai)). * Fixed the `_time` virtual column of the `url`/`urlCluster` table functions always being `NULL` on musl-based builds, caused by non-portable parsing of the HTTP `Last-Modified` header. [#111619](https://github.com/ClickHouse/ClickHouse/pull/111619) ([thevar1able](https://github.com/thevar1able)). * Fixed a server that refuses to start after a table was created with `CREATE TABLE ... AS url('http://host/**/', ...)` while `allow_experimental_url_wildcard_from_index_pages` was enabled. Loading such a table's metadata re-evaluated the experimental check and failed with `SUPPORT_IS_DISABLED`, aborting startup. [#115095](https://github.com/ClickHouse/ClickHouse/pull/115095) ([groeneai](https://github.com/groeneai)). * A table of the `URL` database engine no longer discloses whether a local file exists to a user without the read source grant: `EXISTS TABLE`, which requires only the `SHOW TABLES` privilege, and the resolution of a table no longer probe the filesystem before the grant is confirmed. [#113029](https://github.com/ClickHouse/ClickHouse/pull/113029) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Apply `query_masking_rules` to messages appended via `Exception::addMessage` so URL-encoded credentials in `(in file/uri ...)` suffixes from `jdbc()`/`odbc()` errors are not leaked when masking rules are configured. [#106916](https://github.com/ClickHouse/ClickHouse/pull/106916) ([gaurav0107](https://github.com/gaurav0107)).

Formats and ingestion fixes

* Schema-less reads of Arrow `time32`/`time64` (e.g. `SELECT * FROM file(..., 'Arrow')`) now infer `Time64` instead of `DateTime64` Previously it inferred `DateTime64`. Anything downstream asserting the old inferred type will break on upgrade. Also adds support for exporting `Time`/`Time64` to Arrow with the appropriate Apache Arrow time type selected by precision. [#104316](https://github.com/ClickHouse/ClickHouse/pull/104316) ([Maoyao233](https://github.com/Maoyao233)). * Fixed possible wrong results or out-of-bounds reads when an `INSERT` is rolled back after a mid-batch error (for example in `Buffer` tables, asynchronous inserts, or the `Kafka`/`RabbitMQ`/`FileLog` engines) while lazy column replication (`enable_lazy_columns_replication`) is in effect. [#108935](https://github.com/ClickHouse/ClickHouse/pull/108935) ([groeneai](https://github.com/groeneai)). * Fixed a parser inconsistency where an `INSERT` column list accepted a qualified column matcher written as `t.* LIKE ''` / `t.* ILIKE ''` but rejected its canonical `t.COLUMNS('')` form, breaking the query format round-trip (and aborting the server in debug/sanitizer builds). [#109176](https://github.com/ClickHouse/ClickHouse/pull/109176) ([groeneai](https://github.com/groeneai)). * Fixed `INSERT` into a Microsoft Fabric / OneLake `DataLakeCatalog` table failing with `IncorrectEndpointError` (HTTP 400) by routing ADLS Gen2 (DFS) writes to the `.dfs` endpoint host instead of the `.blob` host. Note: with `remote_url_allow_hosts`, both the `.blob` (read) and `.dfs` (write) Fabric hosts must be allowlisted for `INSERT`. [#110290](https://github.com/ClickHouse/ClickHouse/pull/110290) ([zlareb1](https://github.com/zlareb1)). * Fix writes through data-lake table functions (e.g. `INSERT INTO FUNCTION icebergS3(...)`) producing parquet files whose string columns lost the `String` type annotation, making the written data unreadable for external readers such as Spark. Such writes now use the same format settings as writes through the table engine. [#110465](https://github.com/ClickHouse/ClickHouse/pull/110465) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed `system.session_log` becoming unreadable after an Arrow Flight login: the `interface` column is an `Enum8`, but the `ArrowFlight` value was missing from its enumeration, so a login over the Arrow Flight protocol wrote the raw value `10` and any `SELECT` reading such a row threw the exception `Unexpected value 10 in enum`. [#110758](https://github.com/ClickHouse/ClickHouse/pull/110758) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes schema inference for the `TSV` family of formats inferring `String` for a decimal value written with a leading zero, such as `0.0` or `0.5`, where `CSV` correctly infers `Float64`. A first row of such values was additionally consumed as a header by `TSV` header auto-detection and disappeared from the result. Also fixes `TSKV` inferring a numeric type for a value written using an escape sequence, such as `x=1\x2E5`, which then failed to parse. Values whose leading zero is padding of an integer, such as `007`, keep inferring `String` because the `TSV` value parser cannot read them back. [#112226](https://github.com/ClickHouse/ClickHouse/pull/112226) ([groeneai](https://github.com/groeneai)). * Fixed schema inference proposing a numeric type for text values the value parser cannot read back. Inference accepted a dangling exponent (`1e+`), a missing mantissa (`.`) or a repeated sign (`++inf`), which `readFloatTextPrecise` then rejected with `CANNOT_PARSE_NUMBER` when reading the inferred schema; on a `Dynamic` column this was an error at INSERT time. Inference now validates the number with the parser the reader actually uses. This also removes a case where the inferred type depended on `max_read_buffer_size`, and lets valid leading-zero exponent forms such as `0e5` infer `Float64` in TSV as they already do in CSV. It also fixes malformed exponent forms. [#112453](https://github.com/ClickHouse/ClickHouse/pull/112453) ([groeneai](https://github.com/groeneai)). * Fixes `KILL QUERY` and `max_execution_time` being ignored for minutes while a row-based input format was parsing one block. Row-based input formats now check for query cancellation every 8192 rows, so a killed insert (in particular a background async-insert flush, which parses up to `max_insert_block_size` rows per block) stops promptly instead of running to the end of the block. [#112646](https://github.com/ClickHouse/ClickHouse/pull/112646) ([groeneai](https://github.com/groeneai)). * Fixed the inferred column type depending on the order of previous queries for a source whose schema is cached. Settings that change an inferred type were missing from the schema inference cache key, so the first query's value decided the type every later query saw: `input_format_try_infer_exponent_floats`, `max_parser_depth`, `input_format_json_infer_array_of_dynamic_from_array_of_different_types`, and for `Parquet` also `input_format_parquet_local_time_as_utc`, `input_format_parquet_allow_geoparquet_parser`, `input_format_parquet_skip_columns_with_unsupported_types_in_schema_inference` and `schema_inference_make_json_columns_nullable`. Also registers the missing cache key getter for `Form` and makes the `Template` getter use its own row format's escaping rule. [#113533](https://github.com/ClickHouse/ClickHouse/pull/113533) ([groeneai](https://github.com/groeneai)). * Fixed formatting of a subquery argument of the `view` and `viewIfPermitted` table functions when the enclosing query has a trailing `SETTINGS` clause. Such a query was formatted as `view((SELECT ...))`, which cannot be parsed back, so the query failed the internal format-parse-format check and raised `Inconsistent AST formatting`. [#114658](https://github.com/ClickHouse/ClickHouse/pull/114658) ([groeneai](https://github.com/groeneai)). * Fix JSON data misdetected as TSKV during format auto-detection. [#106009](https://github.com/ClickHouse/ClickHouse/pull/106009) ([Avogar](https://github.com/Avogar)). * Exporting `Time`/`Time64` values outside a valid time-of-day (negative or `>= 24h`) to the `Arrow` format is now rejected with a clear error instead of writing invalid Arrow `time32`/`time64` data. [#107179](https://github.com/ClickHouse/ClickHouse/pull/107179) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed reading `Arrow`, `ArrowStream` and Arrow-based `ORC` files whose timestamp column carries a fixed numeric UTC offset (e.g. `+05:30`, `-08:00`, `00:00`) or the non-IANA marker `fixed` as its timezone, which previously failed with `Cannot load time zone ...` (`BAD_ARGUMENTS`). [#108633](https://github.com/ClickHouse/ClickHouse/pull/108633) ([groeneai](https://github.com/groeneai)). * Serialize a `Nullable(Tuple(...))` column as a single CSV field instead of flattening it into separate columns. Previously a non-null value was written as several CSV fields while a `NULL` was a single `\N`, giving a row-dependent column count that broke `CSVWithNames`/`CSVWithNamesAndTypes` headers and round-trips. [#108959](https://github.com/ClickHouse/ClickHouse/pull/108959) ([groeneai](https://github.com/groeneai)). * Fixed an excessive memory allocation during schema inference of the `MsgPack` format: a corrupted or fuzzed input whose array/map/string/binary header declares a huge element count no longer drives a single multi-gigabyte allocation (it is now rejected as malformed input). This fixes an `allocation-size-too-big` abort under sanitizers and an out-of-memory condition in release builds. [#109019](https://github.com/ClickHouse/ClickHouse/pull/109019) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` when writing a `LowCardinality(Time)` column to the `Arrow` format with `output_format_arrow_low_cardinality_as_dictionary = 1`. [#109730](https://github.com/ClickHouse/ClickHouse/pull/109730) ([groeneai](https://github.com/groeneai)). * Fixed reading a corrupted `Native`-format stream whose `Dynamic` type count or `JSON`/`Object` path count is close to `SIZE_MAX`: such malformed input is now rejected with a clear `INCORRECT_DATA` error instead of an uncaught `std::length_error`. [#110590](https://github.com/ClickHouse/ClickHouse/pull/110590) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a `LOGICAL_ERROR: Cannot sum Bools` (server abort in debug/sanitizer builds) when merging `sumMap`/`sumMapWithOverflow`/`sumMapFiltered` aggregate states with `Bool` values that were serialized in the old (version 0) state format. The same fix also corrects two silent wrong-result cases with version-0 `Bool` states: map keys of type `Bool` were not deduplicated across states, and zero-value compaction dropped the wrong entries. [#110922](https://github.com/ClickHouse/ClickHouse/pull/110922) ([groeneai](https://github.com/groeneai)). * Fix the `format` table function throwing `NOT_FOUND_COLUMN_IN_BLOCK` when the input data parses to zero rows (for example an empty JSON, or a GeoJSON `FeatureCollection` with no features). It now returns an empty result with the correct columns. [#111428](https://github.com/ClickHouse/ClickHouse/pull/111428) ([groeneai](https://github.com/groeneai)). * You can now read GeoParquet, Arrow and ArrowStream files whose WKT-encoded geometry column uses a non-uppercase type keyword (`point(1 2)`) or an empty container (`LINESTRING EMPTY`, `POLYGON()`, `POLYGON(())`). Such files previously failed to import with `BAD_ARGUMENTS` or `CANNOT_PARSE_NUMBER`, even though the same strings parse with the `readWKT` function. This also fixes reading back a file that stores the output of `wkt` for an empty geometry, which is spelled `LINESTRING()`. [#112020](https://github.com/ClickHouse/ClickHouse/pull/112020) ([groeneai](https://github.com/groeneai)). * Fixed `Unknown compression method` being thrown for the `auto` and `none` compression hints written in any letter case other than all-lowercase, for example `file('data.csv.gz', 'CSV', 'x String', 'AUTO')`. Codec names such as `GZIP` were already accepted case-insensitively; the two special hints now behave the same way. [#113631](https://github.com/ClickHouse/ClickHouse/pull/113631) ([groeneai](https://github.com/groeneai)). * Fixed reading a `Tuple` column whose first element is `NULL` in the `CustomSeparated`, `Regexp` and `Template` formats with the `CSV` escaping rule. A bare `Tuple` occupies one field per element there, so the leading `\N` field is that element and not the whole column. Previously it was consumed as the whole column, shifting the remaining element fields onto the following columns, so `CustomSeparated` could not read back its own output. [#114471](https://github.com/ClickHouse/ClickHouse/pull/114471) ([groeneai](https://github.com/groeneai)). * Fixed the `Regexp` input format silently discarding the unparsed rest of a matched field, which stored a truncated or fabricated value instead of reporting an error: `v=-1` into `UInt64` read `0`, and `v=2020-01-01junk` into `Date` read `1975-07-14` under the `JSON` rule. Malformed fields are now rejected under the `Escaped`, `CSV` and `JSON` rules, matching the formats those rules are documented as behaving like. Under `Raw`, a matched field containing a tab is now read as a whole instead of being truncated at the tab. [#114520](https://github.com/ClickHouse/ClickHouse/pull/114520) ([groeneai](https://github.com/groeneai)). * Fixed an `Inconsistent AST formatting` logical error (server abort in debug and sanitizer builds) for `CREATE INDEX` / `CREATE HYPOTHETICAL INDEX` expressions that could not survive a format-parse-format round trip, for example `CREATE HYPOTHETICAL INDEX i0 ON t0 ((a())) TYPE a` and `CREATE INDEX i0 ON t0 ((a, b).1) TYPE minmax`. [#109175](https://github.com/ClickHouse/ClickHouse/pull/109175) ([groeneai](https://github.com/groeneai)). * Fix a logical error (`Logical error: 'KeyCondition uses PREWHERE output'`) in the Parquet v3 reader that occurred when a format-level key condition references a column that is not read from the Parquet file. [#109267](https://github.com/ClickHouse/ClickHouse/pull/109267) ([groeneai](https://github.com/groeneai)). * Fix exception in predefined HTTP handler on absent header [#108324](https://github.com/ClickHouse/ClickHouse/pull/108324) ([vitlibar](https://github.com/vitlibar)). * Fixed a possible `Logical error: 'ReadBuffer is canceled. Can't read from it.'` when a row-based input format (e.g. `TSV`/`CSV`) failed to parse input and the underlying read had already been canceled (for example a malformed HTTP chunk or a truncated body). Building the verbose parse diagnostics no longer reads from a canceled buffer. [#109708](https://github.com/ClickHouse/ClickHouse/pull/109708) ([groeneai](https://github.com/groeneai)). * The `Values` format no longer records parse errors in `system.errors` and `system.error_log` when streaming parsing successfully falls back to SQL expression parsing. [#111141](https://github.com/ClickHouse/ClickHouse/pull/111141) ([pamarcos](https://github.com/pamarcos)). * Fix `SYSTEM REFRESH` on a stopped `RabbitMQ` table sometimes failing to consume the queued backlog. A REFRESH grants exactly one out-of-order streaming cycle, but that cycle could return immediately with zero rows because the AMQP delivery loop had only just been started asynchronously, wasting the single permit and leaving the messages unconsumed until the next `SYSTEM START`. [#111480](https://github.com/ClickHouse/ClickHouse/pull/111480) ([groeneai](https://github.com/groeneai)). * Fixes a `NATS` table with `nats_stream` set silently consuming nothing after the NATS server is restarted. The `JetStream` subscription is now re-established automatically instead of requiring `DETACH TABLE` and `ATTACH TABLE`. [#112828](https://github.com/ClickHouse/ClickHouse/pull/112828) ([groeneai](https://github.com/groeneai)). * Reject an absurdly large `kafka_num_consumers` with a clear error instead of failing an allocation inside the `Kafka` table engine. Previously, with `kafka_disable_num_consumers_limit` enabled, such a value produced a `std::length_error` exception. [#114764](https://github.com/ClickHouse/ClickHouse/pull/114764) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Arrow Flight server: accept `Basic` authentication credentials encoded in Base64 without padding, as sent by Go-based Flight clients such as the ADBC Flight SQL driver, and return a proper authentication error instead of a generic `Unexpected error in RPC handling` when the `authorization` header is malformed. [#115085](https://github.com/ClickHouse/ClickHouse/pull/115085) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Arrow Flight server: reject malformed Basic authentication credentials that omit the username/password separator. [#115347](https://github.com/ClickHouse/ClickHouse/pull/115347) ([alexey-milovidov](https://github.com/alexey-milovidov)).

Data type, function, and serialization fixes

* Fix a logical error `Stream .variant_discr ... is not found` (server abort in debug and sanitizer builds) when `ALTER TABLE ... MODIFY COLUMN` changes a column into a type with dynamic subcolumns (for example `Variant` into `Dynamic`) on a table with Wide parts, and the part is later read or merged. [#110204](https://github.com/ClickHouse/ClickHouse/pull/110204) ([groeneai](https://github.com/groeneai)). * Fixed reading a subcolumn of a `Nullable(Tuple(...))` column whose element is a non-nullable `LowCardinality(T)` when the same query also reads the parent subcolumn. On a Compact part the parent returned wrong values, or the query failed with `Unexpected return type from assumeNotNull`, or the server was killed by a `string_view` hardening assertion, because the extracted subcolumn was deserialized into a `LowCardinality(Nullable(T))` buffer whose substreams were then shared with the parent's read. [#112232](https://github.com/ClickHouse/ClickHouse/pull/112232) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` "Not-ready Set is passed as the second argument for function `globalNullIn`" and the loss of the query's real error message, which could happen when statistics part pruning speculatively executed an `IN`/`GLOBAL IN` subquery during index analysis and that subquery failed. [#114121](https://github.com/ClickHouse/ClickHouse/pull/114121) ([groeneai](https://github.com/groeneai)). * Fixed `Cannot read all data of type FixedString` when reading the `product_quantization_codebook` subcolumn of a column with a `Quantized('product', ...)` codec. The per-part codebook is written after the data of all granules, so it is not delimited by marks, and a read whose mark range ended before the part's final mark could read it short. [#112818](https://github.com/ClickHouse/ClickHouse/pull/112818) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Stopped writing a part whose `Map` layout contradicts its own metadata, for a `Map` value inside a `Dynamic` column. Tables differing only in `map_serialization_version` shared one cached serialization object, so a part could get the `basic` Map layout while its metadata declared `with_buckets`, and reading it later failed with `LOGICAL_ERROR: Stream ...buckets_info ... is not found`. Parts an earlier version already wrote this way stay unreadable and have to be dropped. [#113514](https://github.com/ClickHouse/ClickHouse/pull/113514) ([groeneai](https://github.com/groeneai)). * Fixed `ALP`, `FPC` and `GCD` omitting their type-derived parameter from the codec hash. In a Compact part the writer shares one compressed stream per distinct codec hash, so two columns of different value width under the same codec collided and shared one codec object carrying the wrong width. This rejected valid inserts with `CANNOT_COMPRESS` when the widths disagreed, and otherwise compressed one of the columns worse than it should have. [#114189](https://github.com/ClickHouse/ClickHouse/pull/114189) ([groeneai](https://github.com/groeneai)). * Fix incorrect index pruning when a table's sorting key wraps a `Date` column in `toDateTime` (for example `ORDER BY toDateTime(date_column)`) and a query filters on the original column with a comparison like `WHERE date_column >= '...'`. `toDateTime(Date)` overflows for `Date` values beyond the `DateTime` range (after `2106-02-07`), so the stored key is non-monotonic; ClickHouse will no longer use primary key pruning for this key/predicate combination because doing so could drop granules that contain matching rows. [#101814](https://github.com/ClickHouse/ClickHouse/pull/101814) ([nihalzp](https://github.com/nihalzp)). * Fixed `ALTER MODIFY COLUMN` failing when converting a `Nullable` column to a different type with a `DEFAULT` (e.g. `Nullable(UInt8)` → `String`,`LowCardinality(String)`, or `Array(UInt8)`). [#102156](https://github.com/ClickHouse/ClickHouse/pull/102156) ([il9ue](https://github.com/il9ue)). * Fixed incorrect SQL literal escaping in `StorageSQLite` and `sqlite()` table function when pushing `WHERE` predicates to SQLite: single quotes and control characters (`\n`, `\r`, `\t`, `\`) were escaped with backslashes, which SQLite does not interpret, causing syntax errors or wrong query results. Also fixed the escaping of string literals pushed down to PostgreSQL: strings nested inside `IN` lists kept ClickHouse escaping, and control characters were sent as backslash sequences that PostgreSQL reads back as different bytes. [#104217](https://github.com/ClickHouse/ClickHouse/pull/104217) ([tiandiwonder](https://github.com/tiandiwonder)). * Fix `NOT_FOUND_COLUMN_IN_BLOCK` exception in `INNER JOIN ... ON arrayJoin(...) = ...` queries when `query_plan_convert_join_to_in` is enabled and the SELECT references the source array column. [#104809](https://github.com/ClickHouse/ClickHouse/pull/104809) ([tiandiwonder](https://github.com/tiandiwonder)). * Fix IN and NOT IN expressions with non-constant right-hand side operands referencing columns from the current row, and align new analyzer tuple right-hand side handling with existing ClickHouse IN semantics. Under the old analyzer, a bare source column as the right-hand side (`x IN (arr)`) still resolves as a table name and stays out of scope. [#104993](https://github.com/ClickHouse/ClickHouse/pull/104993) ([niyue](https://github.com/niyue)). * Fixed `JOIN` against `EmbeddedRocksDB`, `Redis`, and `KeeperMap` tables that could silently return zero rows when `DirectKeyValueJoin` is used with `String` or `Nullable(String)` keys. [#105253](https://github.com/ClickHouse/ClickHouse/pull/105253) ([vdimir](https://github.com/vdimir)). * Fix a `LOGICAL_ERROR` ("Unexpected expression in JOIN ON section. Expected boolean (UInt8), got 'Nothing'") when a non-equi `JOIN ... ON` predicate references a `Nothing`-typed column, such as one produced by `ARRAY JOIN []`. [#106981](https://github.com/ClickHouse/ClickHouse/pull/106981) ([groeneai](https://github.com/groeneai)). * Fixed an `Inconsistent AST formatting` logical error that could abort the server in debug and sanitizer builds when an aliased lambda was used as the operand of an access operator (`.N` tuple element or `[]` array element) at a non-first position of an expression list, e.g. `SELECT 1, ((p0, p1) -> p0 AS a7).4[3] FROM t`. [#107092](https://github.com/ClickHouse/ClickHouse/pull/107092) ([groeneai](https://github.com/groeneai)). * A cluster table function (`urlCluster`, `fileCluster`, `s3Cluster`, ...) nested inside another distributed query, such as `clusterAllReplicas(..., urlCluster(...))`, is now rejected with a `BAD_ARGUMENTS` error instead of failing with a logical error (`Distributed task iterator is not initialized`). [#107107](https://github.com/ClickHouse/ClickHouse/pull/107107) ([groeneai](https://github.com/groeneai)). * Fixed the `values` table function failing with `ARGUMENT_OUT_OF_BOUND` when a decimal literal that is not exactly representable in a narrow floating-point column (such as `0.1` for a `Float32` column) is used, e.g. `SELECT * FROM values('x Float32', 0.1)`. Such values are now accepted and converted to the nearest representable value, consistent with `CAST` and `INSERT ... VALUES`. [#108055](https://github.com/ClickHouse/ClickHouse/pull/108055) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `mapFilter`, `mapSort` (and its variants) and `mapConcat` dropping `LowCardinality` from the key and value types of a `Map`. Previously `mapFilter` over a `Map(LowCardinality(String), String)` column returned `Map(String, String)`, which could corrupt the metadata of a table created via `CREATE TABLE ... AS SELECT` and make `CHECK TABLE` fail. [#108057](https://github.com/ClickHouse/ClickHouse/pull/108057) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a cancelled or `KILL`ed `INSERT` continuing to run for a long time while building a skip index (for example an unbounded `set(0)` index on a high-cardinality column). The index build now stops promptly when the query is cancelled. [#108351](https://github.com/ClickHouse/ClickHouse/pull/108351) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed a `SELECT` from the `primes` table function not responding to cancellation: with a large `step` (or `limit`) the query could keep running for a long time after `KILL QUERY` or a timeout. It now stops promptly. [#108353](https://github.com/ClickHouse/ClickHouse/pull/108353) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed parsing of data-type names whose name contains the substring `INT` (for example a function-like name such as `quantileInterpolatedWeighted` used in a data-type position). Such names were mistaken for MySQL integer types and had their first `(...)` argument group silently consumed as a display-width modifier, which could break the query formatting round-trip. [#108354](https://github.com/ClickHouse/ClickHouse/pull/108354) ([groeneai](https://github.com/groeneai)). * Fixed an inconsistency where a malformed `ARRAY JOIN` followed by a comma and a parenthesized table list was misparsed as a cross join instead of being rejected. [#108365](https://github.com/ClickHouse/ClickHouse/pull/108365) ([Algunenano](https://github.com/Algunenano)). * Fix a `LOGICAL_ERROR` ("Invalid action query tree node ...") exception when a distributed query both deduplicated structurally-identical duplicate-`ALIAS` columns and referenced a table function with a `key = value` named-collection argument (for example `oss(s3_conn, filename = '...')`). [#108435](https://github.com/ClickHouse/ClickHouse/pull/108435) ([groeneai](https://github.com/groeneai)). * Fixed a logical error (an exception in release builds, a server abort in debug and sanitizer builds) that could happen for a function call with more than one `LowCardinality` argument over a distributed table, for example `concatAssumeInjective((SELECT toLowCardinality('p')), s)` used as a `GROUP BY` key with `remote()`. The messages were `Default functions implementation for LowCardinality is supported only with a single LowCardinality argument` or `Expected the argument ... to have N rows, but it has M`. [#108871](https://github.com/ClickHouse/ClickHouse/pull/108871) ([groeneai](https://github.com/groeneai)). * Fixed `use_client_time_zone` being ignored for `DateTime`/`DateTime64` string literals interpreted on the server (asynchronous `INSERT`, `SELECT` literals). The client now propagates its local time zone as `session_timezone` when `use_client_time_zone` is enabled, so server-side parsing matches the synchronous `INSERT` path. [#109051](https://github.com/ClickHouse/ClickHouse/pull/109051) ([groeneai](https://github.com/groeneai)). * Fix `max_execution_time` (with `timeout_overflow_mode = 'throw'`) sometimes never cancelling a query: when the internal timeout watcher was already waiting for a query with a later deadline, a query with an earlier deadline registered afterwards could be missed entirely, letting it run long past its time limit. [#109792](https://github.com/ClickHouse/ClickHouse/pull/109792) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed `NOT_FOUND_COLUMN_IN_BLOCK` when a query over a `Distributed` table uses `_shard_num` (or `shardNum()`) that becomes `Nullable` (for example under `FULL JOIN` with `join_use_nulls = 1`) together with `ORDER BY`, with the analyzer enabled. [#109798](https://github.com/ClickHouse/ClickHouse/pull/109798) ([groeneai](https://github.com/groeneai)). * Fixed `NOT_IMPLEMENTED` error ("Method getDataAt is not supported for Nullable(String)") that could be thrown by a join with `enable_join_runtime_filters = 1` (on by default since 26.2) when the build-side join key was `LowCardinality(Nullable(...))` with NULLs and the runtime filter fell back to its bloom filter. [#109824](https://github.com/ClickHouse/ClickHouse/pull/109824) ([groeneai](https://github.com/groeneai)). * Fixed a server crash when a lambda expression was passed where a higher-order function expects a concrete value (for example the accumulator of `arrayFold`, as in `arrayFold(lambda, arr, another_lambda)`). Such queries are now rejected with `ILLEGAL_TYPE_OF_ARGUMENT` instead of crashing when `enable_analyzer = 0`. [#109840](https://github.com/ClickHouse/ClickHouse/pull/109840) ([groeneai](https://github.com/groeneai)). * Reject the `unnest` alias of `arrayJoin` in row policy filters, closing a gap where such a policy could raise a `column->size() == num_rows` logical error at read time. [#109973](https://github.com/ClickHouse/ClickHouse/pull/109973) ([Algunenano](https://github.com/Algunenano)). * Fixed skip indexes defined on `Tuple` subcolumns not being used when the field is accessed via `tupleElement(t, 'name')`, `tupleElement(t, N)`, or `t.N` while the full tuple is also read in the same query (e.g. `SELECT *`). All of these forms now prune granules the same way as the named subcolumn access `t.name`. [#110056](https://github.com/ClickHouse/ClickHouse/pull/110056) ([groeneai](https://github.com/groeneai)). * Fixed `BAD_GET` error when comparing an `Array(String)` column with an array literal (`arr = ['x']`) when the column has a `text`, `tokenbf_v1` or `ngrambf_v1` skip index. Such a comparison is now treated as a full scan by the index instead of failing the whole query. [#110057](https://github.com/ClickHouse/ClickHouse/pull/110057) ([groeneai](https://github.com/groeneai)). * Fixed a spurious `ILLEGAL_COLUMN` error from `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` when the column already exists at apply time. This happened when the same column was added twice with `IF NOT EXISTS` in one statement, or when a concurrent `ALTER` added the column between the prepare and apply phases. [#110080](https://github.com/ClickHouse/ClickHouse/pull/110080) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` (`Bad cast from type ColumnLowCardinality to ColumnString`) when `identity()` (or a scalar subquery result) wraps a value containing a nested `LowCardinality` and the query uses `WITH TOTALS`/`WITH ROLLUP`. [#110138](https://github.com/ClickHouse/ClickHouse/pull/110138) ([groeneai](https://github.com/groeneai)). * Fixed a `NOT_FOUND_COLUMN_IN_BLOCK` error when using `GROUP BY ALL` over a tuple expression together with `ORDER BY`. [#110206](https://github.com/ClickHouse/ClickHouse/pull/110206) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a `NOT_FOUND_COLUMN_IN_BLOCK` error on `INSERT` into a table that has a `CHECK` constraint referencing a subcolumn (such as `x.null` of a `Nullable` column or `arr.size0` of an `Array`). [#110208](https://github.com/ClickHouse/ClickHouse/pull/110208) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix wrong query results from the optimize\_and\_compare\_chain optimization (enabled by default): a condition derived by transitivity from a \< b AND b \< c could use different comparison semantics than the original conditions when the chain mixes types with pair-specific conversion rules (e.g. Enum vs String, Decimal vs Float64, FixedString of different widths). Transitive conditions are now derived only within groups of types sharing a single comparison order. [#110621](https://github.com/ClickHouse/ClickHouse/pull/110621) ([yariks5s](https://github.com/yariks5s)). * Fixed `optimize_injective_functions_in_group_by` changing query results for `GROUP BY GROUPING SETS` and `GROUP BY ... WITH TOTALS`. The setting is documented as result-preserving, but eliminating an injective function of a grouping key produced a wrong value (or dropped a row) for rows where that key is absent from the aggregated set (a `GROUPING SETS` non-member set) or for the `WITH TOTALS` row. The wrong value could coincide with a genuine key value, making a super-aggregate row indistinguishable from a real group in the same result set. [#110721](https://github.com/ClickHouse/ClickHouse/pull/110721) ([groeneai](https://github.com/groeneai)). * Fix a server hang when multiplying a `median`/`quantile` (ReservoirSampler-backed) aggregate function state by a huge integer constant (e.g. `medianState(x) * 18446744073709551615`). The query became unresponsive to cancellation and could run indefinitely. [#110779](https://github.com/ClickHouse/ClickHouse/pull/110779) ([groeneai](https://github.com/groeneai)). * Fixes a crash (null pointer dereference during query analysis) when `getSubcolumn` is called on a `JSON` argument with an internal type-hint subcolumn name, that is one starting with a backtick type marker. [#110940](https://github.com/ClickHouse/ClickHouse/pull/110940) ([groeneai](https://github.com/groeneai)). * Fixed `map[key]` returning the value of a later duplicate key instead of the first one. When a `Map` column contained duplicate keys, the result of `map[key]` could depend on preceding rows in the same block and on the `optimize_functions_to_subcolumns` setting, so a `SELECT` and a `WHERE` over the same row could disagree. [#111246](https://github.com/ClickHouse/ClickHouse/pull/111246) ([groeneai](https://github.com/groeneai)). * Fix a logical error `!rhs_literal->getValue().isNull()` in the logical-expression optimizer when an `AND` chain compares a column against a NULL-valued constant of a non-`Nullable` type (for example a NULL-valued `Variant`) with `use_variant_default_implementation_for_comparisons = 0`. [#111292](https://github.com/ClickHouse/ClickHouse/pull/111292) ([groeneai](https://github.com/groeneai)). * Fixed wrong results when a query has a `set` skip index and a predicate of the shape `(x AND ) OR y`, where the NULL is produced by a function (e.g. `toInt64OrNull('x')`, `nullIf(1, 1)`, `CAST(NULL AS Nullable(Int64))`). Such a value was wrapped by `__bitWrapperFunc` and propagated `NULL` instead of the "unknown" mask, so the set index could prune granules that actually match and the query returned fewer rows than it should. [#111604](https://github.com/ClickHouse/ClickHouse/pull/111604) ([groeneai](https://github.com/groeneai)). * Fix `CREATE TABLE`/`ALTER TABLE` with the analyzer accepting a column `DEFAULT`/`MATERIALIZED` expression that references a virtual column such as `_table` or `_database`. Inserting into such a table failed with `NOT_FOUND_COLUMN_IN_BLOCK`, and a `MATERIALIZED` column (which cannot be supplied explicitly) made the table permanently un-insertable. These expressions are now rejected at definition time, matching the behavior without the analyzer. `ALIAS` columns (read-time) and `EPHEMERAL` columns (non-stored insert inputs) over virtual columns keep working. [#111776](https://github.com/ClickHouse/ClickHouse/pull/111776) ([groeneai](https://github.com/groeneai)). * Fixed `UNKNOWN_QUERY_PARAMETER` when a parameterized view is called with a subquery-valued argument whose type is `Array`, `Tuple` or `LowCardinality`, e.g. `SELECT * FROM v(param = (SELECT groupArray(x) FROM t))`. [#111790](https://github.com/ClickHouse/ClickHouse/pull/111790) ([groeneai](https://github.com/groeneai)). * Fixed three bugs in the analyzer rewrites that turn a chain of comparisons into `IN`/`NOT IN` (`x = c1 OR x = c2 OR ...`, `x != c1 AND x != c2 AND ...`, and `has(const_array, x)`). For a `Variant` expression with `use_variant_default_implementation_for_comparisons = 0` the rewrite changed the expression result type from `UInt8` to `Nullable(UInt8)`, which raised a `LOGICAL_ERROR` from the query-tree validation check in debug and sanitizer builds. For an expression with a dynamic structure (`Dynamic`, `Array(Dynamic)`, `JSON`) the rewrite made a working query fail with `Illegal type ... of argument of function in`, because `IN` rejects such arguments. Both directions also returned wrong results when a constant does not convert losslessly to the expression's type, such as a `DateTime` constant with a time of day compared against a `Date` column. [#111924](https://github.com/ClickHouse/ClickHouse/pull/111924) ([groeneai](https://github.com/groeneai)). * Fix `quantileDeterministic`, `quantilesDeterministic` and `medianDeterministic` returning different results for the same data on every run when the query uses parallel replicas, a distributed table, or external aggregation, or when the states are stored in a table (including inside a `Dynamic` value). States written before this version under the unversioned type spelling keep the previous behavior, since their layout cannot change. [#112052](https://github.com/ClickHouse/ClickHouse/pull/112052) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes wrong results caused by a vector-search query poisoning the query condition cache. A `SELECT ... WHERE ORDER BY LIMIT n` query over a table with a `vector_similarity` index could record granules as not matching ``, so a later ordinary `SELECT ... WHERE ` on the same table silently returned fewer rows. Vector-search reads no longer write to the query condition cache. [#112086](https://github.com/ClickHouse/ClickHouse/pull/112086) ([groeneai](https://github.com/groeneai)). * Fixes `serialize_string_in_memory_with_zero_byte` being dropped from the serialized query plan. With `serialize_query_plan = 1`, a cluster running with the setting disabled could diverge on the in-memory string representation between the initiator and the node executing the deserialized plan, because the setting was read on deserialization but never written on serialization and therefore always resolved to its default on the executing node. [#112131](https://github.com/ClickHouse/ClickHouse/pull/112131) ([groeneai](https://github.com/groeneai)). * Fixed a wrong result for a `JOIN` whose `ON` expression contains a constant conjunct, for example `ON l.id = r.id AND CAST(NULL AS Nullable(UInt8))`. The constant was silently dropped by query plan optimization, so the join ran as if it were not there: `INNER`, `LEFT SEMI` and `RIGHT SEMI` joins returned matched rows where a never-true condition must return none, and `RIGHT ANTI` returned no rows instead of all of them. [#112155](https://github.com/ClickHouse/ClickHouse/pull/112155) ([vdimir](https://github.com/vdimir)). * Fixes wrong results when a parameterized view is called with an argument that is not a `parameter = value` assignment. With `enable_analyzer = 0`, and in `EXPLAIN SYNTAX` at default settings, calls such as `pv(concat(name, '!'))`, `pv(name, 'a')`, `pv(name != 'a')` and `pv(tuple(name = 'a'))` silently bound a view parameter to an unrelated expression instead of reporting `UNKNOWN_QUERY_PARAMETER` as the analyzer does. Such calls are now rejected identically on both execution paths. [#112194](https://github.com/ClickHouse/ClickHouse/pull/112194) ([groeneai](https://github.com/groeneai)). * Fixes a bug where a `CREATE VIEW` or `CREATE FUNCTION` whose definition calls `substr`, `mid` or `byteSlice` in a form the `substring` grammar cannot re-parse is accepted, but the metadata ClickHouse writes for it is not valid SQL. Reading that definition afterwards fails with `SYNTAX_ERROR`, and at server startup it aborts metadata loading, so the server cannot start at all. Such definitions now round-trip, and existing metadata files of the form `substring(x, a, b, c)` become readable again, so an affected server starts up without manual intervention. [#112195](https://github.com/ClickHouse/ClickHouse/pull/112195) ([groeneai](https://github.com/groeneai)). * Fixed wrong results, wrongly ordered `ORDER BY` output and a `BAD_TYPE_OF_FIELD` exception when the primary key is an `Array` or `Tuple` and the query applies `plus`, `minus`, `multiply`, `divide` or `intDiv` to it. Arithmetic on a compound value is evaluated element-wise, while the comparison of a compound value is lexicographic, so primary key analysis no longer treats such an expression as monotonic. [#112339](https://github.com/ClickHouse/ClickHouse/pull/112339) ([groeneai](https://github.com/groeneai)). * Fixed `Code: 62 ... You must not specify ANY or ALL for PASTE JOIN. (SYNTAX_ERROR)` raised by a remote server for a query containing a `PASTE JOIN` that is shipped to another server, for example when the left table is a `Distributed` table or a `*Cluster` table function. The formatter printed `ALL PASTE JOIN`, a spelling the parser rejects, so the receiving server could not parse the query it was sent. [#112404](https://github.com/ClickHouse/ClickHouse/pull/112404) ([groeneai](https://github.com/groeneai)). * Fixes a column alias list such as `(SELECT 1) AS t(x)` or `WITH t(x) AS (SELECT 1)` not being visible inside the `view` table function, where a reference to `t.x` failed with `UNKNOWN_IDENTIFIER` although the same construct works outside `view`. [#112690](https://github.com/ClickHouse/ClickHouse/pull/112690) ([groeneai](https://github.com/groeneai)). * Fixes a logical error in query formatting when an `ARRAY JOIN` clause carries no expressions. Such a clause is now rejected while parsing with `SYNTAX_ERROR`, consistent with an empty `USING`, instead of being accepted and formatted into text that cannot be parsed back. [#112699](https://github.com/ClickHouse/ClickHouse/pull/112699) ([groeneai](https://github.com/groeneai)). * Fixed `connectionId` throwing `Context has expired` when expression actions prepared for a subquery run after their build context has been released. [#112870](https://github.com/ClickHouse/ClickHouse/pull/112870) ([clickgapai](https://github.com/clickgapai)). * Fixes a logical error and wrong results when a non equi `JOIN ON` condition that is evaluated during the join contains `arrayJoin`. A `hash`, `parallel_hash` or `grace_hash` join now rejects such a condition with `INVALID_JOIN_ON_EXPRESSION`. Where the expansion depends on one side only, move it into an `ARRAY JOIN` in a subquery before the join; a condition whose `arrayJoin` argument reads columns from both sides has to be restructured. [#112889](https://github.com/ClickHouse/ClickHouse/pull/112889) ([groeneai](https://github.com/groeneai)). * Fixed `geohashesInBox` spending an unbounded amount of time on a box of zero area, such as `geohashesInBox(0., 0., 180., 0., 12)`. Such a query returned the correct single geohash but burned roughly 5e8 empty loop iterations per row and ignored both `max_execution_time` and `KILL QUERY`. [#112976](https://github.com/ClickHouse/ClickHouse/pull/112976) ([groeneai](https://github.com/groeneai)). * Reject a `SETTINGS` change marked as written without a value when it carries a value other than `true`, and never elide the value of such a change when formatting a query. Previously a crafted AST JSON payload could execute a `Bool` setting with `false` while `system.query_log` and `formatQueryFromJSON` showed the valueless form. [#113025](https://github.com/ClickHouse/ClickHouse/pull/113025) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `UNKNOWN_IDENTIFIER` when selecting from a `VIEW` whose declared column types differ from the types its inner query produces, for example `SELECT sum(length(arr)) FROM v` where `v` declares `arr Array(UInt8)` over a `String` column. With the default `optimize_functions_to_subcolumns = 1` the optimizer rewrote `length(arr)` into a read of the `arr.size0` subcolumn and forwarded that name into the inner query, which has no such subcolumn. [#113057](https://github.com/ClickHouse/ClickHouse/pull/113057) ([groeneai](https://github.com/groeneai)). * Fixed cancellation of `postgresql` table function and PostgreSQL engine reads. Cancelling while the read was still starting up could race on the transaction pointer, and the cancellation could be lost entirely, leaving the query running until PostgreSQL finished it. A cancel that reaches PostgreSQL before the `COPY` statement has begun executing can still be dropped. [#113150](https://github.com/ClickHouse/ClickHouse/pull/113150) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` (`Next task callback is not set for query`) when a `SQL SECURITY DEFINER` or `SQL SECURITY NONE` view over a cluster table function (`s3Cluster`, `urlCluster`, `fileCluster`) was read as a secondary query, for example through `remote` or `clusterAllReplicas`. The unsupported nesting is now rejected with a normal query error instead of aborting the server in debug and sanitizer builds. [#113371](https://github.com/ClickHouse/ClickHouse/pull/113371) ([groeneai](https://github.com/groeneai)). * Fix a logical error `IDENTIFIER is not a table expression` when a remote-family table function with a `view(...)` argument is used on the left side of a `JOIN ... USING` whose key is a `SELECT` list alias, with `analyzer_compatibility_join_using_top_level_identifier = 1`. [#113400](https://github.com/ClickHouse/ClickHouse/pull/113400) ([groeneai](https://github.com/groeneai)). * Fixed `Not-ready Set is passed as the second argument for function 'in'` when a query filters `system.databases` on the `name` column with a subquery inside `indexHint`, for example `SELECT count() FROM system.databases WHERE indexHint(name IN (SELECT 'foo'))`. Such a query was rejected with a `LOGICAL_ERROR` on release builds and aborted the server on debug and sanitizer builds. [#113556](https://github.com/ClickHouse/ClickHouse/pull/113556) ([groeneai](https://github.com/groeneai)). * Fixed a query over a `Variant` column ignoring `max_execution_time` and `KILL QUERY`. Calling a function with several `Variant` arguments resolved the function once per alternative of every argument, and neither of the two loops doing that work checked for cancellation, so such a query could not be stopped until it finished. [#113612](https://github.com/ClickHouse/ClickHouse/pull/113612) ([groeneai](https://github.com/groeneai)). * Fixes `ALTER TABLE ... MODIFY COLUMN` failing with `Cannot specify codec for column type ALIAS` when one statement both turns an `ALIAS` column into a physical one (`DEFAULT` or `MATERIALIZED`) and specifies a `CODEC`. [#113853](https://github.com/ClickHouse/ClickHouse/pull/113853) ([groeneai](https://github.com/groeneai)). * Fixed a server crash when joining with a constant `ON` expression against a right side that contains an `ARRAY JOIN`. Reading a lazily replicated right-side column used a freed `ColumnReplicated`. [#113901](https://github.com/ClickHouse/ClickHouse/pull/113901) ([groeneai](https://github.com/groeneai)). * Fixes a memory leak with `optimize_aggregation_in_order = 1` when the table sorting key is a strict prefix of the `GROUP BY` key and an aggregate function whose state owns heap memory is used, such as `quantileDD`. Server memory grows with every such query until restart. [#114010](https://github.com/ClickHouse/ClickHouse/pull/114010) ([groeneai](https://github.com/groeneai)). * Fixed `isProbablePrime` on `UInt128`/`UInt256` ignoring `max_execution_time` and `KILL QUERY`. A block of wide values ran to completion inside a single function call with no cancellation point, so a query could run for minutes past its deadline. [#114032](https://github.com/ClickHouse/ClickHouse/pull/114032) ([groeneai](https://github.com/groeneai)). * Fixed `Code: 46. DB::Exception: Unknown function exists. (UNKNOWN_FUNCTION)` thrown when a `PREWHERE` clause contains an `IN (subquery)` predicate and `rewrite_in_to_join = 1` (or `make_distributed_plan = 1`, which force-enables it) is set. The same query spelled with `WHERE` worked correctly. [#114067](https://github.com/ClickHouse/ClickHouse/pull/114067) ([RohithPariki](https://github.com/RohithPariki)). * Fixed a `LOGICAL_ERROR` in `arrayAutocorrelation` when the argument is a non-empty array whose element type is `Nothing`, for example `SELECT arrayAutocorrelation([arrayMax([])])`. Such an argument is now rejected with `ILLEGAL_COLUMN`. The empty array literal `[]` keeps returning `[]`. [#114249](https://github.com/ClickHouse/ClickHouse/pull/114249) ([groeneai](https://github.com/groeneai)). * `CREATE TABLE` and `ALTER TABLE` now reject lossy codec such as `SZ3` on a column used in the sorting key. [#114531](https://github.com/ClickHouse/ClickHouse/pull/114531) ([groeneai](https://github.com/groeneai)). * Fixed an infinite, uncancellable loop in functions `hop` and `windowID` when the span of an interval argument in seconds is a multiple of 2^32 (for example, `toIntervalDay(2147483648)`): the wrapped subtraction dodged the time-overflow check, and with constant arguments the loop ran at analysis time, where the query could not even be killed. The same interval could also spin a background thread of a `WINDOW VIEW` forever; such a window view is now rejected at creation. [#114607](https://github.com/ClickHouse/ClickHouse/pull/114607) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the logical error `Not-ready Set is passed as the second argument for function 'in'` for a query with an `IN` subquery reading from a remote cluster: the query now fails with the actual error from the subquery, such as a connection failure. [#114983](https://github.com/ClickHouse/ClickHouse/pull/114983) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed an `INSERT` into a `Log`, `TinyLog` or `StripeLog` table that fails while committing the recorded file sizes. Such an insert could keep its rows in the table even though it reported an error, or leave the data files of an array column inconsistent with each other. Reading a `Log` or `TinyLog` table whose array column holds no elements while its offsets claim some now raises an error naming the column instead of reading past the end of the elements. A partially written elements column was already rejected. [#115027](https://github.com/ClickHouse/ClickHouse/pull/115027) ([groeneai](https://github.com/groeneai)). * Fix `TYPE_MISMATCH` ("CAST AS Array can only be performed between same-dimensional array types") for a query with a constant of type `Array(Map(...))`, such as `has([map('k', 'v')], m)`, executed over a `Distributed` table or with parallel replicas. [#115883](https://github.com/ClickHouse/ClickHouse/pull/115883) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed wrong results when joining against a view or `view()` whose projected column has the same name as a column of one of its own inner relations, for example `SELECT expr(x.c0) AS c0 FROM a AS x, b AS y`. The join order optimizer merged such an expression into the flattened join graph and then applied it twice, so the query silently returned skewed values. When the expression also changed the column type, the same cause produced `AMBIGUOUS_COLUMN_NAME` at plan time instead. [#116152](https://github.com/ClickHouse/ClickHouse/pull/116152) ([groeneai](https://github.com/groeneai)). * Fixed an exception in correlated subqueries when outer columns become `Nullable` under `group_by_use_nulls` with ROLLUP/CUBE. [#100365](https://github.com/ClickHouse/ClickHouse/pull/100365) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Reject `max_replication_lag_to_enqueue = 0` on `Replicated` databases. The value made the post-recovery unsynced check trivially true and aborted the server with `LOGICAL_ERROR` in debug and sanitizer builds. `0` is now rejected at parse time with `BAD_ARGUMENTS` from every source (`CREATE DATABASE ... SETTINGS`, `` server-config block, `ATTACH` replay, and upgrade-time replay of existing metadata). The smallest valid value is `1`. [#106006](https://github.com/ClickHouse/ClickHouse/pull/106006) ([groeneai](https://github.com/groeneai)). * Forbid creating minmax skip index on JSON (Object) columns to prevent NO\_COMMON\_TYPE exception when inserting mixed-type arrays. Use typed subcolumns instead (e.g., `INDEX idx json.field TYPE minmax`). [#106094](https://github.com/ClickHouse/ClickHouse/pull/106094) ([linhaojie](https://github.com/linhaojie)). * Fix `JSON_QUERY`/`JSON_VALUE`/`JSON_EXISTS` returning Dynamic instead of String/UInt8 on Dynamic arguments. [#106877](https://github.com/ClickHouse/ClickHouse/pull/106877) ([Avogar](https://github.com/Avogar)). * Fixed a signed integer overflow when a wait-timeout setting, such as `interactive_delay`, was set to a huge value: the wait could time out immediately instead of waiting, and the server could abort under the Undefined Behavior Sanitizer. [#106961](https://github.com/ClickHouse/ClickHouse/pull/106961) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` ("Unsupported argument types") in the `geohashesInBox` function when its coordinate arguments mixed constant and non-constant values, or when a coordinate was a `BFloat16`. Mixed const/non-const `Float32` arguments now work, and `BFloat16` arguments are rejected with a clear error. [#107063](https://github.com/ClickHouse/ClickHouse/pull/107063) ([groeneai](https://github.com/groeneai)). * Fixed the `CSVWithNames` and `CSVWithNamesAndTypes` header having fewer columns than the data when `output_format_csv_serialize_tuple_into_separate_columns` is enabled (the default). The header (and the types row) now flattens `Tuple` columns into their leaf fields with dotted names (e.g. `t.a`, `t.b`), so the header column count matches the data. A new setting `output_format_csv_header_serialize_tuple_into_separate_columns` (default `1`) controls this and can be set to `0` to restore the previous single-name header. [#107371](https://github.com/ClickHouse/ClickHouse/pull/107371) ([groeneai](https://github.com/groeneai)). * Fixed a memory leak in the bundled `mongo-c-driver` that could occur when reading from MongoDB (for example via a MongoDB dictionary or the `mongodb` table function) if a retryable read error was followed by a failed retry server selection. [#107448](https://github.com/ClickHouse/ClickHouse/pull/107448) ([groeneai](https://github.com/groeneai)). * Fix a logical error (`WhichDataType(const_type).isArray()`) and server abort during primary-key analysis when `pointInPolygon` is called with a constant polygon argument of a wrapper type such as `Variant` or `Dynamic` (for example `pointInPolygon((x, y), if(c, [(0, 0), ...], NULL))`). [#107589](https://github.com/ClickHouse/ClickHouse/pull/107589) ([groeneai](https://github.com/groeneai)). * Fixed logical errors (`Bad cast`) caused by inconsistent stripping of `LowCardinality` nested inside `Variant` and `Dynamic` columns, for example in `concat`, `format`, and primary-key analysis. [#107773](https://github.com/ClickHouse/ClickHouse/pull/107773) ([Avogar](https://github.com/Avogar)). * Fixed executable user-defined function command parameter parsing so placeholders with empty or invalid names are not accepted as parameters. [#107983](https://github.com/ClickHouse/ClickHouse/pull/107983) ([goutamadwant](https://github.com/goutamadwant)). * Fix `accurateCastOrNull` of a `Tuple` whose element is `Dynamic`/`Variant`: a genuine source `NULL` was treated as a conversion failure. For a `Nullable` target element a source `NULL` now stays an element `NULL` (matching a plain `Tuple(Nullable(...))` source); for a non-`Nullable` target element a source `NULL` now nulls the whole tuple (matching the non-`Dynamic` reference) instead of producing the element default. Parse/overflow failures still null the whole tuple. [#108023](https://github.com/ClickHouse/ClickHouse/pull/108023) ([groeneai](https://github.com/groeneai)). * The `mongodb` table function now accepts `oid_columns` passed as a named argument (e.g. `oid_columns='_id'`), instead of rejecting it with `BAD_ARGUMENTS`. [#108039](https://github.com/ClickHouse/ClickHouse/pull/108039) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allow querying a `range_hashed` or `complex_key_range_hashed` dictionary that uses a `DateTime64`, `Decimal` or floating-point range with a matching argument to `dictGet`/`dictHas`. Previously such queries failed with `must be convertible to Int64`, and open-ended intervals of `Decimal`/`DateTime64` ranges incorrectly returned the default value. [#108052](https://github.com/ClickHouse/ClickHouse/pull/108052) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix a server abort (`Logical error` in `IColumn::insertFrom`) when casting an `Array(Dynamic)` or `Array(Variant)` to `QBit` with `accurateCastOrNull`, e.g. `accurateCastOrNull(CAST(range(114), 'Array(Dynamic)'), 'QBit(Float32, 114)')`. [#108288](https://github.com/ClickHouse/ClickHouse/pull/108288) ([groeneai](https://github.com/groeneai)). * Fixed a server crash (`Received signal 4`, illegal instruction) that could be triggered by a malformed or desynchronized native TCP protocol stream. The `initial_address` field of `ClientInfo` is now validated to be a numeric `host:port` before it is parsed, instead of letting a non-numeric port reach the trapped `getservbyname` libc function. [#108410](https://github.com/ClickHouse/ClickHouse/pull/108410) ([groeneai](https://github.com/groeneai)). * Fixed excessive memory allocation when deserializing crafted aggregate function states for `mannWhitneyUTest`, `rankCorr`, `largestTriangleThreeBuckets`, `quantileGK`, `sequenceMatch`/`sequenceCount` and `groupArrayIntersect`. A malformed state could declare a huge element count and make the server try to allocate tens of gigabytes from a few bytes of input; such states are now rejected with `TOO_LARGE_ARRAY_SIZE`. [#108465](https://github.com/ClickHouse/ClickHouse/pull/108465) ([groeneai](https://github.com/groeneai)). * Fixed a logical error (`Bad cast from type DB::ColumnNullable to DB::ColumnString`) and possible wrong results when using `group_by_use_nulls` with a `LowCardinality` constant grouping key in `GROUPING SETS`/`ROLLUP`/`CUBE`. [#108771](https://github.com/ClickHouse/ClickHouse/pull/108771) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `TYPE_MISMATCH` in `mapFilter`, `mapSort`, `mapReverseSort`, `mapPartialSort` and `mapConcat` when a `Map` value contains a nested `Map` with `LowCardinality`, e.g. `mapFilter((k, v) -> 1, map('a'::LowCardinality(String), map('x'::LowCardinality(String), 'y')))`. [#108798](https://github.com/ClickHouse/ClickHouse/pull/108798) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` ("Inconsistent AST formatting") that aborted the server in debug and sanitizer builds when formatting a `Tuple` data type that mixes named and unnamed elements (for example `Tuple(a UInt8, UInt16)`). [#108915](https://github.com/ClickHouse/ClickHouse/pull/108915) ([groeneai](https://github.com/groeneai)). * `JSONExtract` now honours `cast_string_to_date_time_mode` when converting string JSON values to `DateTime`/`DateTime64`, consistently with `CAST`. [#109252](https://github.com/ClickHouse/ClickHouse/pull/109252) ([Utkal059](https://github.com/Utkal059)). * Fixed `IN` with a bare array column on the right argument silently returning a wrong (always-false) result or throwing an exception, so that `x IN arr` behaves like `has(arr, x)`. [#109416](https://github.com/ClickHouse/ClickHouse/pull/109416) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix a `LOGICAL_ERROR` in `arrayFold` over a non-const `Array(LowCardinality(T))` argument (for example `arrayFold((acc, x) -> acc + x, materialize([1, 2, 3]::Array(LowCardinality(Int64))), toInt64(0))`), which failed with "Arguments of 'plus' have incorrect data types" (a server abort in debug and sanitizer builds). [#109462](https://github.com/ClickHouse/ClickHouse/pull/109462) ([groeneai](https://github.com/groeneai)). * Fixed an `ILLEGAL_COLUMN` exception in `conv` by correctly casting `FixedString` arguments to `String`. [#109771](https://github.com/ClickHouse/ClickHouse/pull/109771) ([hp77-creator](https://github.com/hp77-creator)). * Fix wrong results for table-function reads with `FINAL` / `SAMPLE` under `serialize_query_plan = 1`, and fix the aggregation hash-table stats cache key for table functions (previously all table functions shared one preallocation entry). [#109847](https://github.com/ClickHouse/ClickHouse/pull/109847) ([azat](https://github.com/azat)). * Fix parsing of numeric `1`/`0` as `Bool` values inside container types (e.g. `Array(Bool)`, `Tuple(Bool, ...)`). Previously `[1,0]` failed with `CANNOT_READ_ARRAY_FROM_TEXT` while `[true,false]` worked. [#109976](https://github.com/ClickHouse/ClickHouse/pull/109976) ([groeneai](https://github.com/groeneai)). * Fixed `NOT_IMPLEMENTED` error (`Method getDataAt is not supported for Nullable(String) in case if value is NULL`) when a `text` index is built on `Array(LowCardinality(Nullable(String)))` (including `Nested` fields stored that way) and an indexed array contains a `NULL` element. `NULL` array elements are now skipped during index construction, matching `Array(Nullable(String))`. [#110055](https://github.com/ClickHouse/ClickHouse/pull/110055) ([groeneai](https://github.com/groeneai)). * Fixed incorrect results from `subBitmap`, `bitmapSubsetInRange` and `bitmapSubsetLimit` for bitmaps still held in the small representation, including `UInt64` and `Int64` element values above `2^32`, which were truncated to 32 bits. Comparisons in bitmap functions over signed element types now consistently use the unsigned value of the element type, so in an `Int8` bitmap the element `-1` is compared as `255` instead of as the sign-extended `4294967295`; this also fixes `bitmapMin`, `bitmapMax`, `bitmapContains` and `bitmapTransform` on bitmaps that have grown past the small representation. Queries that passed sign-extended thresholds have to be adjusted: over an `Int8` bitmap, `bitmapSubsetInRange(bm, 4294967168, 4294967296)` becomes `bitmapSubsetInRange(bm, 128, 256)`. Also fixed `groupNumericIndexedVector` returning different results for `Int8` and `Int16` index columns than for wider index types, and `numericIndexedVectorGetValue` returning `0` for negative indexes. Corrected the bitmap function documentation, including the subset functions that were described as using 1-based indexing and the signed `bitmapBuild` / `bitmapToArray` support. [#110072](https://github.com/ClickHouse/ClickHouse/pull/110072) ([RamiDarwiche](https://github.com/RamiDarwiche)). * Fixed undefined signed-overflow paths in interval arithmetic for interval-kind `WITH FILL STEP` and for `add*` / `subtract*` date/time interval functions with extreme deltas, making extreme interval values deterministic instead of relying on signed overflow. Also fixed a stall of interval-kind `WITH FILL` with a huge `YEAR` step, which previously ran the calendar backward. [#110158](https://github.com/ClickHouse/ClickHouse/pull/110158) ([groeneai](https://github.com/groeneai)). * Fixed a quadratic-time blowup (and unresponsiveness to `max_execution_time`) in aggregation in order when grouping by multiple keys whose sort order is only a prefix of the grouping key. [#110159](https://github.com/ClickHouse/ClickHouse/pull/110159) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed comparisons that previously threw `ILLEGAL_TYPE_OF_ARGUMENT` when the operands have no least common supertype. Arrays such as `Array(Int64)` vs `Array(UInt64)` and `Array(Int256)` vs `Array(UInt256)` can now be compared lexicographically with `=`, `!=`, `<`, `<=`, `>`, `>=`, `IS DISTINCT FROM`, and `IS NOT DISTINCT FROM`. Null-safe comparisons such as `isDistinctFrom` / `isNotDistinctFrom` now also compare supported non-array numeric or decimal pairs by value instead of throwing. Equality comparisons also work for arrays with nested `Tuple(Nullable(...))` elements. [#110245](https://github.com/ClickHouse/ClickHouse/pull/110245) ([diegomestre2](https://github.com/diegomestre2)). * Fix a signed-integer overflow (UBSan) when converting an extreme `DateTime64` value (e.g. `INT64_MIN`) to `Time`/`Time64` in a timezone with a negative offset. [#110451](https://github.com/ClickHouse/ClickHouse/pull/110451) ([groeneai](https://github.com/groeneai)). * Conversion of out-of-range floating-point values (`BFloat16`, `Float32`, `Float64`) and of `UInt64` values above `Int64::max()` to `DateTime` and `Time` now reliably saturates to the range boundaries on all architectures (previously the result of an out-of-range conversion was architecture-dependent and could wrap around on x86-64), and non-finite values (`NaN`, `inf`) now throw an exception instead of producing an arbitrary result. The same fix is applied to the float-to-`Date` conversion path of `toDate` and to the conversions of `UInt64` and of wide integers (`(U)Int128`, `(U)Int256`) to `Date` and `Date32`, which could return a pre-epoch day instead of saturating. Conversion of a number to `Time` now saturates to the range of the type for every numeric source type (previously values from narrow unsigned or wide integer sources were stored unclamped and clamped only when printed), and the accurate cast to `Time` no longer rejects the negative values that `Time` supports, nor silently saturates the values that do not fit into it. The accurate cast of a number to `Date32` (`accurateCast`, `accurateCastOrNull`) no longer silently saturates an out-of-range or non-finite value, but throws or returns `NULL` as it already did for `Date` and `DateTime`, and the `OrDefault` flavours return the default value of `Date32` for such an input. The accurate cast of a non-integral floating-point number to `Date`, `Date32`, `DateTime` or `Time` is now rejected instead of being silently truncated. Also fixed the `toDateTime32` function documentation, which mistakenly showed `toDateTime64` examples and described the `DateTime64` value range. [#110459](https://github.com/ClickHouse/ClickHouse/pull/110459) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix schema inference failing with `Map cannot have a key of type LowCardinality(Nullable(String))` when reading ORC files whose map keys are dictionary-encoded — including ORC files written by Spark and files written by ClickHouse itself with `output_format_orc_dictionary_key_size_threshold` enabled. [#110492](https://github.com/ClickHouse/ClickHouse/pull/110492) ([tiandiwonder](https://github.com/tiandiwonder)). * Fix a logical error (`Function node with name '...' is not resolved as ordinary function`) that could occur with `optimize_or_like_chain` enabled when an aggregate or window function appeared on the left-hand side of a `LIKE`/`ILIKE`/`match` inside an `OR` chain. [#110641](https://github.com/ClickHouse/ClickHouse/pull/110641) ([groeneai](https://github.com/groeneai)). * Fixed a signed integer overflow in `toStartOfInterval` with an extreme `MONTH`, `WEEK`, or `DAY` interval count on `Date`/`Date32`/`DateTime64` values. [#110688](https://github.com/ClickHouse/ClickHouse/pull/110688) ([groeneai](https://github.com/groeneai)). * Fixed `readWKT` and `readWKTPoint` returning uninitialized (scalar) or stale prior-row (vectorized) coordinates for `POINT EMPTY`. `POINT EMPTY` is now rejected with `CANNOT_PARSE_TEXT`, since a ClickHouse `Point` is a fixed `Tuple(Float64, Float64)` with no empty representation. [#110692](https://github.com/ClickHouse/ClickHouse/pull/110692) ([groeneai](https://github.com/groeneai)). * Fixed `flipCoordinates` losing the `Geometry` type: for a `Geometry` argument the result is now typed `Geometry` again instead of the underlying `Variant(...)`, so it can be passed directly to functions like `areaCartesian`. [#110694](https://github.com/ClickHouse/ClickHouse/pull/110694) ([groeneai](https://github.com/groeneai)). * Fixed non-canonical WKB serialization of an empty polygon: `wkb(POLYGON EMPTY)` now emits `numRings = 0` instead of a spurious single zero-point ring, so a WKB round-trip of a standard empty polygon is identity-preserving. [#110796](https://github.com/ClickHouse/ClickHouse/pull/110796) ([groeneai](https://github.com/groeneai)). * Numeric `1`/`0` in JSON `Bool` parsing now honors the `allow_special_bool_values` setting. Previously JSON input such as `{"v":1}` for `Variant(Bool, UInt32)` was read as `Bool` even with `allow_special_bool_values_inside_variant = 0`, because the JSON `Bool` parsers ignored the setting and `Bool` has a higher `Variant` deserialize priority than integer types. [#110835](https://github.com/ClickHouse/ClickHouse/pull/110835) ([groeneai](https://github.com/groeneai)). * Fixed `throwIf(notLike(col, pattern))` (and other `throwIf` over a function) throwing unconditionally when `col` is `LowCardinality`. The default `LowCardinality` implementation ran `throwIf` on the whole dictionary, which always holds the reserved default value even when no row references it, so a non-zero value in that unused slot made `throwIf` throw for data that does not satisfy the condition. [#110864](https://github.com/ClickHouse/ClickHouse/pull/110864) ([groeneai](https://github.com/groeneai)). * Fixed a crash when an aggregate function with the `-Tuple` combinator was used with `RESPECT NULLS` and produced an intermediate state (`-State`, distributed aggregation, `WITH ROLLUP`/`WITH CUBE`). The combined function was named after the wrong state variant, so its serialized `AggregateFunction` type re-resolved to an incompatible in-memory layout on a round-trip. [#110930](https://github.com/ClickHouse/ClickHouse/pull/110930) ([groeneai](https://github.com/groeneai)). * Fixed `Nested type LineString cannot be inside Nullable type (ILLEGAL_TYPE_OF_ARGUMENT)` when reading a nullable MySQL spatial column (`LINESTRING`, `POLYGON`, `MULTILINESTRING`, `MULTIPOLYGON`, `MULTIPOINT`, or the generic `GEOMETRY`) through the `mysql` table function or the `MySQL` table/database engines. Such columns now fall back to `Nullable(String)` holding the value as MySQL returns it (a 4-byte SRID prefix followed by the WKB payload). `POINT` keeps mapping to `Nullable(Point)`. [#110943](https://github.com/ClickHouse/ClickHouse/pull/110943) ([groeneai](https://github.com/groeneai)). * Fixes `Native` serialization of `SimpleAggregateFunction` wrappers containing versioned aggregate states for older peers. [#110997](https://github.com/ClickHouse/ClickHouse/pull/110997) ([groeneai](https://github.com/groeneai)). * Fix `CANNOT_CONVERT_TYPE` error for constants of `Variant` type (including `Geometry`) in distributed queries and queries with parallel replicas. [#111136](https://github.com/ClickHouse/ClickHouse/pull/111136) ([fm4v](https://github.com/fm4v)). * Fix a `LOGICAL_ERROR` ("Lambda resolved type ... is not equal to type from actions DAG ...") when a higher-order function's lambda body is a constant-condition `if`/`multiIf` that mixes a `Bool` literal branch with a `UInt8` comparison branch, e.g. `arrayFilter(p -> multiIf(false, true, p = 'ALL'), ['ALL'])`. [#111245](https://github.com/ClickHouse/ClickHouse/pull/111245) ([groeneai](https://github.com/groeneai)). * Fixed a signed integer overflow in `toStartOfInterval` with subsecond intervals: for `DateTime64` values within one interval of the lower bound of `Int64`, the rounded result silently wrapped to a positive garbage value; now the function throws a `DECIMAL_OVERFLOW` exception. [#111370](https://github.com/ClickHouse/ClickHouse/pull/111370) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed an exception (`CANNOT_CONVERT_TYPE` or a logical error) in distributed queries using the `-Tuple` combinator with a `RESPECT NULLS` / `IGNORE NULLS` modifier under nested combinators, e.g. `anyRespectNullsStateTuple(...) IGNORE NULLS`. [#111570](https://github.com/ClickHouse/ClickHouse/pull/111570) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes a `FixedString` text try-parse that leaves partially appended bytes in the column when parsing fails, which can cause a `Sizes of nested column and null map of Nullable column are not equal` logical error during serialization of `Variant`/`Array`/`Nullable` columns, or silently shifted result bytes. [#111908](https://github.com/ClickHouse/ClickHouse/pull/111908) ([groeneai](https://github.com/groeneai)). * Fix an exception and silently skipped data when reading a dynamic subcolumn (a path inside a `JSON` or a `Dynamic` column) through a `Buffer` table. [#111948](https://github.com/ClickHouse/ClickHouse/pull/111948) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed quadratic complexity in `countSubstringsCaseInsensitiveUTF8`, which made the function take minutes on a haystack of a few megabytes with many matches. The character offset of each match is now maintained incrementally instead of being recounted from the start of the row. [#112003](https://github.com/ClickHouse/ClickHouse/pull/112003) ([groeneai](https://github.com/groeneai)). * Fixed a signed integer overflow in the three-argument overload of `toStartOfInterval`: for an `origin` argument near the lower bound of `Int64` the function returned a wrapped-around value instead of reporting an error. [#112045](https://github.com/ClickHouse/ClickHouse/pull/112045) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Disallow aggregate states in table function parameters, because they are not supported at all. [#112087](https://github.com/ClickHouse/ClickHouse/pull/112087) ([PedroTadim](https://github.com/PedroTadim)). * Fixed the `values` table function rendering a non-literal `Bool` constant (e.g. `CAST(true, 'Nullable(Bool)')`) inserted into a `String`/textual column as `1`/`0` instead of `true`/`false`. [#112308](https://github.com/ClickHouse/ClickHouse/pull/112308) ([yakov-olkhovskiy](https://github.com/yakov-olkhovskiy)). * Fixes a logical error (which aborts in debug and sanitizer builds) when `finalizeAggregation` is applied to a column whose aggregate states come from different functions that share a state representation but finalize to different types — for example `quantileState` and `quantilesState(0.9)` brought together by a `UNION` or produced via `arrayReduce`. Such queries now raise a normal error instead of crashing. [#112662](https://github.com/ClickHouse/ClickHouse/pull/112662) ([zainulabidin302](https://github.com/zainulabidin302)). * Fixed parsing of integers that do not fit into the target type in `Poco::NumberParser`, which is used to read JSON numbers. Such a number was silently parsed as a wrong value (for example, `18446744073709551617` became `1`) instead of being rejected, and a `UInt64` number above the `Int64` maximum did not survive an AST JSON round trip. [#112904](https://github.com/ClickHouse/ClickHouse/pull/112904) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a `Bool` constant on the right-hand side of `IN` being converted to a `String` left-hand side as `'1'`/`'0'` instead of the canonical `'true'`/`'false'` (now consistent with `CAST` and the `values` table function). [#113051](https://github.com/ClickHouse/ClickHouse/pull/113051) ([yakov-olkhovskiy](https://github.com/yakov-olkhovskiy)). * Fix inconsistent AST formatting of `viewIfPermitted`: a function normally written as an operator (e.g. `not`) in the `ELSE` branch of the table function form, and the expression form `viewIfPermitted(...)` being wrongly formatted with `ELSE`. [#113652](https://github.com/ClickHouse/ClickHouse/pull/113652) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix undefined behavior and wrong results in `avg` over `Date`/`DateTime`/`DateTime64`/`Time`/`Time64`: the average is now computed exactly in integer space, fixing both the `Int64`-boundary overflow (UB, wrong result on x86) and `Float64` precision loss above 2^53 (visible at nanosecond scale). [#113912](https://github.com/ClickHouse/ClickHouse/pull/113912) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix a memory leak in the experimental `SZ3` compression codec: when compression of poorly-compressible or corrupted data failed with an exception, temporary scratch buffers were left allocated. [#113916](https://github.com/ClickHouse/ClickHouse/pull/113916) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes `hasAny`, `hasAll`, `has`, `indexOf`, `mapContainsKey`, `mapContainsValue` and `mapContains` returning too few rows, or failing with `TOO_LARGE_STRING_SIZE`, when a `bloom_filter` index is queried with a `FixedString` constant. The index hashed the padded form of the constant while the function compares the unpadded one, so a matching granule was skipped. [#114089](https://github.com/ClickHouse/ClickHouse/pull/114089) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `min` and `max` on `DateTime64` columns returning wrong results once the aggregate is JIT compiled and the data contains timestamps before 1970. `is_signed` was not specialised for `DateTime64` and `Time64`, which derive from `Decimal64`, so the generated code compared tick counts as unsigned integers and a pre-1970 timestamp won `max` and lost `min`. [#114168](https://github.com/ClickHouse/ClickHouse/pull/114168) ([groeneai](https://github.com/groeneai)). * Fixes `arrayExists(x -> x = needle, arr)` returning a wrong result, or failing with `TOO_LARGE_STRING_SIZE`, when the array element and the needle are different string types (for example a `String` needle against an `Array(FixedString(N))` element). The `optimize_rewrite_array_exists_to_has` optimization, enabled by default, rewrote such a call to `has`, which does not compare zero-padded the way `=` does. [#114496](https://github.com/ClickHouse/ClickHouse/pull/114496) ([groeneai](https://github.com/groeneai)). * Fix text index evaluation of `LIKE/ILIKE` operator built on Map or JSON containers. [#114544](https://github.com/ClickHouse/ClickHouse/pull/114544) ([ahmadov](https://github.com/ahmadov)). * Fixed undefined behavior and incorrect boundary results in `quantile` and `quantiles` for `DateTime64` and wide `Decimal` values. [#114919](https://github.com/ClickHouse/ClickHouse/pull/114919) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `if` and `multiIf` returning a large positive value instead of a negative one when a `Time` branch is combined with `Time64`, `DateTime` or `DateTime64` and the expression is JIT-compiled. [#115146](https://github.com/ClickHouse/ClickHouse/pull/115146) ([groeneai](https://github.com/groeneai)). * Fixed `timezoneOffset` (alias `timeZoneOffset`) returning an offset wrong by exactly one day for time zones whose UTC offset differs from their 1970-01-01 offset by a whole day, such as `Pacific/Kiritimati` and `Pacific/Apia` after they crossed the international date line, and for most zones before 1970. This also fixes `parseDateTime`, `parseDateTimeOrNull`, `parseDateTime64`, `parseDateTimeInJodaSyntax`, `EXTRACT(TIMEZONE_HOUR / TIMEZONE_MINUTE ...)`, `formatDateTime`'s `%z`, `toUTCTimestamp` and `fromUTCTimestamp`, which consume that offset. [#115332](https://github.com/ClickHouse/ClickHouse/pull/115332) ([groeneai](https://github.com/groeneai)). * Fixed `Code: 349. Cannot convert NULL value to non-Nullable type` when a `has` or `notHas` predicate carries a NULL array element and the primary key is a `String`, `Array(String)` or `Map(String, String)`, and fixed wrong results when that element is a `Dynamic` or `Variant` holding NULL. The NULL is now dropped from the set during key analysis, as it already was for other key types. [#115647](https://github.com/ClickHouse/ClickHouse/pull/115647) ([groeneai](https://github.com/groeneai)). * Fixed `cast_keep_nullable` not preserving nullability when the target type is `LowCardinality`. Casting a NULL-capable value to `LowCardinality(T)` threw `CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN` instead of producing `LowCardinality(Nullable(T))`. [#115851](https://github.com/ClickHouse/ClickHouse/pull/115851) ([groeneai](https://github.com/groeneai)).

Index and query-cache fixes

* Fix reading subcolumns of a column that has a `DEFAULT` expression and is not materialized in a part (e.g. after `ALTER TABLE ADD COLUMN` or during `ALTER TABLE MATERIALIZE COLUMN`): the subcolumn was filled with type default values instead of the evaluated `DEFAULT` expression, and the transposed vector distance functions (`cosineDistanceTransposed` and others) on such `QBit` columns failed with the `SIZES_OF_ARRAYS_DONT_MATCH` error. [#110636](https://github.com/ClickHouse/ClickHouse/pull/110636) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix the `Not-ready Set is passed as the second argument` exception that could occur when building an `IN` subquery set during primary key analysis failed silently (for example, a subquery timeout with `overflow_mode = 'break'`), leaving the set permanently unbuilt for the query pipeline. [#107924](https://github.com/ClickHouse/ClickHouse/pull/107924) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a `SELECT` failing with `NOT_FOUND_COLUMN_IN_BLOCK` on a table that has a `set` data-skipping index when a row policy filters it using an always-true condition combined with a check on a non-indexed column. [#107971](https://github.com/ClickHouse/ClickHouse/pull/107971) ([tiandiwonder](https://github.com/tiandiwonder)). * `getClientHTTPHeader` is now correctly treated as non-deterministic, so its result is no longer incorrectly reused by the query result cache. [#108029](https://github.com/ClickHouse/ClickHouse/pull/108029) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed wrong results when the query condition cache (`use_query_condition_cache = 1`) reused a skip-index-derived mark exclusion for a query that ran a different set of skip indexes, for example with `use_skip_indexes = 0`, `ignore_data_skipping_indices`, or a different `use_skip_indexes_for_disjunctions` mode. Skip-index-derived cache entries are now keyed by the effective set of skip indexes that ran, while row-level query-condition-cache entries stay reusable across all of these. [#108548](https://github.com/ClickHouse/ClickHouse/pull/108548) ([groeneai](https://github.com/groeneai)). * Fixed a hang where a query reading through the filesystem cache could keep waiting on an in-progress download after being cancelled with `KILL QUERY`, and where dropping or `SYSTEM STOP VIEW`-ing a refreshable materialized view whose refresh was stuck in such a wait would block. [#109116](https://github.com/ClickHouse/ClickHouse/pull/109116) ([murphy-4o](https://github.com/murphy-4o)). * Fix wrong query results caused by the primary key index incorrectly pruning granules for tables with a reversed (`DESC`) key column when parts have no final mark (non-adaptive granularity, `index_granularity_bytes = 0`). [#109901](https://github.com/ClickHouse/ClickHouse/pull/109901) ([nihalzp](https://github.com/nihalzp)). * Fix the logical error `Expected CommonSubplanReferenceStep to reference CommonSubplanStep`, the error `Subplan cannot be used to build pipeline`, and the logical error `Trying to extract chunk from ChunkBuffer before all inputs are finished` for `IN (subquery)` where the subquery contains a correlated subquery and the set is built during index analysis. [#110491](https://github.com/ClickHouse/ClickHouse/pull/110491) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `NOT_FOUND_COLUMN_IN_BLOCK` error for the transposed distance functions over `QBit` columns (such as `cosineDistanceTransposed`) called with a scalar-subquery or constant reference vector when parallel replicas are enabled. [#110729](https://github.com/ClickHouse/ClickHouse/pull/110729) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix text index not being used when its expression contains an empty string comparison such as `arrayFilter(s -> s != '', ...)`, because the setting `optimize_empty_string_comparisons` rewrites the comparison to `notEmpty` on the query side only. [#112115](https://github.com/ClickHouse/ClickHouse/pull/112115) ([Ergus](https://github.com/Ergus)). * Do not remove columns when it's referenced in both `PREWHERE` and `WHERE` filter conditions even if it's rewritten into a text index virtual column. [#114460](https://github.com/ClickHouse/ClickHouse/pull/114460) ([ahmadov](https://github.com/ahmadov)). * Fixed logical error `Multi-block postings must be compressed` in queries over tables with a text index in the lazy posting-list apply mode, when the query was canceled during the read. [#114636](https://github.com/ClickHouse/ClickHouse/pull/114636) ([CurtizJ](https://github.com/CurtizJ)). * Fixed wrong results when a `WHERE` clause contained a `NULL` expression under a `NOT`, for example `SELECT count() FROM t WHERE NOT (id >= 20 AND NULL)`. Index analysis treated such a condition as provably true for the whole range, so the filter was skipped: `count()` returned rows the query rejects, and `ALTER TABLE ... DELETE` with that condition removed rows it should have kept. [#115152](https://github.com/ClickHouse/ClickHouse/pull/115152) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` (`Join is supported only for pipelines with one output port`) for a `full_sorting_merge` JOIN with `query_plan_join_shard_by_pk_ranges` enabled when one of the two sides is pruned to no parts at all - by the primary key, by column statistics, or because the table is empty. [#115352](https://github.com/ClickHouse/ClickHouse/pull/115352) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Disable the query condition cache for filters over \_part\_starting\_offset [#115358](https://github.com/ClickHouse/ClickHouse/pull/115358) ([azat](https://github.com/azat)). * Fixed `LOGICAL_ERROR: Index with name auto_minmax_index_ already exists` when a single `ALTER TABLE` statement combined `RENAME COLUMN` with another command, such as `MODIFY SETTING`, on a table with implicit minmax indices (`add_minmax_index_for_numeric_columns` and friends). Renaming a plain column also no longer leaves a following `DROP COLUMN` failing with `UNKNOWN_IDENTIFIER`, nor a following `ADD COLUMN` without its implicit index. [#116063](https://github.com/ClickHouse/ClickHouse/pull/116063) ([groeneai](https://github.com/groeneai)). * A `LIKE`/`NOT LIKE` pattern without wildcards (`%`, `_`) now uses an exact primary key range, so it reads the same number of granules as the equivalent `=`/`!=` predicate instead of a wider prefix range. [#107077](https://github.com/ClickHouse/ClickHouse/pull/107077) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` (`Invalid binary search result in MergeTreeSetIndex`) and missed index pruning in release builds when an `IN`/`NOT IN` condition on the primary key wraps the key in `intDiv` by a constant and the key is an unsigned integer whose values cross the signed boundary of the `intDiv` result type (for example `intDiv(uint64_column, -9223372036854775807)`). [#107586](https://github.com/ClickHouse/ClickHouse/pull/107586) ([groeneai](https://github.com/groeneai)). * Fixed `numericIndexedVectorPointwiseMultiply` returning an empty result when multiplying by an all-ones vector that has a different BSI bit configuration. [#108027](https://github.com/ClickHouse/ClickHouse/pull/108027) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a heap-buffer-overflow when building a text index with `positions = 1` over many distinct short tokens. [#108659](https://github.com/ClickHouse/ClickHouse/pull/108659) ([groeneai](https://github.com/groeneai)). * Allow the `preprocessor` and `postprocessor` expressions of a text index to reference `ALIAS` columns. [#110014](https://github.com/ClickHouse/ClickHouse/pull/110014) ([Ergus](https://github.com/Ergus)). * Skips vector search optimization when `LIMIT ... WITH TIES` is used, because the optimization bounds the ANN search to exactly n candidates and rows tied with the n-th row are dropped. [#110453](https://github.com/ClickHouse/ClickHouse/pull/110453) ([tamish560](https://github.com/tamish560)). * Fixed the text index not being used for `ILIKE` when the index is defined over an expression rather than a bare column (for example `assumeNotNull(col)` with a `lower(...)` preprocessor). Such queries now use the index instead of reading all granules. [#110595](https://github.com/ClickHouse/ClickHouse/pull/110595) ([ahmadov](https://github.com/ahmadov)). * Fix a `LOGICAL_ERROR` (`LazyMaterializingTransform: Number of rows in lazy chunk N does not match number of offsets M`) that could happen for vector search queries with `vector_search_with_rescoring = 1` combined with lazy materialization when several distances tie. [#111003](https://github.com/ClickHouse/ClickHouse/pull/111003) ([groeneai](https://github.com/groeneai)). * Reject vectors whose squared magnitude overflows to infinity when using a `vector_similarity` index with `i8` quantization (previously such vectors triggered undefined behavior in usearch and produced silent garbage). [#111085](https://github.com/ClickHouse/ClickHouse/pull/111085) ([groeneai](https://github.com/groeneai)). * Fixed stale reads on `plain_rewritable` disks with the page cache enabled. [#111105](https://github.com/ClickHouse/ClickHouse/pull/111105) ([Michicosun](https://github.com/Michicosun)). * Fixed wrong `count()` results and dropped rows when a `String` (or narrower `FixedString`) key or minmax skip index is filtered by a comparison with a wider `FixedString` constant, e.g. `toFixedString('abc', 257) = string_col`. Primary-key and minmax pruning built the range from the NUL-padded constant and wrongly skipped matching granules. [#111106](https://github.com/ClickHouse/ClickHouse/pull/111106) ([groeneai](https://github.com/groeneai)). * Fixed reading Parquet files whose offset index does not start at row 0. Such a file could make the native Parquet reader return wrong rows, or report `LOGICAL_ERROR` instead of `INCORRECT_DATA`. The offset index is now validated when it is read. [#114530](https://github.com/ClickHouse/ClickHouse/pull/114530) ([groeneai](https://github.com/groeneai)).

Security, access, backup, and restore fixes

* `RESTORE` now fsyncs the restored part files when the destination table has `fsync_after_insert` enabled, so a restored part is as durable against power loss as an inserted one. Previously restored files were only written and closed (never fsynced), so a power loss right after `RESTORE` returned `RESTORED` could leave the parts torn and the table empty. [#111378](https://github.com/ClickHouse/ClickHouse/pull/111378) ([groeneai](https://github.com/groeneai)). * Fix a security issue where an unauthenticated TCP client could probe table existence and replication status via the interserver port. [#99854](https://github.com/ClickHouse/ClickHouse/pull/99854) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed `DELETE FROM` (lightweight delete) requiring the `ALTER UPDATE` privilege in addition to `ALTER DELETE`. A user granted only `ALTER DELETE` can now run `DELETE FROM`, as documented. [#107491](https://github.com/ClickHouse/ClickHouse/pull/107491) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed `CREATE TABLE ... AS SELECT` on Atomic databases leaving an empty table behind when the query fails — for example when the user has no access to a table referenced from a subquery. Previously a retry reported that the table already exists instead of the original error. The table is now created via a temporary table and becomes visible only after it has been fully populated. [#108048](https://github.com/ClickHouse/ClickHouse/pull/108048) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a `LOGICAL_ERROR` (`No available columns`) when executing `SELECT count()` (or other trivial queries) on a table where the user is granted `SELECT` access only on an `ALIAS` column. [#108056](https://github.com/ClickHouse/ClickHouse/pull/108056) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes an issue where unauthenticated requests could cause ClickHouse to resolve hostnames supplied in forwarded client-address headers. ClickHouse now accepts only numeric IP addresses from these headers. Invalid values are ignored for forwarded-address quota attribution without DNS resolution. When `auth_use_forwarded_address` is enabled, invalid forwarded addresses are rejected during authentication. Rejections are logged at debug level. [#111060](https://github.com/ClickHouse/ClickHouse/pull/111060) ([otselnik](https://github.com/otselnik)). * Fix `SQL SECURITY DEFINER` (and `SQL SECURITY NONE`) not being honored for parameterized views when the analyzer is disabled (`enable_analyzer = 0`). Previously selecting from such a view as a user without privileges on the underlying table failed with `ACCESS_DENIED`, even though the same view works with the analyzer enabled. [#111458](https://github.com/ClickHouse/ClickHouse/pull/111458) ([groeneai](https://github.com/groeneai)). * Fixed a defect in the `MySQL` integrations where overriding a TLS credential of a named collection with an empty `ssl_ca_pem`/`ssl_cert_pem`/`ssl_key_pem` value was accepted when the collection stored the credential in the contents form, silently dropping the configured CA or client certificate instead of rejecting the override. [#113947](https://github.com/ClickHouse/ClickHouse/pull/113947) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `EXPLAIN ANALYZE` losing the join statistics of a view declared `SQL SECURITY DEFINER` or `SQL SECURITY NONE` whose body contains a JOIN. In debug and sanitizer builds the query aborted with the logical error `JoinStep analyzed without the analyze mode`. [#115156](https://github.com/ClickHouse/ClickHouse/pull/115156) ([groeneai](https://github.com/groeneai)). * Fix `RESTORE ... AS ` leaving the `REFRESH ... DEPENDS ON` list of a refreshable materialized view pointing at the original database. The restored view refreshed in response to a table it no longer read, ignored refreshes of its own parent, and stopped refreshing entirely once the original database was dropped. [#115689](https://github.com/ClickHouse/ClickHouse/pull/115689) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` exception during backup of a `Replicated` database that is being dropped and recreated concurrently; such backups now fail cleanly with `CANNOT_GET_REPLICATED_DATABASE_SNAPSHOT`. [#100651](https://github.com/ClickHouse/ClickHouse/pull/100651) ([alexey-milovidov](https://github.com/alexey-milovidov)). * `GRANT role TO role` on the same role is now rejected with `BAD_ARGUMENTS` instead of silently creating a self-referential entry in `system.role_grants`. [#103315](https://github.com/ClickHouse/ClickHouse/pull/103315) ([zxuhan](https://github.com/zxuhan)). * `BACKUP` of a `MaterializedPostgreSQL` database no longer hangs forever with "Table ... were created or changed its definition during scanning", and now actually backs up the table data (delegated to the underlying `ReplacingMergeTree`), which can be restored as a standalone `ReplacingMergeTree`. [#107433](https://github.com/ClickHouse/ClickHouse/pull/107433) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Allow `EXISTS ` for a user that has only the `SHOW DICTIONARIES` privilege on the dictionary (previously it required `SHOW TABLES`). [#108084](https://github.com/ClickHouse/ClickHouse/pull/108084) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed cleanup of a `BACKUP ... TO Memory(...)` that fails before finalization: the failed backup was left registered so its name could not be reused. [#109947](https://github.com/ClickHouse/ClickHouse/pull/109947) ([jkartseva](https://github.com/jkartseva)). * Fixed a `ReadBuffer is canceled. Can't read from it.` logical error (server abort in debug/sanitizer builds) that could occur while reading a zip archive (e.g. during `RESTORE` from a zip backup) after a prior read from the same archive failed mid-stream. [#110197](https://github.com/ClickHouse/ClickHouse/pull/110197) ([groeneai](https://github.com/groeneai)). * Fixes heap corruption, which could crash the server at shutdown, caused by the `const` method `AccessRights::getFilters` modifying an access-rights tree shared by concurrent queries. [#112692](https://github.com/ClickHouse/ClickHouse/pull/112692) ([groeneai](https://github.com/groeneai)).

Distributed execution, tables, views, and dictionaries

* Fixed a spurious `Inconsistent AST formatting` error for `ALTER TABLE ... MOVE PART ... TO SHARD ''`. The query formatter dropped the `SHARD` keyword, so the reformatted query could not be parsed back, raising a `LOGICAL_ERROR` (a handled exception in release builds, an abort in debug and sanitizer builds). [#109060](https://github.com/ClickHouse/ClickHouse/pull/109060) ([groeneai](https://github.com/groeneai)). * Fixed a logical error (`!part.empty()`) when a `Distributed` table defined with an empty remote database name is used in a JOIN with `distributed_product_mode = 'local'`. [#110677](https://github.com/ClickHouse/ClickHouse/pull/110677) ([groeneai](https://github.com/groeneai)). * Fixed `DISTINCT` with an ordered `LIMIT`/`OFFSET` returning wrong rows when the DISTINCT early-stop limit hint could not bound the head of the result: a negative `LIMIT` (returned the head instead of the tail), a fractional `LIMIT`/`OFFSET` (the fraction is only resolved after a full read), and a bare `OFFSET` with no `LIMIT` (the tail after the offset was dropped). For example `SELECT DISTINCT intDiv(x, 100) FROM t ORDER BY intDiv(x, 100) LIMIT -1` over a multi-part table returned the smallest distinct value instead of the largest. [#111326](https://github.com/ClickHouse/ClickHouse/pull/111326) ([groeneai](https://github.com/groeneai)). * Fixes the case where an alias after a subquery in DESCRIBE TABLE is not accepted by the parser and results in a syntax error. Fixes [https://github.com/ClickHouse/ClickHouse/issues/100031](https://github.com/ClickHouse/ClickHouse/issues/100031). [#100205](https://github.com/ClickHouse/ClickHouse/pull/100205) ([yariks5s](https://github.com/yariks5s)). * Fixed a server abort (`Logical error: 'Block structure mismatch'`) that could occur during a concurrent `INSERT` while an `ALTER RENAME COLUMN` is in flight in an `Atomic` database with `lazy_load_tables = 1` after `DETACH DATABASE` / `ATTACH DATABASE`. [#104852](https://github.com/ClickHouse/ClickHouse/pull/104852) ([groeneai](https://github.com/groeneai)). * Fix column-name corruption and `BAD_ARGUMENTS` exception caused by the setting `inject_random_order_for_select_without_order_by`. With the setting enabled, multi-column SELECTs no longer fail and output column names (e.g. for `CREATE TABLE AS SELECT`, `CREATE VIEW AS SELECT`, `JSONEachRow`) are preserved instead of being replaced with internal `__subquery_column_` aliases. [#105896](https://github.com/ClickHouse/ClickHouse/pull/105896) ([groeneai](https://github.com/groeneai)). * Fix confusing "Maybe you meant X?" hint after a server restart (or `DETACH/ATTACH DATABASE`), where dropping an already-dropped table would suggest the just-dropped name as the alternative. The async-load tasks are now cleared from `DatabaseOrdinary::startup_table` and `load_table` on detach. [#106238](https://github.com/ClickHouse/ClickHouse/pull/106238) ([groeneai](https://github.com/groeneai)). * Fix the `MaterializedPostgreSQL` table and database engines so that a single table or database can be replicated from a non-default PostgreSQL schema (`materialized_postgresql_schema`), including the case where tables with the same name exist in several schemas of the same database (previously they would share a publication and replication slot and cross-talk). [#107425](https://github.com/ClickHouse/ClickHouse/pull/107425) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `NO_SUCH_COLUMN_IN_TABLE` when a query with parallel replicas selects an `ALIAS` column of a table shipped as a `GLOBAL JOIN` temporary table, `MULTIPLE_EXPRESSIONS_FOR_ALIAS` when both sides of a distributed `JOIN` declare an `ALIAS` column with the same name, and `UNKNOWN_IDENTIFIER` when a `Distributed` table declares an `ALIAS` column over another `ALIAS` column, including as a `JOIN USING` key. [#107700](https://github.com/ClickHouse/ClickHouse/pull/107700) ([yakov-olkhovskiy](https://github.com/yakov-olkhovskiy)). * Fixed a signed integer overflow when a refreshable materialized view retries a failed refresh with `refresh_retries` set to a very large value (near `Int64` max). The overflow could abort the server in builds with the undefined-behavior sanitizer. [#108005](https://github.com/ClickHouse/ClickHouse/pull/108005) ([groeneai](https://github.com/groeneai)). * Fixed an exception (`CANNOT_PARSE_TEXT`) when loading a dictionary with a composite key whose key columns are not the first columns in the dictionary definition, with a local `ClickHouse` source using an explicit query. Source columns are now matched to the dictionary structure by name instead of by position. [#108053](https://github.com/ClickHouse/ClickHouse/pull/108053) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a crash (null pointer dereference) when querying a `Hive` table without a `WHERE` clause, e.g. `SELECT * FROM hive_table`. [#108094](https://github.com/ClickHouse/ClickHouse/pull/108094) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a server crash (segmentation fault) that could occur when a distributed query referenced a not-yet-materialized `MATERIALIZED` CTE as an external table and the remote source was created lazily (delayed source). The crash was a null `Logger` dereference inside `RemoteQueryExecutor::sendExternalTables()`. [#108547](https://github.com/ClickHouse/ClickHouse/pull/108547) ([groeneai](https://github.com/groeneai)). * Fix a logical error (`Equal values are not contiguous within the range assumed to be sorted`) when running `DISTINCT` over a `STREAM` read (`SELECT DISTINCT ... FROM table STREAM`). Read-in-order optimizations are no longer applied to `STREAM` reads, which produce rows in commit order rather than sorting-key order. [#108568](https://github.com/ClickHouse/ClickHouse/pull/108568) ([groeneai](https://github.com/groeneai)). * Fix an exception (`Cannot find sharding key column`, and in debug/sanitizer builds a server abort) when a `Distributed` table's sharding key is an expression the analyzer const-folds, for example `if(1, toInt32(id), toInt32(id) + 1)`, with `optimize_skip_unused_shards = 1`. [#108737](https://github.com/ClickHouse/ClickHouse/pull/108737) ([groeneai](https://github.com/groeneai)). * Fixed a server abort (`LOGICAL_ERROR` / assertion in debug and sanitizer builds) when an `INSERT ... SELECT` through an `Alias` table is cancelled without an exception, for example with `timeout_overflow_mode = 'break'`. [#108783](https://github.com/ClickHouse/ClickHouse/pull/108783) ([groeneai](https://github.com/groeneai)). * Fixed a server abort (`LOGICAL_ERROR` / assertion in debug and sanitizer builds) when an `INSERT ... SELECT` into a `TimeSeries` table is cancelled without an exception, for example with `timeout_overflow_mode = 'break'`. [#108796](https://github.com/ClickHouse/ClickHouse/pull/108796) ([tiandiwonder](https://github.com/tiandiwonder)). * Fix distributed queries occasionally failing with `UNEXPECTED_PACKET_FROM_SERVER` ("expected TablesStatusResponse, got ProfileInfo") when a connection that a previous cancelled query left out of sync was reused from the pool. [#108854](https://github.com/ClickHouse/ClickHouse/pull/108854) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix a `LOGICAL_ERROR` (`CTE '...' does not have query tree, but was not planned yet`) when a `MATERIALIZED` CTE is referenced from more than one join-tree position (for example two joined subqueries, or a joined subquery plus a scalar subquery in `WHERE`) in a query over a `Distributed` table. The query now executes instead of raising the exception (it aborted the server in debug/sanitizer builds). [#108924](https://github.com/ClickHouse/ClickHouse/pull/108924) ([groeneai](https://github.com/groeneai)). * Fixes a memory safety issue where asynchronous work scheduled from materialized view processing could keep using query-level accounting after the query had finished. [#108988](https://github.com/ClickHouse/ClickHouse/pull/108988) ([filimonov](https://github.com/filimonov)). * Fix a `LOGICAL_ERROR` ("Cannot find `__grouping_set` column in header of MergingAggregatedTransform with grouping sets", or "Chunk info was not set for chunk in MergingAggregatedTransform") that could occur, without the analyzer, for a `UNION ALL`/`INTERSECT`/`EXCEPT` where one branch uses `GROUP BY GROUPING SETS` with parallel replicas and another branch uses `FINAL`. [#109003](https://github.com/ClickHouse/ClickHouse/pull/109003) ([groeneai](https://github.com/groeneai)). * Fix `CREATE TABLE ... AS SELECT FROM s3Cluster(...)` (and other cluster table functions such as `fileCluster`/`urlCluster`) failing with `NOT_FOUND_COLUMN_IN_BLOCK` inside a `Replicated` database. [#109266](https://github.com/ClickHouse/ClickHouse/pull/109266) ([groeneai](https://github.com/groeneai)). * Fixed an error (`NOT_FOUND_COLUMN_IN_BLOCK`, or `std::bad_function_call` in older versions) when running `CREATE TABLE ... ON CLUSTER ... AS SELECT` reading a `Distributed` table on a cluster with two or more shards. [#109407](https://github.com/ClickHouse/ClickHouse/pull/109407) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a confusing internal error (`Method getResultType is not supported for TABLE query tree node`) when a table expression was used as the left argument of the `IN` operator; such queries now produce a clear error message. [#109412](https://github.com/ClickHouse/ClickHouse/pull/109412) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix a `Bad cast from type DB::FunctionNode to DB::ConstantNode` logical error (server abort in debug/sanitizer builds) when running `SELECT ... ORDER BY ... WITH FILL` against a `Distributed` table with a low `optimize_const_name_size`. [#109938](https://github.com/ClickHouse/ClickHouse/pull/109938) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` (`Left and right columns have same names`) in the join order optimizer that could be triggered by a comma join over a multi-table view when `query_plan_merge_expression_into_join` is enabled. [#110227](https://github.com/ClickHouse/ClickHouse/pull/110227) ([groeneai](https://github.com/groeneai)). * Fix a rare `Logical error: 'No more packets are available.'` raised for a valid distributed query with a `LIMIT`, which aborted the server in debug and sanitizer builds. [#110334](https://github.com/ClickHouse/ClickHouse/pull/110334) ([groeneai](https://github.com/groeneai)). * Fixed a parser bug where COMMENT was incorrectly consumed as an implicit alias when a SELECT query ended directly after a bare table identifier (e.g. `... FROM t COMMENT 'x'`), causing a misleading syntax error instead of the comment being applied to the view/table. [#110372](https://github.com/ClickHouse/ClickHouse/pull/110372) ([adityaksolves](https://github.com/adityaksolves)). * Fixed a wrong result where `length()` (and the QBit dimension) on a `FixedString` column returned `0` instead of the fixed size `N` under parallel replicas when the query had a selective filter. [#110427](https://github.com/ClickHouse/ClickHouse/pull/110427) ([groeneai](https://github.com/groeneai)). * Fix `NOT_FOUND_COLUMN_IN_BLOCK` error for queries with a subquery in `FROM` when parallel replicas run in `custom_key_sampling` / `custom_key_range` mode. Such queries (e.g. `SELECT * FROM (SELECT id, k, v FROM t WHERE id < 20) ORDER BY k`) previously failed with `Not found column __table2.id in block ...`. [#110690](https://github.com/ClickHouse/ClickHouse/pull/110690) ([groeneai](https://github.com/groeneai)). * Fix `Logical error: 'clock'` (in debug/sanitizer builds the server dies; an infinite hang in release builds) when running `EXPLAIN ANALYZE` over a query with a streaming (`FROM ... STREAM`) read that is not a plain top-level table expression: nested in a subquery (for example inside `WHERE ... IN (...)` or a CTE), or hidden behind an ordinary or parameterized view. `EXPLAIN ANALYZE` now rejects such queries with `NOT_IMPLEMENTED`, the same way it already rejects a top-level streaming read. [#110935](https://github.com/ClickHouse/ClickHouse/pull/110935) ([groeneai](https://github.com/groeneai)). * Fixes the resolution of fully qualified column names such as `db.table.column` in the analyzer for tables that have the same name as their database. [#110976](https://github.com/ClickHouse/ClickHouse/pull/110976) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `NOT_FOUND_COLUMN_IN_BLOCK` / `THERE_IS_NO_COLUMN` errors (and, on some releases, silently wrong results) for a `RIGHT JOIN` when `parallel_replicas_min_number_of_rows_per_replica` is set and a left-table column is projected. [#111332](https://github.com/ClickHouse/ClickHouse/pull/111332) ([groeneai](https://github.com/groeneai)). * Fix a race in executable UDFs, the `executable` table engine, and executable dictionaries where an `fcntl` on an already-closed file descriptor could corrupt an unrelated descriptor of a concurrent query (stripping `O_NONBLOCK`), which could make queries hang indefinitely. [#111779](https://github.com/ClickHouse/ClickHouse/pull/111779) ([Algunenano](https://github.com/Algunenano)). * Fixed a `LOGICAL_ERROR` ("Invalid number of rows in Chunk") — and, under hardened builds, a related out-of-bounds abort — for `ORDER BY ... WITH FILL ... INTERPOLATE` over a `Distributed` table with two or more shards, over `clusterAllReplicas`, or over custom-key parallel replicas, including on empty results. `WITH FILL` is now applied only on the node producing the final result, which also fixes duplication of the generated fill rows across shards/replicas. [#111919](https://github.com/ClickHouse/ClickHouse/pull/111919) ([yakov-olkhovskiy](https://github.com/yakov-olkhovskiy)). * Fixed `DROP TABLE ... SYNC` hanging forever when a query over `system.replicas` had been killed before its status requests were processed. [#112164](https://github.com/ClickHouse/ClickHouse/pull/112164) ([evillique](https://github.com/evillique)). * Fixes a `LOGICAL_ERROR` exception (`Block structure mismatch in joined block stream`) on a multi-table join whose two sub-joins both produce no output columns, for example `SELECT count() FROM t1, t2, t3, t4 WHERE (t1.b = t2.b) AND (t3.a = t4.a)` with `query_plan_optimize_join_order_limit` set to `0` or `1`. [#112205](https://github.com/ClickHouse/ClickHouse/pull/112205) ([groeneai](https://github.com/groeneai)). * Fixes a logical error `Cannot add step Expression to QueryPlan because it has incompatible header with root step JoinLazyColumnsStep` for a query with `FINAL`, a `PREWHERE` filter, a small `LIMIT` and `make_distributed_plan = 1`, when the selected columns are listed in an order that differs from the table's column order. Lazy materialization replaced a plan node without preserving that node's output header, which made the query plan inconsistent. [#112303](https://github.com/ClickHouse/ClickHouse/pull/112303) ([groeneai](https://github.com/groeneai)). * Fixes wrong results and a logical error when an argument that must be constant, such as the scale of `toDecimal32` or the timezone of `toDateTime`, comes from a scalar subquery and the old analyzer is used. Such an argument was analyzed as if its value were zero or empty, so the declared result type disagreed with the computed value, and `CREATE VIEW` and `CREATE MATERIALIZED VIEW` persisted a wrong column type. [#112492](https://github.com/ClickHouse/ClickHouse/pull/112492) ([groeneai](https://github.com/groeneai)). * Fixes a logical error and a wrong dependency record when an ordinary `VIEW` body references a table or dictionary through a query parameter, as in `CREATE VIEW v AS SELECT x FROM {db:Identifier}.t`. Such a view no longer records a referential dependency on a same-named table of the current database, so unrelated tables can be dropped again. [#112704](https://github.com/ClickHouse/ClickHouse/pull/112704) ([groeneai](https://github.com/groeneai)). * Fixes a logical error when a `Join` table engine is used on the right side of a JOIN whose `ON` section contains a condition referencing columns of both tables, for example `ON (t.key = j.key) AND (t.a < j.a)`. Such a query is now rejected with a clear exception instead of raising `required columns: ... but not found any in left table` or `std::bad_variant_access`. [#112815](https://github.com/ClickHouse/ClickHouse/pull/112815) ([groeneai](https://github.com/groeneai)). * Fixed undefined behavior and a silent scheduling error when a background task is scheduled with a very large delay, for example by a refreshable materialized view with a huge `REFRESH AFTER` period. Delays are now bounded to the largest representable value instead of overflowing. [#113041](https://github.com/ClickHouse/ClickHouse/pull/113041) ([groeneai](https://github.com/groeneai)). * Fixed the error reported when an asynchronous `INSERT` into a `Distributed` table needs a queue directory whose name exceeds the 255-byte filesystem limit (reachable with `use_compact_format_in_distributed_parts_names = 0`). It was a `LOGICAL_ERROR` for a shard with `internal_replication` and an unattributed `Code: 1001. std::exception` otherwise; both now report `ARGUMENT_OUT_OF_BOUND` naming the table, the cluster and the limit. [#113083](https://github.com/ClickHouse/ClickHouse/pull/113083) ([groeneai](https://github.com/groeneai)). * `UNDROP TABLE` can now be interrupted by `KILL QUERY` while it waits for running queries to release the dropped table. [#113263](https://github.com/ClickHouse/ClickHouse/pull/113263) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a permanent `DROP TABLE` hang and memory leak after a query using a materialized CTE failed. The table stayed stuck in the drop queue until the server was restarted. [#113397](https://github.com/ClickHouse/ClickHouse/pull/113397) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixes a segfault and a `LOGICAL_ERROR` (`No host found for exchange stream final_result__0_0`) when a subquery sets `distributed_plan_execute_locally` in its own `SETTINGS` clause under `make_distributed_plan`. The distributed plan and the code executing it read the setting from two different contexts and could disagree, so the initiator built a self-contradictory pipeline. [#113422](https://github.com/ClickHouse/ClickHouse/pull/113422) ([groeneai](https://github.com/groeneai)). * Fixed a `CREATE TABLE ... AS SELECT` whose populating `SELECT` failed becoming unkillable when `database_atomic_wait_for_drop_and_detach_synchronously` is enabled: the internal cleanup `DROP` of the temporary table waited for the background drop queue and ignored both `KILL QUERY` and `max_execution_time`. [#113504](https://github.com/ClickHouse/ClickHouse/pull/113504) ([groeneai](https://github.com/groeneai)). * Fix wrong results and, in a build with assertions enabled, an aborted assertion for a `JOIN` onto a dictionary whose `ON` clause has a non-equi condition over both tables, such as `ON (t.key = d.key) AND (t.a * 10 < d.a)`, when `join_use_nulls` is enabled. The condition was evaluated over a column read through a mismatched type, so rows could match arbitrarily. [#113534](https://github.com/ClickHouse/ClickHouse/pull/113534) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a parameterized view losing its database when a query referencing it unqualified is sent to a shard, so the shard resolved the name against its own default database. Depending on what that database contained, the query either failed with `UNKNOWN_FUNCTION` or silently returned rows from a different view. Affects `IN`, `JOIN` and `remote()` through a `Distributed` table. [#113550](https://github.com/ClickHouse/ClickHouse/pull/113550) ([groeneai](https://github.com/groeneai)). * Fixed `Unsupported JOIN keys of type keys256 in StorageJoin` when reading a `Join`-engine table whose key is composite (several columns) or a single 16/32-byte value such as `UUID`, `Int256` or `FixedString(4)`. Such a table could be created, written and joined against, but never read back with `SELECT`. [#113578](https://github.com/ClickHouse/ClickHouse/pull/113578) ([groeneai](https://github.com/groeneai)). * Fix a logical error `Sending a distributed query with unknown (zero) client version` when a structure-less `Distributed` table over a remote shard is attached or loaded from metadata written by an older server version. [#113675](https://github.com/ClickHouse/ClickHouse/pull/113675) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a `LOGICAL_ERROR` when a `SELECT` used the `STREAM` modifier on a named table together with parallel replicas in the read-tasks mode. With a materialized CTE used as an `IN` set in `PREWHERE`, the server raised `Reading from materialized CTE ... DelayedPortsProcessor gate is missing in the query plan` (a server abort in debug and sanitizer builds). A table carrying `STREAM` is now rejected when parallel replicas are requested: at `enable_parallel_replicas = 2` the query fails with `SUPPORT_IS_DISABLED`, at `1` it runs without them, exactly as already happens for `FINAL`. [#113754](https://github.com/ClickHouse/ClickHouse/pull/113754) ([groeneai](https://github.com/groeneai)). * Fix quadratic complexity of parsing the address argument of the `remote`, `mysql`, and similar table functions: a very long address (e.g. a megabyte-sized zero-padded `FixedString`) made the query hang during analysis, where it could not be cancelled. [#113918](https://github.com/ClickHouse/ClickHouse/pull/113918) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a spurious `Failed to drop temporary table after refresh. Table ... is left behind and requires manual cleanup.` error logged when a refreshable materialized view's refresh is cancelled (for example by a server shutdown) while `database_atomic_wait_for_drop_and_detach_synchronously` is enabled. The temporary table was in fact dropped, so no manual cleanup was ever needed. [#113957](https://github.com/ClickHouse/ClickHouse/pull/113957) ([groeneai](https://github.com/groeneai)). * Fixes a missing `system.query_views_log` row for a materialized view that fails while its dependencies are being collected with `materialized_views_ignore_errors` enabled. In debug and sanitizer builds this also aborts the server with `std::out_of_range` instead of logging the view. [#114064](https://github.com/ClickHouse/ClickHouse/pull/114064) ([groeneai](https://github.com/groeneai)). * Fix inconsistent results when a filter on a non-key column of the right table was pushed down below an ANY INNER JOIN. [#114313](https://github.com/ClickHouse/ClickHouse/pull/114313) ([kirillshokhin](https://github.com/kirillshokhin)). * Fixes `CREATE OR REPLACE` leaking an internal `_tmp_replace_*` table, and `CREATE OR REPLACE VIEW` failing with `NOT_IMPLEMENTED`, when `ignore_drop_queries_probability` is set. The DROPs it issues internally are steps of one user statement, so DROP fault injection no longer applies to them. [#114420](https://github.com/ClickHouse/ClickHouse/pull/114420) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` when reading a Delta Lake table whose `metaData.partitionColumns` names a column that `metaData.schemaString` does not declare. Such metadata is now rejected with `BAD_ARGUMENTS` on the delta-kernel reader and `INCORRECT_DATA` on the legacy reader, whenever the table or its schema is read from a snapshot, not only when the query carries a predicate. [#114538](https://github.com/ClickHouse/ClickHouse/pull/114538) ([groeneai](https://github.com/groeneai)). * Fixed hash-table statistics — and with them aggregation hash-table preallocation — being silently and permanently disabled for the whole server when a query using the lazy `FINAL` optimization (`query_plan_optimize_lazy_final`) was the first aggregation to run after server startup. [#114597](https://github.com/ClickHouse/ClickHouse/pull/114597) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a refreshable materialized view leaking its rotated-out target table when `ignore_drop_queries_probability` is enabled. The `DROP` a refresh issues to clean up the previous target is a step of the refresh, not a `DROP` the user asked for, so the fault injection no longer applies to it. [#114622](https://github.com/ClickHouse/ClickHouse/pull/114622) ([groeneai](https://github.com/groeneai)). * Fix `ANY RIGHT JOIN` and `SEMI RIGHT JOIN` with several `OR`-ed conditions in the `ON` section returning some rows of the right table twice and losing the matches of others. A written `ANY LEFT JOIN` was affected too, because the planner may swap the tables and execute it as `ANY RIGHT JOIN`. [#114676](https://github.com/ClickHouse/ClickHouse/pull/114676) ([vdimir](https://github.com/vdimir)). * Fixes a `LOGICAL_ERROR` (`Cannot find input column ... on its position in inputs of expression actions DAG`) when a query has a `JOIN` above a table whose text-indexed column is used by both a `PREWHERE` and a `WHERE` text-search predicate. [#114771](https://github.com/ClickHouse/ClickHouse/pull/114771) ([groeneai](https://github.com/groeneai)). * Fixes `NOT_FOUND_COLUMN_IN_BLOCK` on every `INSERT` into a table that declares `STATISTICS(...)` on an `ALIAS` or `EPHEMERAL` column, which made such tables read-only. [#115231](https://github.com/ClickHouse/ClickHouse/pull/115231) ([groeneai](https://github.com/groeneai)). * Fix `dictGet` and its variations in distributed queries: an unqualified dictionary name is now bound to the current database of the initiator, instead of being resolved against the current database of each shard, where it either was not found or resolved to a different dictionary of the same name. [#115541](https://github.com/ClickHouse/ClickHouse/pull/115541) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a bug where restoring a database or table under a new name dropped the table aliases from the bodies of stored views, so reading a restored view failed with `UNKNOWN_IDENTIFIER`. [#115645](https://github.com/ClickHouse/ClickHouse/pull/115645) ([groeneai](https://github.com/groeneai)). * Fix `LOGICAL_ERROR` "Reading from materialized CTE 'X' before its materialization completed - DelayedPortsProcessor gate is missing in the query plan" when a `MATERIALIZED` CTE is read from inside an `IN` subquery over a `Distributed` table. [#115740](https://github.com/ClickHouse/ClickHouse/pull/115740) ([groeneai](https://github.com/groeneai)). * Fix BAD\_ARGUMENTS "Dictionary not found" on dictGet after a DETACH DICTIONARY ... PERMANENTLY that was rejected by the dependency check (HAVE\_DEPENDENT\_OBJECTS). The dictionary now remains fully usable when the detach is rejected. [#105259](https://github.com/ClickHouse/ClickHouse/pull/105259) ([groeneai](https://github.com/groeneai)). * Fixed a `Logical error: 'removed'` (server abort in debug builds) in the background table-drop queue. It could be triggered when the same explicit UUID is reused across several `CREATE OR REPLACE TABLE` queries, which enqueues more than one dropped table sharing that UUID. [#107031](https://github.com/ClickHouse/ClickHouse/pull/107031) ([groeneai](https://github.com/groeneai)). * Fix `MaterializedPostgreSQL` stopping replication of an entire database (with `LOGICAL_ERROR: Columns number mismatch`) when a single replicated table's structure changed while the server was down. Now only the affected table is skipped and can be recovered with `DETACH`/`ATTACH`. [#107427](https://github.com/ClickHouse/ClickHouse/pull/107427) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a severe work-distribution skew with parallel replicas when the cluster contains inactive replicas (for example, stale entries left after autoscaling). Such replicas are no longer counted by the reading coordinator, so work is balanced across the online replicas instead of piling onto a single one. [#107805](https://github.com/ClickHouse/ClickHouse/pull/107805) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed a freeze of `SYSTEM RELOAD DICTIONARIES` (and `RELOAD DICTIONARY`) when many MySQL dictionaries defined in XML shared a single connection pool via `share_connection`. Such pools now honor `connection_pool_size` and `connection_wait_timeout` (default 5 seconds) from the configuration, matching dictionaries defined through named collections. [#108083](https://github.com/ClickHouse/ClickHouse/pull/108083) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Prevent exhausted retries for leader-executed replicated distributed DDL from permanently blocking subsequent `ON CLUSTER` DDL queries. [#108505](https://github.com/ClickHouse/ClickHouse/pull/108505) ([ahaanlimaye](https://github.com/ahaanlimaye)). * Fix a possible crash (use-after-free) when `SYSTEM FLUSH DISTRIBUTED` runs concurrently with `DROP TABLE` of the same `Distributed` table. [#108684](https://github.com/ClickHouse/ClickHouse/pull/108684) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` (`local_replica_plan_reading_step->getAnalyzedResult() == nullptr`) in the automatic parallel replicas planner that could occur when `automatic_parallel_replicas_mode` is enabled together with `parallel_replicas_min_number_of_rows_per_replica` greater than 0. [#109011](https://github.com/ClickHouse/ClickHouse/pull/109011) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` (`creation_csn is not set while removal_csn is set to 1`) that could be thrown when a non-transactional `TRUNCATE` ran concurrently with an uncommitted transaction that had inserted into the same table. Such a `TRUNCATE` no longer removes parts created by not-yet-committed transactions. [#109598](https://github.com/ClickHouse/ClickHouse/pull/109598) ([tuanpach](https://github.com/tuanpach)). * External database engines (e.g. `PostgreSQL`) no longer push down range comparisons (`>=`, `>`, `<=`, `<`) on `UUID` columns. ClickHouse and external databases order UUIDs differently, so pushing these down silently dropped rows from the result; such predicates are now evaluated by ClickHouse instead. [#109833](https://github.com/ClickHouse/ClickHouse/pull/109833) ([vismaytiwari](https://github.com/vismaytiwari)). * Fixed a server crash in the `MaterializedPostgreSQL` database engine that could happen when a table was detached (or its structure changed) while it was still queued for synchronization. [#110596](https://github.com/ClickHouse/ClickHouse/pull/110596) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed the type of an `EPHEMERAL` column being rewritten by the server's `query_masking_rules`. A rule matching inside the type made `CREATE TABLE` store the wrong type, or fail. [#112048](https://github.com/ClickHouse/ClickHouse/pull/112048) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes an unsigned underflow in the table name length check that allowed creating a table which could never be dropped. When the escaped database name reached 214 bytes, the limit computed for the `metadata_dropped` filename wrapped around, so `CREATE TABLE` was accepted and the subsequent `DROP TABLE` or `DROP DATABASE` failed with `File name too long`. Such a name is now rejected with `ARGUMENT_OUT_OF_BOUND`, the same error already returned just below that boundary. [#112527](https://github.com/ClickHouse/ClickHouse/pull/112527) ([groeneai](https://github.com/groeneai)). * Fixed `RENAME TABLE` checking the maximum table name length against the source database instead of the destination one. Renaming a table into a database with a long name could produce a table that `DROP TABLE` could not remove, and renaming a table out of such a database was wrongly rejected. [#113019](https://github.com/ClickHouse/ClickHouse/pull/113019) ([groeneai](https://github.com/groeneai)). * Fixes `views` column in `query_log` for inserts to an `Alias` table whose target triggers materialized views, which was always empty. [#113076](https://github.com/ClickHouse/ClickHouse/pull/113076) ([eclbg](https://github.com/eclbg)). * Invalid `parallel_replicas_custom_key` expressions such as `tuple()` or `(x, y)` are now rejected with `ILLEGAL_TYPE_OF_COLUMN_FOR_FILTER` instead of reading past the custom-key type list. [#113359](https://github.com/ClickHouse/ClickHouse/pull/113359) ([nickitat](https://github.com/nickitat)). * Fixed `RENAME DATABASE` accepting a target name long enough to make the database's tables impossible to drop. The table name length check now runs regardless of `check_table_dependencies`, and it also covers detached tables, which were never checked and so were affected even at default settings. [#113557](https://github.com/ClickHouse/ClickHouse/pull/113557) ([groeneai](https://github.com/groeneai)). * When using background inserts into tables with the `Distributed` engine, delays caused by repeated errors when sending data to remote shards now decrease once the errors are resolved. Previously, the delay would only increase and never reset. [#87378](https://github.com/ClickHouse/ClickHouse/pull/87378) ([Blackmorse](https://github.com/Blackmorse)). * Reject native protocol handshakes that use the `USER_INTERSERVER_MARKER` for an unknown cluster, or for a cluster configured without a ``. Previously such a handshake would let an unauthenticated client enter interserver mode and exercise pre-auth protocol packets (e.g. `TablesStatusRequest`) backed by a `fake_interserver_context`, learning information about local tables without proving knowledge of the cluster secret. [#99675](https://github.com/ClickHouse/ClickHouse/pull/99675) ([MillaFleurs](https://github.com/MillaFleurs)).

SQL, analyzer, and JOIN fixes

* Fix substitution of query parameters inside the definitions of a named `WINDOW` clause. Previously, parameters there were silently kept unsubstituted, and an unset `Identifier` parameter could lead to the exception `Logical error: '!part.empty()'` during `EXPLAIN SYNTAX`. [#110506](https://github.com/ClickHouse/ClickHouse/pull/110506) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix a rare server crash when refreshable materialized views are dropped or replaced concurrently. A null pointer was dereferenced in `RefreshTask::doScheduling` when another thread set the task's `view` to `nullptr` while the scheduling code had temporarily released its mutex. [#105588](https://github.com/ClickHouse/ClickHouse/pull/105588) ([groeneai](https://github.com/groeneai)). * Fix wrong result from a view defined with `EXCEPT` or `INTERSECT` whose operand is a `UNION` chain after `DETACH`/`ATTACH` or server restart. The formatter was missing parentheses around `UNION` children of `INTERSECT`/`EXCEPT`, so the stored SQL reparsed with reversed precedence. [#105935](https://github.com/ClickHouse/ClickHouse/pull/105935) ([groeneai](https://github.com/groeneai)). * Fixed wrong results (duplicate rows) for `SELECT DISTINCT` over a `GROUP BY` with `WITH CUBE`, `WITH ROLLUP`, or `GROUPING SETS` on the same keys. The `query_plan_remove_redundant_distinct` optimization no longer removes the `DISTINCT` in these cases, because the grouping modifiers emit extra rows with the key columns defaulted, which can collide with real group values. [#107072](https://github.com/ClickHouse/ClickHouse/pull/107072) ([groeneai](https://github.com/groeneai)). * Fix a wrong result in an OUTER JOIN when the `WHERE` filter is a conjunction that contains a constant term (for example `p AND 1`, or `WHERE p QUALIFY NULL` where `QUALIFY NULL` is merged into `WHERE`). The constant conjunct could be pushed into the non-preserved side of the join and removed from the post-join filter, so the join produced non-matched rows with the side columns defaulted and those rows incorrectly passed the filter. [#107084](https://github.com/ClickHouse/ClickHouse/pull/107084) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` ("Cannot find input column ... on its position in inputs of expression actions DAG") that aborted the server for a correlated scalar subquery over a source projecting the same column identifier twice (e.g. `SELECT number, *`) when `correlated_subqueries_default_join_kind = 'left'`. [#107112](https://github.com/ClickHouse/ClickHouse/pull/107112) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` (server abort in debug/sanitizer builds) when a `JOIN` uses the null-safe comparison operator (`<=>` / `IS NOT DISTINCT FROM`) and one of the keys is a scalar subquery, e.g. `... JOIN t2 ON (SELECT x FROM t1) <=> t2.k`, with the old analyzer (`enable_analyzer = 0`). Such a query now returns a regular `NOT_FOUND_COLUMN_IN_BLOCK` error instead of a logical error. [#108123](https://github.com/ClickHouse/ClickHouse/pull/108123) ([groeneai](https://github.com/groeneai)). * Fixed `LOGICAL_ERROR` (`Join is supported only for pipelines with one output port`) and `NOT_IMPLEMENTED` (`MergeJoinAlgorithm is not implemented for strictness Semi`) exceptions when `join_algorithm = 'full_sorting_merge'` was used for a `SEMI`/`ANTI` join, including joins produced by decorrelating an `EXISTS` correlated subquery. `full_sorting_merge` now declines strictness/kind combinations it cannot execute, so the planner falls back to another enabled algorithm or reports that the join cannot be executed. [#108658](https://github.com/ClickHouse/ClickHouse/pull/108658) ([groeneai](https://github.com/groeneai)). * Fixed a bug where the `WITH TOTALS` row of a `JOIN` could contain default values (such as `0`) instead of the constant values coming from a constant subquery on one side of the join. The wrong result appeared only for some join orderings or with the `query_plan_join_swap_table` setting. [#108807](https://github.com/ClickHouse/ClickHouse/pull/108807) ([vdimir](https://github.com/vdimir)). * Fixed a startup abort (`Cannot allocate ThreadStack`, `EINVAL`) on glibc builds running on CPUs with a large signal-stack size (for example Intel Sapphire Rapids / Granite Rapids with an AMX-aware kernel), where the signal alt-stack size was not rounded up to a multiple of the page size before `aligned_alloc`. [#108813](https://github.com/ClickHouse/ClickHouse/pull/108813) ([groeneai](https://github.com/groeneai)). * Fix a `Block structure mismatch in JoinStep` logical error (server abort in debug/sanitizer builds) when a correlated subquery is decorrelated into a join and the source relation carries a column name more than once, e.g. `WITH t AS (SELECT number, * FROM numbers(3)) SELECT *, (SELECT t.number WHERE t.number >= 0) FROM t` with `correlated_subqueries_default_join_kind = 'right'`. [#109114](https://github.com/ClickHouse/ClickHouse/pull/109114) ([groeneai](https://github.com/groeneai)). * Fixed `CREATE USER ... HOST LIKE 'a', 'b'` (and the equivalent `ALTER USER`) silently keeping only the first pattern. All specified `HOST LIKE` patterns are now stored in the user's host allow-list. [#109187](https://github.com/ClickHouse/ClickHouse/pull/109187) ([groeneai](https://github.com/groeneai)). * Fix unexpected result for `ANY JOIN` with a constant `ON` condition (e.g. `t1 ANY INNER JOIN t2 ON 1`): the query returned the full cartesian product instead of the ANY join result [#109207](https://github.com/ClickHouse/ClickHouse/pull/109207) ([vdimir](https://github.com/vdimir)). * Fixed the global `max_rows_in_join` / `max_bytes_in_join` limit not being enforced for the parallel\_hash join algorithm, where a query with `join_overflow_mode = 'throw'` could silently succeed instead of raising `SET_SIZE_LIMIT_EXCEEDED`. [#109488](https://github.com/ClickHouse/ClickHouse/pull/109488) ([m-selmi](https://github.com/m-selmi)). * Fixed silent data loss when an `INSERT ... VALUES` in a multi-query stream is followed by a trailing SQL comment (for example `VALUES (1) -- comment`) under the server-side inline insert parsing path (`send_table_structure_on_insert_with_inline_data = 0`). The trailing comment was scanned as row data past the terminating `;`, causing the following queries in the stream to be silently skipped. [#109643](https://github.com/ClickHouse/ClickHouse/pull/109643) ([groeneai](https://github.com/groeneai)). * Fixed deferred `PREWHERE` with `optimize_move_to_prewhere = 1` when `PREWHERE` runs after `FINAL`: `WHERE` conditions are no longer moved ahead of a deferred row policy, which could change query results. [#109703](https://github.com/ClickHouse/ClickHouse/pull/109703) ([yariks5s](https://github.com/yariks5s)). * Fix a `LOGICAL_ERROR` (`query tree node does not have valid source node`) when a recursive CTE resolved an identifier from an outer scope as a correlated column (with `allow_experimental_correlated_subqueries = 1`). Such queries now return a clear `UNSUPPORTED_METHOD` error. [#109863](https://github.com/ClickHouse/ClickHouse/pull/109863) ([groeneai](https://github.com/groeneai)). * Fix a signed integer overflow when `ORDER BY ... WITH FILL` skips a very large gap over an `Int64`/`UInt64` column (e.g. a step of 2 across a gap near 2^63). The overflow was undefined behavior under sanitizers and silently wrapped in release builds. [#109937](https://github.com/ClickHouse/ClickHouse/pull/109937) ([groeneai](https://github.com/groeneai)). * Fixed a `Invalid number of rows in Chunk` logical error when an `INTERPOLATE` target is an alias of a `WITH FILL` column with the old analyzer (`enable_analyzer = 0`). Such queries are now rejected with a clear `INVALID_WITH_FILL_EXPRESSION` error. [#110103](https://github.com/ClickHouse/ClickHouse/pull/110103) ([groeneai](https://github.com/groeneai)). * Fix a logical error (`Unexpected number of columns in result sample block`) when a filter containing a correlated subquery (for example `exists((SELECT ...))`) is optimized with `convert_query_to_cnf` or `optimize_and_compare_chain`. [#110187](https://github.com/ClickHouse/ClickHouse/pull/110187) ([alexey-milovidov](https://github.com/alexey-milovidov)). * A nested correlated `EXISTS` subquery that references a column from a scope beyond its immediate outer query (skipping an intermediate scope) now fails with a clear `NOT_IMPLEMENTED` error instead of an internal `NOT_FOUND_COLUMN_IN_BLOCK`. [#110310](https://github.com/ClickHouse/ClickHouse/pull/110310) ([groeneai](https://github.com/groeneai)). * Fixed wrong results from correlated scalar subqueries when the outer query has duplicate values for the correlated column and the subquery is decorrelated via the CROSS JOIN path. Previously aggregates such as `count()` and `sum()` were multiplied by the duplicate count. This affects the experimental `allow_experimental_correlated_subqueries` feature. [#110311](https://github.com/ClickHouse/ClickHouse/pull/110311) ([groeneai](https://github.com/groeneai)). * Fixed `ORDER BY ... WITH FILL` not respecting `max_execution_time` and being slow to cancel when generating a large fill range (especially with `INTERPOLATE`). [#110332](https://github.com/ClickHouse/ClickHouse/pull/110332) ([alexey-milovidov](https://github.com/alexey-milovidov)). * The lazy `FINAL` optimization (`query_plan_optimize_lazy_final`, disabled by default) is no longer applied when the query reads in the order of the sorting key, because its replacement plan does not produce rows in sorting-key order, while the plan above may already rely on that after the read-in-order optimization, which could result in incorrectly ordered results. It is also no longer applied when a`LIMIT` smaller than the number of rows to read applies directly to the reading step, where the mandatory set-building pass would defeat early termination. [#110576](https://github.com/ClickHouse/ClickHouse/pull/110576) ([KochetovNicolai](https://github.com/KochetovNicolai)). * A background asynchronous insert flush now stops promptly when killed with `KILL QUERY`; previously a flush of a large buffered payload kept parsing to the end and ignored the cancellation. [#110652](https://github.com/ClickHouse/ClickHouse/pull/110652) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed wrong results from `GROUP BY ... optimize_aggregation_in_order` and `DISTINCT ... optimize_distinct_in_order` placed over a `partial_merge` JOIN. The read-in-order optimization was propagated through `PartialMergeJoin`, which re-sorts its left input by the join key and therefore does not preserve the left stream's original order, so rows were grouped incorrectly. [#110671](https://github.com/ClickHouse/ClickHouse/pull/110671) ([groeneai](https://github.com/groeneai)). * Fixes a case when using CTE column aliases with `INTERSECT` or `EXCEPT` queries, e.g. `WITH t (a, b) AS (SELECT 1, 2 INTERSECT SELECT 1, 2) SELECT a, b FROM t`, the query was failing with UNKNOWN\_IDENTIFIER error. [#110885](https://github.com/ClickHouse/ClickHouse/pull/110885) ([yariks5s](https://github.com/yariks5s)). * Fixed a `LOGICAL_ERROR: ReadBuffer is canceled. Can't read from it.` that could occur in a `grace_hash` join when reading spilled blocks failed mid-read while multiple threads shared the reader. [#111056](https://github.com/ClickHouse/ClickHouse/pull/111056) ([groeneai](https://github.com/groeneai)). * Fixes a bug where materializing the right-side columns during \*\_hash JOINs was involving the wrong timezone. [#111074](https://github.com/ClickHouse/ClickHouse/pull/111074) ([yariks5s](https://github.com/yariks5s)). * Fix `CREATE` statements with a `COMMENT` clause to raise a syntax error when the comment string literal is missing, instead of silently accepting a misparsed query (e.g. `CREATE VIEW v COMMENT AS SELECT 1` no longer creates a view without a comment). [#111159](https://github.com/ClickHouse/ClickHouse/pull/111159) ([tiandiwonder](https://github.com/tiandiwonder)). * Fix a `LOGICAL_ERROR` (`Reading from materialized CTE '...' before its materialization completed - DelayedPortsProcessor gate is missing in the query plan`) when a reused materialized CTE (`enable_materialized_cte`) is read on a branch whose `WITH TOTALS` or `extremes` output is discarded, including outer aggregation, `INSERT ... SELECT`, set operations and `EXPLAIN ANALYZE`. [#111194](https://github.com/ClickHouse/ClickHouse/pull/111194) ([groeneai](https://github.com/groeneai)). * Fix `join_algorithm = 'full_sorting_merge'` filling the non-joined rows of a `LEFT`/`RIGHT`/`FULL`/`ASOFJOIN` with a raw zero instead of the column type default. For an `Enum` column (whose default is its first element, not `0`) this produced wrong results (a filter on the padded value dropped the unmatched rows, so an outer join silently behaved like an inner one) and a `UNKNOWN_ELEMENT_OF_ENUM` error when the column was selected. [#111197](https://github.com/ClickHouse/ClickHouse/pull/111197) ([groeneai](https://github.com/groeneai)). * Fix wrong results from `SummingMergeTree` (and `CoalescingMergeTree`) with `FINAL` when `optimize_move_to_prewhere_if_final = 1` and the `WHERE` clause is an implicit-boolean condition on a sorting-key column (e.g. `WHERE k`). The key column was returned summed across the unmerged parts instead of preserved. [#111342](https://github.com/ClickHouse/ClickHouse/pull/111342) ([groeneai](https://github.com/groeneai)). * Fix a logical error (`Unexpected number of columns in result sample block`) that could occur for a `RIGHT`/`FULL JOIN` when the left side carries a duplicated same-named column (for example `1 AS b, 1 AS b`) that survives through `UNION ALL`. [#111554](https://github.com/ClickHouse/ClickHouse/pull/111554) ([groeneai](https://github.com/groeneai)). * Fixes a client crash when formatting an `EXECUTE AS` query whose AST has a child replaced (as the AST fuzzer does). `ASTExecuteAsQuery` now owns its `target_user` and `subquery` children instead of holding non-owning pointers into `children`. [#111725](https://github.com/ClickHouse/ClickHouse/pull/111725) ([groeneai](https://github.com/groeneai)). * Fixed wrong results when `query_plan_join_swap_table` swaps an `ANY JOIN` while `join_any_take_last_row = 1`: ClickHouse now keeps the original join-side order so `ANY JOIN` still returns the last matching row. [#112458](https://github.com/ClickHouse/ClickHouse/pull/112458) ([vdimir](https://github.com/vdimir)). * Fix the `Context has expired` exception when `getClientHTTPHeader` is used inside a subquery. [#112533](https://github.com/ClickHouse/ClickHouse/pull/112533) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes wrong results for a `JOIN` whose `ON` clause carries a mixed condition, that is a cross-side non-equi residual such as `ON (t1.key = t2.key) AND (t1.a * 10 < t2.a)`. Only the hash family evaluates such a condition, but `full_sorting_merge`, `partial_merge` and the direct key-value join claimed they could run these joins and then silently ignored the residual, returning extra rows. They now decline, so a hash algorithm is used instead. [#112831](https://github.com/ClickHouse/ClickHouse/pull/112831) ([groeneai](https://github.com/groeneai)). * Fixed wrong results for a comma or `CROSS` join whose `WHERE` compares columns of two tables through a non-deterministic expression such as `rand`. Such a query could return rows that contradict its own `WHERE` clause, because rewriting the join into an `INNER` join moved the expression into the join key, where it is evaluated again per row of each side. [#112938](https://github.com/ClickHouse/ClickHouse/pull/112938) ([groeneai](https://github.com/groeneai)). * Fix a quadratic slowdown when parsing a PromQL query with a long tail of unrecognized characters (e.g. a `FixedString` value padded with NUL bytes): the parse of a megabyte-sized invalid input used to keep a thread busy for tens of minutes and was uncancellable, because it happens during query analysis. Now the parser stops at the first error. [#112944](https://github.com/ClickHouse/ClickHouse/pull/112944) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `intDiv` on a non-constant column with the constant divisor -1 silently wrapping the division of the minimal signed number (e.g. `intDiv(-9223372036854775808, -1)`) instead of throwing the `ILLEGAL_DIVISION` exception like the other execution paths do. The silent wrap could also produce wrong `ORDER BY`/`DISTINCT` results, because the read-in-order optimization correctly treats `intDiv` by a negative constant as monotonic, and the wrapped value violated the resulting sort order. [#113048](https://github.com/ClickHouse/ClickHouse/pull/113048) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `UNKNOWN_IDENTIFIER` with `enable_analyzer = 0` when a `WHERE` refers to a column produced by `untuple` inside a subquery, for example `SELECT t.keys FROM (SELECT untuple(arrayJoin(m)) AS t FROM ...) WHERE t.values > 0`. The `enable_optimize_predicate_expression` rewrite pushed such a predicate into the subquery as a `HAVING`, where the name does not yet exist. [#113760](https://github.com/ClickHouse/ClickHouse/pull/113760) ([groeneai](https://github.com/groeneai)). * Fix wrong results in `JOIN` with `join_algorithm = 'partial_merge'` when the right side carries totals: the right side could be left unmerged, so `LEFT` joins substituted default values for real matches and `INNER`/`RIGHT` joins returned no rows at all. In the `RIGHT`/`FULL` non-joined path the same state dereferenced a null pointer, which crashed the server. [#113936](https://github.com/ClickHouse/ClickHouse/pull/113936) ([groeneai](https://github.com/groeneai)). * Fix wrong results for `SELECT DISTINCT` with a `LIMIT` that has to read its whole input: `LIMIT n WITH TIES` dropped tying rows, `exact_rows_before_limit = 1` reported a truncated `rows_before_limit_at_least`, and `GROUP BY ... WITH TOTALS` computed the totals row from a prefix of the stream. [#114025](https://github.com/ClickHouse/ClickHouse/pull/114025) ([groeneai](https://github.com/groeneai)). * Fixed a `LOGICAL_ERROR` ("Unexpected number of kept output positions after removing unused columns from `ReadFromMergeTree`") raised by a `SELECT ... FINAL` whose `PREWHERE` uses a column that `FINAL` needs for merging (`ver`, `is_deleted`, `sign`) and which does not select that column. [#114746](https://github.com/ClickHouse/ClickHouse/pull/114746) ([groeneai](https://github.com/groeneai)). * Fixed a lifetime issue in asynchronous INSERT flushing, materialized-view processing, and `EXPLAIN ANALYZE` that could access expired query-tracking state. [#114940](https://github.com/ClickHouse/ClickHouse/pull/114940) ([azat](https://github.com/azat)). * Fixed `CREATE QUOTA` and `ALTER QUOTA` silently discarding all but the last `FOR INTERVAL` block when the separating comma between intervals was omitted. Such a statement succeeded while installing none of the earlier limits. The intervals now accumulate again, as they did before 2020. [#115320](https://github.com/ClickHouse/ClickHouse/pull/115320) ([groeneai](https://github.com/groeneai)). * Fixed `randConstant` returning two different values in one query when its calls carry different aliases. `randConstant` is documented to hold a single value for the whole query, and `SELECT randConstant() = randConstant()` already returned `1`, but adding aliases as in `SELECT randConstant() AS a, randConstant() AS b` produced two unrelated values. [#115384](https://github.com/ClickHouse/ClickHouse/pull/115384) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `SELECT count()` never terminating, or returning an invented row count, on a corrupted `MsgPack` or `ProtobufList` file. The `optimize_count_from_files` fast path counted rows without requiring the read position to advance, so a two byte `ProtobufList` file produced billions of rows and never returned. Such input is now rejected with the same error the ordinary read path gives. [#115884](https://github.com/ClickHouse/ClickHouse/pull/115884) ([groeneai](https://github.com/groeneai)). * Fix `broken_data_files` in `system.distribution_queue` and the `BrokenDistributedFilesToInsert` metric always reporting `0`: when scanning the async-insert queue at startup, `DistributedAsyncInsertDirectoryQueue::initializeFilesFromDisk` counted broken bytes but never incremented the broken-files counter. [#92124](https://github.com/ClickHouse/ClickHouse/pull/92124) ([squalfof](https://github.com/squalfof)).

Other bug fixes

* Secondary queries executed as part of internal queries will now be logged as internal queries. [#108506](https://github.com/ClickHouse/ClickHouse/pull/108506) ([mstetsyuk](https://github.com/mstetsyuk)). * Fix a DEFAULT column missing in a part being filled with the type default instead of the DEFAULT expression when the PREWHERE condition was split into multiple read steps. [#111795](https://github.com/ClickHouse/ClickHouse/pull/111795) ([yariks5s](https://github.com/yariks5s)). * Quotas keyed by `normalized_query_hash` now account all resources (`read_rows`, `read_bytes`, `result_rows`, `result_bytes`, `execution_time`, `written_bytes`, `errors`) per query pattern, like the query-count counters, instead of accounting them against a single shared per-user bucket. [#107681](https://github.com/ClickHouse/ClickHouse/pull/107681) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed the MySQL `connect_timeout`, read and write timeouts not taking effect: a connection attempt to an unresponsive MySQL server could hang for more than two minutes even with `connect_timeout = 1`, because the sampling query profiler's periodic signals kept resetting the MySQL client's internal poll deadline. [#109592](https://github.com/ClickHouse/ClickHouse/pull/109592) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed the block-wait timeout of streaming inserts (`input_format_max_block_wait_ms`) potentially never expiring while the query profiler is active: the file-descriptor poll restarted with the full timeout after every profiler signal, so a stalled input source could delay the partial-block flush indefinitely instead of flushing when the wait limit is reached. [#109602](https://github.com/ClickHouse/ClickHouse/pull/109602) ([tiandiwonder](https://github.com/tiandiwonder)). * Fixed a `LOGICAL_ERROR` (`std::length_error`) when a query with the experimental setting `make_distributed_plan = 1` used a very large `distributed_plan_default_reader_bucket_count` or `distributed_plan_default_shuffle_join_bucket_count`. Such values are now rejected with `INVALID_SETTING_VALUE`. [#109770](https://github.com/ClickHouse/ClickHouse/pull/109770) ([groeneai](https://github.com/groeneai)). * Fix `distinct_overflow_mode = 'break'`: DISTINCT now returns the partial result accumulated up to the limit and stops reading the source, as documented. Previously the chunk that crossed `max_rows_in_distinct` / `max_bytes_in_distinct` was discarded (truncated or empty results) and the query kept reading and inserting into the hash set to the end of the input, which could end in `MEMORY_LIMIT_EXCEEDED`. [#110075](https://github.com/ClickHouse/ClickHouse/pull/110075) ([skuznetsov-clickhouse](https://github.com/skuznetsov-clickhouse)). * Fix `optimize_aggregation_in_order` ignoring query cancellation. `AggregatingInOrderTransform` now checks for cancellation while aggregating a chunk, so a query stopped by `KILL QUERY` or by `max_execution_time` (in the default `timeout_overflow_mode = 'throw'`) stops promptly instead of running the whole chunk to completion. [#110104](https://github.com/ClickHouse/ClickHouse/pull/110104) ([groeneai](https://github.com/groeneai)). * Fixed a server abort (in debug and sanitizer builds) when a value from a `mysql`/`postgresql`/`sqlite`-family PostgreSQL source is read into a declared type it cannot be parsed into (for example a `text` column declared as `Int32`). Such a type mismatch is now reported as a query error instead of aborting the server. [#110264](https://github.com/ClickHouse/ClickHouse/pull/110264) ([groeneai](https://github.com/groeneai)). * Fixed a server abort when parsing certain PRQL queries (`SET dialect = 'prql'`). A panic inside the `prqlc` compiler was aborting the process instead of being reported as a query error. [#110316](https://github.com/ClickHouse/ClickHouse/pull/110316) ([groeneai](https://github.com/groeneai)). * Fix `NOT_FOUND_COLUMN_IN_BLOCK` when a row policy and `additional_table_filters` reference the same column that is not otherwise selected by the query. [#111099](https://github.com/ClickHouse/ClickHouse/pull/111099) ([groeneai](https://github.com/groeneai)). * Fixed a hang in `randChiSquared`, `randStudentT`, `randFisherF` and `randBinomial` with extreme parameters. The query could not be interrupted by `max_execution_time` or `KILL QUERY`. Such parameters are now rejected even when `max_rand_distribution_parameter` or `max_rand_distribution_trials` is set to 0, and `max_rand_distribution_trials` can no longer be raised above its default of 10^9 for `randBinomial`. [#111909](https://github.com/ClickHouse/ClickHouse/pull/111909) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixes `connection_pool_max_wait_ms` so that its documented and default value `0` means an infinite timeout. A query that found the connection pool full used to retry in a tight loop, consuming CPU and logging `No free connections in pool. Waiting 0 ms.` on every iteration. It now waits until a connection is returned to the pool. [#112380](https://github.com/ClickHouse/ClickHouse/pull/112380) ([groeneai](https://github.com/groeneai)). * Fixes `replaceOne`, `replaceAll`, `replaceRegexpOne` and `replaceRegexpAll` ignoring `max_execution_time` and `KILL QUERY`. A single call now checks for cancellation while it works, so a long-running replacement over a large value or a large number of rows stops instead of holding a server thread until it finishes. Such a call has no partial result to return, so under `timeout_overflow_mode = 'break'` it raises the deadline as an error: always when constant-folded, and in the pipeline whenever this checkpoint is reached before the executor's between-block check, which still stops the query silently. [#112483](https://github.com/ClickHouse/ClickHouse/pull/112483) ([groeneai](https://github.com/groeneai)). * Fixed `countSubstrings`, `countSubstringsCaseInsensitive` and `countSubstringsCaseInsensitiveUTF8` ignoring `KILL QUERY` and `max_execution_time`: a call over many rows or many matches could run for minutes after the query was cancelled. [#113369](https://github.com/ClickHouse/ClickHouse/pull/113369) ([groeneai](https://github.com/groeneai)). * Fix possible query hanging (for the `receive_timeout` seconds) in case it very quickly finishes on the initiator. [#113551](https://github.com/ClickHouse/ClickHouse/pull/113551) ([nickitat](https://github.com/nickitat)). * Fixes `countMatches` and `countMatchesCaseInsensitive` ignoring `max_execution_time` and `KILL QUERY` while counting the matches of one large value, which kept a cancelled query running for minutes. With a constant argument, the call happens during query analysis, so no pipeline exists to cancel it. [#113886](https://github.com/ClickHouse/ClickHouse/pull/113886) ([groeneai](https://github.com/groeneai)). * A query could be cancelled up to 1 ms before its `max_execution_time` had passed, failing with a self-contradictory `Timeout exceeded: elapsed 999.672 ms, maximum: 1000 ms`. [#114559](https://github.com/ClickHouse/ClickHouse/pull/114559) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed an out-of-bounds read in `pointInPolygon` with a constant polygon whose bounding box is unbounded, for example when a coordinate reaches `+-DBL_MAX`. Such a polygon is now rejected with `BAD_ARGUMENTS`, and a polygon with an empty ring is treated as having no interior. Previously the query could return a wrong result, or abort the server in builds with hardening or sanitizers enabled. [#115035](https://github.com/ClickHouse/ClickHouse/pull/115035) ([groeneai](https://github.com/groeneai)). * Fix a race that could make a `STREAM BOUNDED` query return fewer rows than were committed before it started, usually none. [#115543](https://github.com/ClickHouse/ClickHouse/pull/115543) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `Unknown identifier` raised during query plan optimization under `make_distributed_plan = 1` when a window query's sort column passes through an `Expression` step that does not mention it. [#115890](https://github.com/ClickHouse/ClickHouse/pull/115890) ([groeneai](https://github.com/groeneai)). * Fixed `h3kRing`, `h3HexRing`, `h3Line` and `h3ToChildren` ignoring `max_execution_time` and `KILL QUERY` while expanding a block of rows. [#115893](https://github.com/ClickHouse/ClickHouse/pull/115893) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `NOT_FOUND_COLUMN_IN_BLOCK` in the native Parquet V3 reader (after a previous regression) when a `PREWHERE` had conjuncts sharing a common intermediate expression. [#107059](https://github.com/ClickHouse/ClickHouse/pull/107059) ([groeneai](https://github.com/groeneai)). * Fixes a hang when writing Parquet with `output_format_parquet_parallel_encoding` enabled (the default) and `max_threads` greater than 1. If the encoder failed to schedule an additional thread, its live thread counter underflowed and the write could stop making progress permanently instead of finishing or reporting an error. [#112959](https://github.com/ClickHouse/ClickHouse/pull/112959) ([groeneai](https://github.com/groeneai)). * Fixed unbounded memory allocation when reading a Parquet file whose Thrift metadata declares a container or string length larger than the metadata itself. Such a length is now rejected instead of being allocated for. [#113212](https://github.com/ClickHouse/ClickHouse/pull/113212) ([groeneai](https://github.com/groeneai)). * Fixes a `std::out_of_range` exception (`Logical error: 'std::exception. Code: 1001, type: std::out_of_range'`, which also aborts the server in debug and sanitizer builds) in the Parquet v3 reader when a column needed only for filter evaluation was scheduled for decoding after its output slot had already been dropped. [#113956](https://github.com/ClickHouse/ClickHouse/pull/113956) ([groeneai](https://github.com/groeneai)). * Fix data races in profile events and memory tracker. [#112466](https://github.com/ClickHouse/ClickHouse/pull/112466) ([mstetsyuk](https://github.com/mstetsyuk)). * Fix a theoretical data race in profile events when using the `trace_profile_events_list` setting to write stack traces of certain profile events to `system.trace_log`. [#113553](https://github.com/ClickHouse/ClickHouse/pull/113553) ([mstetsyuk](https://github.com/mstetsyuk)). * Fixed `intDivOrNull`, `moduloOrNull` and `positiveModuloOrNull` returning `0` instead of `NULL`, and `intDivOrNull` and `intDivOrZero` raising an exception instead of returning `NULL` / `0`, when the division leads to a floating-point exception (division by zero or `INT_MIN / -1`), including for mixed signed/unsigned arguments. [#101976](https://github.com/ClickHouse/ClickHouse/pull/101976) ([yariks5s](https://github.com/yariks5s)). * Fixed `IS DISTINCT FROM` for `Array` and `Map` values compared with `NULL`. [#103162](https://github.com/ClickHouse/ClickHouse/pull/103162) ([ylw510](https://github.com/ylw510)). * Fixed `groupConcat` when the parametric and two-argument spellings are mixed, for example `groupConcat(',', 2)(x, '/')`: the row-limit parameter was silently dropped, so every row was returned instead of the requested number; the delimiter from the second argument now correctly overrides the parameter. [#104882](https://github.com/ClickHouse/ClickHouse/pull/104882) ([yariks5s](https://github.com/yariks5s)). * Fixes bugs when using CAST from smaller to larger interval units. [#105058](https://github.com/ClickHouse/ClickHouse/pull/105058) ([yariks5s](https://github.com/yariks5s)). * Fixed a logical error (`Assertion 'row < chunk.getNumRows()' failed`) in `LIMIT ... WITH TIES` queries running with the read-in-order pipeline (`optimize_read_in_order = 1` and `read_in_order_use_virtual_row_per_block = 1`). [#105102](https://github.com/ClickHouse/ClickHouse/pull/105102) ([groeneai](https://github.com/groeneai)). * Fix `Logical error: 'Pipeline stuck'` in queries that use `enable_sharding_aggregator = 1` together with a `UNION ALL` and `max_streams_for_union_step` smaller than the pipeline width. `BufferedShardByHashTransform` now finishes empty-queue output ports as soon as its input is exhausted, and keeps pulling input when a demanded empty port has nothing drainable even if a sibling queue hit the back-pressure cap. [#106251](https://github.com/ClickHouse/ClickHouse/pull/106251) ([groeneai](https://github.com/groeneai)). * Fixed `removeDirectory` on `plain` object-storage disks (such as `s3_plain`) removing all files inside a non-empty directory as if the removal were recursive. Removing a non-empty directory now fails with `CANNOT_RMDIR`, and recursive removal handles the contents explicitly. [#106281](https://github.com/ClickHouse/ClickHouse/pull/106281) ([RinChanNOWWW](https://github.com/RinChanNOWWW)). * Fixed `executable_pool` user-defined functions configured with `` not picking up changes to the underlying script. Previously, only `SYSTEM RELOAD FUNCTIONS` would re-read an edited script; periodic `` reloads kept executing the old version. [#107087](https://github.com/ClickHouse/ClickHouse/pull/107087) ([groeneai](https://github.com/groeneai)). * Fixed a data race on the server-global trace collector between a worker thread starting its profiler and server shutdown. [#107307](https://github.com/ClickHouse/ClickHouse/pull/107307) ([groeneai](https://github.com/groeneai)). * Fixed a rare server abort during shutdown (`std::future_error`, "The associated promise has been destructed prior to the associated state becoming ready") caused by a task scheduled through `threadPoolCallbackRunnerUnsafe` being dropped from the thread pool queue before it ran. [#107383](https://github.com/ClickHouse/ClickHouse/pull/107383) ([groeneai](https://github.com/groeneai)). * Fixed a `FileLog` logical error (`Last stored last_written_position in meta file ... is bigger than current last_written_pos`) that could happen when a watched file was deleted and recreated reusing the same inode. [#107617](https://github.com/ClickHouse/ClickHouse/pull/107617) ([groeneai](https://github.com/groeneai)). * Fixed `toTime64` and `CAST(... AS Time64)` not clamping out-of-range values to the `Time64` range in `saturate` and `ignore` overflow modes, which could produce values that display identically but compare as different. [#108028](https://github.com/ClickHouse/ClickHouse/pull/108028) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed incorrect handling of zero-width assertions (`\b`, `^`, `$`) in the regexp "match all" functions `extractAll`, `extractAllGroupsVertical`, `extractAllGroupsHorizontal`, `countMatches` and `splitByRegexp`. For example, `extractAll('new york is the greatest', '\b(\w)')` now correctly returns the first letter of each word instead of every letter. [#108047](https://github.com/ClickHouse/ClickHouse/pull/108047) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix incorrect results when using `hasToken` on text indexes with non-`splitByNonAlpha` tokenizer. [#108066](https://github.com/ClickHouse/ClickHouse/pull/108066) ([rschu1ze](https://github.com/rschu1ze)). * Fix `toStartOfInterval` and `dateTrunc` returning a rounded value instead of the start of the containing interval when the input has finer precision than the interval unit, e.g. `toStartOfInterval(toDateTime64('2023-10-09 10:11:12.000999', 6), INTERVAL 1 millisecond)` returned `10:11:12.001` instead of `10:11:12.000`. The overload with an explicit origin was affected too. [#108186](https://github.com/ClickHouse/ClickHouse/pull/108186) ([yariks5s](https://github.com/yariks5s)). * Fixed `replaceRegexpOne` and `replaceRegexpAll` so `.` matches newline characters by default, consistently with other regular expression functions. [#108265](https://github.com/ClickHouse/ClickHouse/pull/108265) ([linjiayu1025-collab](https://github.com/linjiayu1025-collab)). * Fixes lightweight UPDATE/DELETE conditions being evaluated twice. [#108323](https://github.com/ClickHouse/ClickHouse/pull/108323) ([ofeliacode](https://github.com/ofeliacode)). * Fix `arrayPartialSort()` with a non-constant limit argument. [#108327](https://github.com/ClickHouse/ClickHouse/pull/108327) ([vitlibar](https://github.com/vitlibar)). * Fixed regular expression functions (`match`, `extract`, `extractAll`, `replaceRegexpOne`, `replaceRegexpAll`, etc.) silently returning wrong results when the pattern contained a NUL (`\0`) byte. The NUL is now treated as an ordinary literal byte, consistent with RE2. [#108427](https://github.com/ClickHouse/ClickHouse/pull/108427) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `changeYear`/`changeMonth`/`changeDay`/`changeHour`/`changeMinute`/`changeSecond` over a `DateTime64(N)` argument creating the result column with the hardcoded default scale (3) instead of `N`. The values were computed at scale `N` but the column object declared scale 3, leaving it structurally inconsistent with its type. [#108551](https://github.com/ClickHouse/ClickHouse/pull/108551) ([groeneai](https://github.com/groeneai)). * Fix a `LOGICAL_ERROR` ("Trying to extract chunk from ChunkBuffer before all inputs are finished") when a set operation (`INTERSECT` / `UNION ALL` / `EXCEPT`) combines branches that use correlated subqueries and `correlated_subqueries_default_join_kind = 'left'`. [#108554](https://github.com/ClickHouse/ClickHouse/pull/108554) ([groeneai](https://github.com/groeneai)). * Fixed `DateTime64` bugs in `changeYear`, `changeMonth`, `changeDay`, `changeHour`, `changeMinute` and `changeSecond`: nanosecond-precision (scale 9) inputs no longer throw `DECIMAL_OVERFLOW`, and pre-epoch sub-second inputs now return the correct calendar second. [#108681](https://github.com/ClickHouse/ClickHouse/pull/108681) ([takumihara](https://github.com/takumihara)). * Avoid excessive server log output and an oversized error message when compiling a very large regular expression (for example a `LIKE` or `match` pattern with hundreds of thousands of wildcards); such patterns now fail with a clear `CANNOT_COMPILE_REGEXP` error. [#108821](https://github.com/ClickHouse/ClickHouse/pull/108821) ([Algunenano](https://github.com/Algunenano)). * Fix `reinterpret(x, 'Decimal128(scale)')` (and `Decimal32`/`Decimal64`/`Decimal256`/`DateTime64` targets) producing a result column whose internal scale was the source scale instead of the requested target scale when the source and target had the same physical type. The values were correct but the column object was structurally inconsistent with its declared type. [#108878](https://github.com/ClickHouse/ClickHouse/pull/108878) ([groeneai](https://github.com/groeneai)). * Fixed undefined behaviour when stringifying an out-of-range protocol packet type (e.g. in the "Unexpected packet from server" / "Received ... packet" error messages) for a desynced or fuzzed connection. [#108885](https://github.com/ClickHouse/ClickHouse/pull/108885) ([groeneai](https://github.com/groeneai)). * Fix `changeYear`/`changeMonth`/`changeDay`/`changeHour`/`changeMinute`/`changeSecond` over `DateTime64(N)` building the result column at the hardcoded default scale 3 instead of `N`, and `timeSlots` over `DateTime64` ignoring the optional `Size` argument scale in its declared return type. Both produced a column whose physical scale diverged from its declared type, which led to a `LOGICAL_ERROR` (`writeSlice expects same column types`) when the result was passed to `arrayPushBack`/`arrayPushFront`/`arrayConcat`, and `timeSlots` with the largest scale in `Size` returned wrong timestamps. [#108994](https://github.com/ClickHouse/ClickHouse/pull/108994) ([groeneai](https://github.com/groeneai)). * Fix non-monotonic ProfileEvents increments [#109180](https://github.com/ClickHouse/ClickHouse/pull/109180) ([azat](https://github.com/azat)). * Fix `SAMPLE` ratios with an exponent whose magnitude overflows `Int32` (e.g. `SAMPLE 1e-3000000000`) being silently treated as `SAMPLE 1`. [#109197](https://github.com/ClickHouse/ClickHouse/pull/109197) ([Algunenano](https://github.com/Algunenano)). * Fixed two bugs on the `DateTime64` path of `toUTCTimestamp`/`fromUTCTimestamp` (and their `to_utc_timestamp`/`from_utc_timestamp` aliases) and `toTime64`. First, a signed integer overflow near the `Int64` boundary (now computed in `Int128` and clamped to the representable range). Second, a wrong result for negative fractional values near a timezone-offset boundary: the seconds split truncated toward zero, so `timezoneOffset()` inspected the next second and could pick the wrong side of a DST/offset change. [#109738](https://github.com/ClickHouse/ClickHouse/pull/109738) ([groeneai](https://github.com/groeneai)). * Fixed a logical error (`column->size() == num_rows`) when a row policy filter expression uses `arrayJoin`; such filters are now rejected. [#109753](https://github.com/ClickHouse/ClickHouse/pull/109753) ([Algunenano](https://github.com/Algunenano)). * Fixed `generateRandomStructure()` occasionally producing an invalid structure string with two data types concatenated (for example `Decimal32(7)IPv4`) when type nesting exceeded the internal depth limit, which made the result unparseable. [#109928](https://github.com/ClickHouse/ClickHouse/pull/109928) ([groeneai](https://github.com/groeneai)). * Fixed query-setting propagation in `accurateCastOrDefault` and preservation of source `NULL`s encoded by `Dynamic` and `Variant`. [#109946](https://github.com/ClickHouse/ClickHouse/pull/109946) ([Avogar](https://github.com/Avogar)). [#114912](https://github.com/ClickHouse/ClickHouse/pull/114912) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix `max_bytes_in_distinct` and `max_bytes_in_set`: string keys are stored in an arena that was not counted by the limit checks, so DISTINCT / IN over string keys could hold memory exceeding the byte limit by the whole key payload (unbounded in the key length). The limits now account for the arena, matching their documentation; queries with string keys close to a byte limit may now trip it earlier (correctly). [#110120](https://github.com/ClickHouse/ClickHouse/pull/110120) ([skuznetsov-clickhouse](https://github.com/skuznetsov-clickhouse)). * Fixed reading Npy files with zero-sized inner dimensions so materialized rows and optimized and non-optimized `count()` results agree. [#110146](https://github.com/ClickHouse/ClickHouse/pull/110146) ([qiuyanjun888](https://github.com/qiuyanjun888)). * Fix a logical error (server abort in debug/sanitizer builds) when the `indexHint`/`ignore`/`isZeroOrNull` functions are given an argument whose type resolves to `Nothing`, e.g. inside expressions like `indexHint(assumeNotNull(materialize(NULL)))`. [#110192](https://github.com/ClickHouse/ClickHouse/pull/110192) ([groeneai](https://github.com/groeneai)). * Fix reading files on `plain_rewritable` disks after an existing path was rewritten with content of a different size: the in-memory metadata kept the stale file size, which could fail reads with `UNEXPECTED_END_OF_FILE`. [#110304](https://github.com/ClickHouse/ClickHouse/pull/110304) ([thevar1able](https://github.com/thevar1able)). * Rejects `vector_search_index_fetch_multiplier` values below 1.0 to prevent empty ANN search results when the computed fetch count truncates to zero. [#110452](https://github.com/ClickHouse/ClickHouse/pull/110452) ([tamish560](https://github.com/tamish560)). * Fix a sporadic `Cannot parse string ... as UInt64 ... While executing ValuesBlockInputFormat` error that could occur when inserting values whose expressions contain a numeric cast such as `initializeAggregation('sumState', 0::UInt64)`. [#110637](https://github.com/ClickHouse/ClickHouse/pull/110637) ([groeneai](https://github.com/groeneai)). * Fixed `readWKT` rejecting WKT strings with leading whitespace (e.g. `readWKT(' POINT(1 2)')`), which are accepted by the typed `readWKTPoint`/`readWKTPolygon`/... readers and by the WKT grammar. [#110706](https://github.com/ClickHouse/ClickHouse/pull/110706) ([groeneai](https://github.com/groeneai)). * Fix a server crash (integer division by zero) when a quota with a zero-length interval (for example CREATE QUOTA q FOR INTERVAL 0 SECOND MAX queries = 1000) was consumed. A non-positive quota interval duration is now rejected at quota creation. [#110846](https://github.com/ClickHouse/ClickHouse/pull/110846) ([PedroTadim](https://github.com/PedroTadim)). * Fixed a crash in FIPS builds when using an Ed25519 SSH key (`CREATE USER ... IDENTIFIED WITH ssh_key ... TYPE 'ssh-ed25519'`, or such a key in `users.xml`). Ed25519 is not FIPS-approved; such keys are now rejected with a clear `LIBSSH_ERROR` error. [#110891](https://github.com/ClickHouse/ClickHouse/pull/110891) ([thevar1able](https://github.com/thevar1able)). * The transposed distance functions over `QBit` (`cosineDistanceTransposed`, `L2DistanceTransposed`, `dotProductTransposed`) now use a bounded low-precision reconstruction of a value truncated to `precision` bit planes, instead of reconstructing it to the coarse cell's lower edge (dropped bits zero-filled). Zero-filling biased every reconstructed value towards zero and was degenerate at low precision: for `QBit(BFloat16)` at precision 1 only the sign bit survives, so every value reconstructed to `±0.0` and the distance was the same constant for every row, carrying no ranking information. The reconstruction now depends on which bits are dropped: raw `Int8` codes are reconstructed to their cell centre (matching the `Lloyd-Max` reconstruction the `...TransposedQuantized` path already used); a floating-point value at precision 1 becomes a proper sign quantization; a floating-point value that keeps its whole exponent and drops only mantissa bits is reconstructed to the bounded midpoint within its own binade (while a genuine zero or `±0.0` is left at exact zero and the non-finite cell is carved out so `±inf` stays exactly infinite; see the note below on the ambiguous `NaN` cell); and while exponent bits are still being truncated the bounded lower edge is kept, so a smaller `precision` trades accuracy for speed without blowing a magnitude up by orders of magnitude. Full-precision results are unchanged. [#110911](https://github.com/ClickHouse/ClickHouse/pull/110911) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed an out-of-range read in the native ORC reader during schema inference of a corrupt ORC file whose type tree declares more columns than the stripe footer has column encodings. It is now rejected with `CANNOT_EXTRACT_TABLE_STRUCTURE` instead of aborting the server in debug/sanitizer builds. [#110967](https://github.com/ClickHouse/ClickHouse/pull/110967) ([groeneai](https://github.com/groeneai)). * Each SSH key now counts individually toward the `max_authentication_methods_per_user` limit, so the limit applies consistently regardless of syntax. Previously, multiple keys in a single `ssh_key` method counted as one. [#111181](https://github.com/ClickHouse/ClickHouse/pull/111181) ([TheMC47](https://github.com/TheMC47)). * Iceberg REST catalogs now report stale credentials with the underlying error instead of silently reporting `UNKNOWN_TABLE`. [#111379](https://github.com/ClickHouse/ClickHouse/pull/111379) ([alesapin](https://github.com/alesapin)). * Fix `readWKTPoint`/`readWKT` returning uninitialized (scalar) or previous-row (vectorized) coordinates for a dimension-tagged empty point such as `POINT M EMPTY`. Such inputs are now rejected with `CANNOT_PARSE_TEXT`, consistent with `POINT EMPTY`. [#111517](https://github.com/ClickHouse/ClickHouse/pull/111517) ([groeneai](https://github.com/groeneai)). * Creating `minmax` statistics now emits a warning instead of throwing an exception. [#111785](https://github.com/ClickHouse/ClickHouse/pull/111785) ([hanfei1991](https://github.com/hanfei1991)). * Keep `randBinomial` usable with the degenerate probabilities `0` and `1` for any number of trials. [#112021](https://github.com/ClickHouse/ClickHouse/pull/112021) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed `system.s3_queue_settings` reporting incorrect values for `bucketing_mode`, `partitioning_mode`, `partition_regex`, and `partition_component`. [#112300](https://github.com/ClickHouse/ClickHouse/pull/112300) ([bharatnc](https://github.com/bharatnc)). * Report the correct position for errors in PromQL queries. [#112494](https://github.com/ClickHouse/ClickHouse/pull/112494) ([fallintoplace](https://github.com/fallintoplace)). * `SYSTEM DISABLE FAILPOINT` now rejects a fail point name that does not exist, raising `BAD_ARGUMENTS` like `SYSTEM ENABLE FAILPOINT` already did. Previously a mistyped name reported success while leaving the intended fail point enabled, with no indication that nothing had been disabled. [#112680](https://github.com/ClickHouse/ClickHouse/pull/112680) ([groeneai](https://github.com/groeneai)). * Fixes a wrongly typed read of a `Nested` element subcolumn whose column was dropped and re-added. Selecting such a subcolumn together with its own parent column returned the whole element in the subcolumn's slot while the block still declared the element type, which produced arbitrary values in release builds and a logical error in debug and sanitizer builds. [#112769](https://github.com/ClickHouse/ClickHouse/pull/112769) ([groeneai](https://github.com/groeneai)). * Reject PromQL durations whose units are repeated or not ordered from longest to shortest. [#112954](https://github.com/ClickHouse/ClickHouse/pull/112954) ([fallintoplace](https://github.com/fallintoplace)). * Fixed undefined behavior (null pointer passed to `memcpy`) in `replaceRegexpOne`/`replaceRegexpAll` when the replacement string substitutes a capturing group that did not participate in the match, e.g. `replaceRegexpAll('abc', '(a)|(b)', '\\2')`. [#113012](https://github.com/ClickHouse/ClickHouse/pull/113012) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fix an incompatibility that text indexes created in ClickHouse 26.3 with `unicode_word` tokenizer cannot be loaded in later versions [#113061](https://github.com/ClickHouse/ClickHouse/pull/113061) ([rschu1ze](https://github.com/rschu1ze)). * Reject invalid Unicode surrogate code points in PromQL string escapes. [#113203](https://github.com/ClickHouse/ClickHouse/pull/113203) ([fallintoplace](https://github.com/fallintoplace)). * Reject invalid cross-delimiter escapes in PromQL strings. [#113587](https://github.com/ClickHouse/ClickHouse/pull/113587) ([fallintoplace](https://github.com/fallintoplace)). * Fix `theilsU` over a window frame returning an arbitrary value instead of 0 when the first argument is constant within the frame. [#113691](https://github.com/ClickHouse/ClickHouse/pull/113691) ([alexey-milovidov](https://github.com/alexey-milovidov)). * Fixed the error code reported for some malformed `Replicated`-serialized columns received over the Native protocol: these are now rejected with `INCORRECT_DATA` instead of `LOGICAL_ERROR` (which aborted the server in debug and sanitizer builds). [#114186](https://github.com/ClickHouse/ClickHouse/pull/114186) ([Avogar](https://github.com/Avogar)). * Fixes incorrect, often sign-flipped, `Time64` literals and `IN`-list constants produced when rescaling a lower-scale `Decimal64` overflowed `Int64`. Such a conversion now reports `DECIMAL_OVERFLOW`, matching the `DateTime64` branch and explicit `CAST`. [#114546](https://github.com/ClickHouse/ClickHouse/pull/114546) ([groeneai](https://github.com/groeneai)). * Support quoted metric and label names in PromQL selectors. [#114551](https://github.com/ClickHouse/ClickHouse/pull/114551) ([fallintoplace](https://github.com/fallintoplace)). * Fixed `ReplicatedMergeTree` tables remaining read-only indefinitely after a failed startup attempt that ran through the attach thread. The restart retry is now scheduled correctly, restoring automatic recovery and its session checks. [#114802](https://github.com/ClickHouse/ClickHouse/pull/114802) ([azat](https://github.com/azat)). * `timeSeriesRange` and `timeSeriesFromGrid` threw a false `DECIMAL_OVERFLOW` for timestamps before 1970 because the start/end timestamps were read as unsigned values. Also, for `DateTime`/`UInt32` timestamps the step was silently truncated to its lower 32 bits. Now all the calculations are done in `Int64`. [#114815](https://github.com/ClickHouse/ClickHouse/pull/114815) ([vitlibar](https://github.com/vitlibar)). * Fixed a possible exception in asynchronous inserts after a query fails with `MEMORY_LIMIT_EXCEEDED`, and fixed `async_insert_queue_flush_on_shutdown` so queued inserts are flushed during server shutdown. [#114839](https://github.com/ClickHouse/ClickHouse/pull/114839) ([azat](https://github.com/azat)). * Fixed an `out_of_range` exception raised as an internal `Code: 1001` error when a `sequenceMatch`, `sequenceCount` or `sequenceMatchEvents` pattern contains the event number `0`, for example `sequenceMatch('(?0)')`. Such a pattern is now rejected with `BAD_ARGUMENTS`. A temporal condition holding a lone sign, such as `sequenceMatch('(?1)(?t>+)(?2)')`, was silently treated as `(?t>0)` and is now rejected with `SYNTAX_ERROR`. [#115056](https://github.com/ClickHouse/ClickHouse/pull/115056) ([groeneai](https://github.com/groeneai)). * Fixed the error message of `geoToH3` naming the wrong argument position and the wrong type when a coordinate is not `Float64`. Under the default `geotoh3_argument_order = 'lat_lon'` the message pointed at the other argument and printed that argument's type. [#115503](https://github.com/ClickHouse/ClickHouse/pull/115503) ([groeneai](https://github.com/groeneai)). * Fix invalid backslash escape characters in text, tokenbf\_v1 and ngrambf\_v1 indexes. [#115634](https://github.com/ClickHouse/ClickHouse/pull/115634) ([ahmadov](https://github.com/ahmadov)). * Fix an issue: the server couldn't start when the `TZ` environment variable is empty. [#68921](https://github.com/ClickHouse/ClickHouse/pull/68921) ([ardenwick](https://github.com/ardenwick)).