The module lifecycle stage: Generally available version

The module has requirements for installation

Release notes for the Prom++ module. Each entry follows a unified structure: a release date, an optional short summary and highlights, and sections ordered as Highlights → New features → Improvements → Fixes → Security updates → Breaking changes → Upgrade notes → Known issues → Docs → Dependencies. Empty sections are omitted.

v3.8.19

Release date: 2026-10-01

Summary: lower memory use and faster writes around head rotation, a crash fix for malformed native histograms coming from a scrape target, and a new way to tell that an instance runs with feature flags other than the expected ones.

New features

  • A new optional PROMPP_FEATURES_DEFAULT environment variable holds the feature set you expect an instance to run with. At startup it is compared with PROMPP_FEATURES and the result is published as the prompp_features_differ_from_default metric — 1 when they differ, 0 when they match — so you can alert on configuration drift. The difference is also written to the log. The comparison ignores order and spacing and treats equivalent values as equal, so 60m and 1h do not count as a difference.

Improvements

  • Memory use after head rotation is lower: a rotated head now releases the lookup structures it only needed while taking data, instead of holding them until retention.
  • Writes are no longer slowed down right after a rotation. The expensive preparation of the new head’s first WAL segment now happens during the rotation itself, before the head starts taking data. The rotation takes longer as a result, while the previous head keeps accepting writes.
  • Data encoding became about 10% faster.
  • Head identifiers are now UUIDv7, so head directories and catalog records sort in creation order — handy when looking through the data directory.

Fixes

  • Fixed a crash caused by a scrape target returning a malformed native histogram over protobuf. Such a histogram is now rejected by validation instead of taking the process down.
  • Fixed a latent fault that would have emptied the head catalog on a future head.log format migration, losing the active head, the heads not yet written to disk and the remote write positions.
  • Added mount points for prometheus-main-rulefiles-5 through -99, so installations whose alerting and recording rules do not fit into five ConfigMaps get all of them mounted.

Security updates

  • Updated moment in the web UI to 2.31.0, closing GHSA-4p3w-j4w9-5jqw (path traversal through a crafted locale name).
  • Updated the base image set from v3.0.3 to v3.1.6.
  • Fixed the VEX attestation stage being reused across registries.

Breaking changes

  • The prompp_data_storage_gorilla_count metric is removed together with the combined encoding path it counted. Remove it from dashboards and alerts.

v3.8.18

Release date: 2026-09-20

Summary: fixed a memory leak with jemalloc arenas enabled and a case where live series could disappear from the index; Go moved to 1.26.8 and several dependencies were updated, closing a number of known vulnerabilities.

New features

  • Series without samples are now rejected while the remote-write request is parsed, instead of at the more expensive index-write stage. Set PROMPP_FEATURES=enable_skip_no_samples_series to skip such series silently if your client sends them on purpose.
  • The experimental WAL V2 segment format can now be switched on with PROMPP_FEATURES=enable_wal_writer_v2. It is off by default.

Fixes

  • Fixed a memory leak: with jemalloc arenas enabled, the memory of finalized chunks was never returned to the system.
  • Fixed series that were still receiving data over remote write being treated as inactive and pruned from the index, so queries stopped finding them.
  • Fixed the head’s internal bookkeeping going out of sync with the label-set index when relabeling ended with an error.

Security updates

  • The Go toolchain the module is built with moves from 1.26.5 to 1.26.8, closing seven standard-library advisories (net/url, html/template, crypto/tls, net/http, encoding/xml, encoding/asn1 and DNS record parsing).
  • Updated sanitize-html in the web UI, fixing a stored XSS (GHSA-g8qq-57p8-ggw5).
  • Updated gRPC to 1.83.2 (a request missing the :authority and Host headers could crash a gRPC server) and all OpenTelemetry modules to 1.45.0.

v3.8.17

Release date: 2026-09-08

Summary: fixed a chunk querier bug that could crash the process, a remote-write config reload issue that could miss a rotated secret, and a shutdown race that could also crash the process; plus a grpc/crypto security update.

Fixes

  • Fixed a chunk querier bug that could crash the process when running out of chunks to recode instead of correctly stopping the scan.
  • Fixed remote-write config reload not detecting a changed secret (for example a rotated authorization token), which left the destination sending with the old, invalid credential.
  • Fixed a race between shutdown and a head-rotation trigger that could panic the process.
  • The /debug/jemalloc profiling endpoint now serializes concurrent requests and returns 404 when jemalloc profiling isn’t actually enabled, instead of failing unpredictably.

Security updates

  • Updated google.golang.org/grpc and golang.org/x/crypto with upstream security fixes.

Breaking changes

  • The experimental pp_protocol remote-write WebSocket and refill endpoints are temporarily disabled (return 404) while the protocol is reworked.

v3.8.16

Release date: 2026-09-01

Summary: disabled the shrink_shard_copier feature flag by default after it caused metrics received via Remote Write to occasionally disappear.

Fixes

  • Disabled the shrink_shard_copier feature by default: it could cause metrics received via Remote Write to occasionally disappear.

v3.8.15

Release date: 2026-08-27

Summary: MADV_RANDOM for block file mappings is now opt-in to avoid extra disk IOPS on some setups, and the scraper now validates UTF-8 across the whole scrape payload by default.

Fixes

  • Disabled MADV_RANDOM for block index/chunk file memory mappings by default; on some storage backends it could increase disk IOPS. Re-enable it with the enable_madvise_random feature flag.

Improvements

  • The scraper now validates UTF-8 across the entire scraped payload by default, catching invalid characters anywhere in it instead of only in specific fields as before. The old behavior can be restored with the disable_scraper_full_utf8 feature flag.

v3.8.14

Release date: 2026-08-21

Summary: fixed crashes when converting a head into a historical block, removed a possible state error when processing out-of-order samples that could lead to a SIGSEGV, plus dependency security updates.

Fixes

  • Fixed an issue when converting a head into a historical block: under high churn, an out-of-bounds array access could occur, causing a crash.
  • Fixed a possible state error when processing out-of-order samples.

Security updates

  • Updated the Go module golang.org/x/mod and the web UI package nanoid with upstream security fixes.

v3.8.13

Release date: 2026-08-18

Summary: a module-only release that restores the weight field in the module definition, required to install the module on older DKP versions.

Fixes

  • Restored the weight field in module.yaml, dropped in v3.7.10: without it the module cannot be installed on older DKP versions.

v3.8.12

Release date: 2026-08-14

Summary: blocks can now be grouped by segment before compacting in the block-manager storage scheme, remote-write retries are now traceable, and a new flag lets you force remote write onto HTTP/1.1.

Improvements

  • Reworked the block-manager storage scheme’s compactor: blocks can now be grouped by time range and segment before compaction planning, so blocks from different segments are never combined.
  • Remote-write retries now carry a stable idempotency key and the real retry attempt number; failed-send logs report the destination, attempt, duration and message size.
  • Lower page-cache footprint for block files: their memory mappings no longer trigger the kernel’s sequential readahead.

Fixes

  • Fixed a potential race condition in the series index’s internal reference counting.

v3.8.11

Release date: 2026-08-07

Summary: scrapes over 4 GiB are no longer dropped, lower query memory, faster scrape parsing, and better behaviour with a large number of rule groups.

Highlights

  • Scrapes larger than 4 GiB are no longer dropped in full: neither metrics nor metadata were stored before.
  • Better performance with a large number of rule groups (1000+), where the memory allocator itself used to slow down.

Improvements

  • Lower memory use for queries with many label matchers.
  • Faster scrape parsing.
  • Short-lived storages no longer create memory arenas that brought them no benefit and only slowed the allocator down.
  • C++ memory is now reclaimed earlier under growing load.

Fixes

  • Fixed scraping of very large responses: markup offsets are now 64-bit, so a /federate response over 4 GiB is parsed and appended correctly instead of being lost entirely.

Security updates

  • Updated the web UI packages postcss and sanitize-html with upstream security fixes.

Docs

  • Fixed broken links in the module documentation and added the configuration reference pages.

v3.8.10

Release date: 2026-07-31

Summary: lower query-path memory usage, crash fixes for the active query log and DataStorage metrics path, and faster WAL segment reads.

Highlights

  • Query results no longer keep large off-heap buffers alive until GC, reducing memory growth under query load.
  • Fixed SIGBUS crashes related to the active query log on some filesystems.

Improvements

  • Lower memory usage when executing queries by releasing query result buffers as soon as the response is built.
  • Faster WAL segment reads through pooled buffer reuse.

Fixes

  • Fixed SIGBUS in ActiveQueryTracker when the active query log was created as a sparse file.
  • Fixed a race condition during static object initialization that could cause startup errors on older processors.

Security updates

  • Updated Go dependencies (github.com/klauspost/compress, go.opentelemetry.io/otel) and the web UI postcss package with upstream security fixes.

v3.8.9

Release date: 2026-07-24

Summary: correct CPU feature detection for better out-of-the-box performance, faster scrape parsing, lower ingestion memory, and stability and security fixes.

Highlights

  • Fixed CPU feature detection so the engine selects the right instruction set on modern CPUs instead of always falling back to the baseline, improving performance out of the box.
  • Faster scrape parsing.

Improvements

  • Faster scrape parsing through a reworked scraper write-ahead-log encoding.
  • Lower peak memory during ingestion: the transaction snapshot is now released as soon as the transaction is committed.
  • The module image now uses a statically-linked curl, reducing runtime dependencies.
  • Module release notes are now shipped with the release and shown in the Deckhouse UI (unified changelog format).

Fixes

  • Fixed CPU feature detection that always selected baseline instruction bindings even on modern CPUs, degrading performance.

Security updates

  • Updated Go dependencies (golang.org/x/text, golang.org/x/net, google.golang.org/grpc) and web UI packages (body-parser, webpack-dev-server, immutable) with upstream security fixes.

v3.8.8

Release date: 2026-07-17

Summary: incident-recovery tooling and observability improvements, lower memory usage when writing blocks, and stability and security fixes.

Highlights

  • Added a persist-head command to prompptool for recovering an individual head to TSDB blocks during incident handling.
  • Reduced memory consumption when writing blocks over wide or sparse time ranges.
  • Added on-demand jemalloc heap profiling over HTTP for troubleshooting memory usage.

Improvements

  • The block writer now skips blocks that already fall entirely outside the retention period, avoiding wasted disk writes.
  • Added a /debug/jemalloc HTTP endpoint to capture a jemalloc heap profile on demand (requires profiling enabled at startup).
  • Added a metric for the total size of unknown objects in local storage, making disk leftovers visible.
  • Reduced memory consumption when writing blocks over wide or sparse time ranges.

Fixes

  • Fixed a rare crash (heap-buffer-overflow) in the outdated chunk merger.
  • Leftover temporary block directories from an interrupted compaction or persist are now cleaned up on startup.

Security updates

  • Updated the Go toolchain, addressing known security vulnerabilities.

v3.8.7

Release date: 2026-07-16

Fixes

  • Removed internal C++ metrics registration because it led to a memory leak.

v3.8.6

Release date: 2026-07-15

Fixes

  • Disabled the internal C++ metrics collector by default to prevent a rare crash (use-after-free) that could occur during metrics collection.

v3.8.5

Release date: 2026-07-09

Improvements

  • Reworked historical block storage with a new block-manager engine, improving compaction planning and adding block layout diagnostics metrics.
  • Core dumps are no longer written on crash, preventing unexpected disk usage in constrained environments such as Kubernetes.
  • Added metrics for observing the internal data storage.
  • Improved performance of label processing and index writing.

Security updates

  • Updated the build toolchain and dependencies, addressing known security vulnerabilities.

v3.8.4

Release date: 2026-06-11

Improvements

  • Improved instant query performance.
  • Reduced memory consumption during head rotation and when processing high-cardinality metrics through the federation API.

Security updates

  • Updated base images and security-related components.

v3.8.3

Release date: 2026-06-02

Fixes

  • Fixed a chunk encoder issue that could, in rare cases, lead to historical data corruption.
  • Updated PromQL range query behavior to match the current Prometheus implementation. As a result, the output of the rate, sum_over_time, and count_over_time functions may differ slightly from previous versions.

Security updates

  • Fixed a vulnerability that could lead to memory exhaustion when processing remote read queries.
  • Fixed multiple XSS vulnerabilities in the web interface.

v3.8.1

Release date: 2026-05-15

Improvements

  • Improved data processing performance and reduced memory consumption when executing queries in sharded environments.

Fixes

  • Fixed WAL processing during shard rotation.

Security updates

  • Updated dependencies and addressed known security vulnerabilities.

v3.8.0

Release date: 2026-04-22

Improvements

  • Added metrics for analyzing mutex and semaphore contention in the Go runtime.
  • Added a metric for jemalloc resident memory usage.
  • Improved resource cleanup during WAL rotation.

v3.7.11

Release date: 2026-05-15

Improvements

  • Added metrics for monitoring jemalloc arena reuse efficiency.

v3.7.10

Release date: 2026-04-22

Improvements

  • Added support for WAL v2 format in remote write.

Fixes

  • Fixed false positives for the PrometheusRemoteWriteDesiredShards alert.
  • Resolved potential issues affecting remote write under high load.

v3.7.9

Release date: 2026-03-30

New features

  • Added new PromQL functions: op_defined, op_replace_nan, op_smoothie, op_zero_if_none, and op_top.

Improvements

  • Improved remote write performance.
  • Added a data age limit for remote write to prevent WAL file accumulation when external systems are unavailable.

Fixes

  • Fixed ingestion metric calculations.
  • Resolved increasing memory consumption when using custom jemalloc arenas.

Security updates

  • Updated dependencies and addressed known security vulnerabilities.

v3.7.6

Release date: 2026-02-20

Improvements

  • Improved rule evaluation and remote write performance.

Security updates

  • Updated security-related dependencies.

v3.7.5

Release date: 2026-01-23

Improvements

  • Optimized data ingestion and processing workflows.

v3.7.4

Release date: 2026-01-12

Improvements

  • Added support for splitting high-cardinality metrics when using the federation API, reducing peak memory consumption.

v3.7.2

Release date: 2025-12-08

Improvements

  • Improved rule evaluation and query execution performance.

Fixes

  • Fixed stability issues that could cause intermittent failures.
  • Fixed handling of empty labels in OTLP Protobuf.

v3.7.0

Release date: 2025-11-05

Improvements

  • Data copying during rotation is now performed in the background, reducing the impact of rotation on system responsiveness.
  • Improved query execution and rule evaluation performance.

Fixes

  • Fixed handling of StaleNaN values during data rotation.

v3.6.5

Release date: 2025-10-22

Improvements

  • Added a mechanism for offloading infrequently used time series to disk, significantly reducing memory consumption in large-scale deployments.
  • Reduced memory consumption during metrics collection and processing.

Fixes

  • Fixed regular expression handling in matchers.
  • Improved resilience to corrupted WAL files and invalid data.
  • Fixed handling of StaleNaN values for backfilled data.

v3.5.0

Release date: 2025-08-25

Improvements

  • Upgraded the underlying Prometheus version to 2.55.1.
  • Added support for the Prometheus 3.x storage format.

Security updates

  • Updated dependencies and addressed known security vulnerabilities.