A Workload-Adaptive Compaction Policy for LSM-Based Analytics Stores with Read Amplification Constraints

Abstract

Log-structured merge (LSM) trees are widely used in analytics-oriented stores because they support high ingest rates while maintaining reasonable query performance over large datasets. In these systems, background compaction is the primary mechanism that reconciles write-optimized organization with read-efficient layouts, but compaction also consumes substantial I/O, CPU, and cache resources and can interfere with latency-sensitive scan and point-query workloads. A central operational difficulty is that compaction tuning is typically static, while real analytics workloads are non-stationary and may shift between scan-heavy, filter-heavy, and mixed read-write regimes. This paper presents a workload-adaptive compaction policy designed explicitly for LSM-based analytics stores subject to explicit read amplification constraints. The policy continuously estimates workload descriptors and storage-state signals, then solves a constrained optimization that allocates compaction effort across levels, key ranges, and file sets to satisfy a target bound on read amplification while minimizing a weighted cost of write amplification, compaction interference, and tail-latency risk. The approach combines analytic modeling of fanout, overlap, and filter effectiveness with backprop-friendly surrogate objectives so that the controller can be trained or calibrated online. The design integrates approximation-aware error accounting, explicit budget and energy constraints, and distributed execution mechanics that avoid global coordination bottlenecks. The result is a compaction controller that is responsive to workload change, provides interpretable safety margins for read amplification, and can be implemented with modest modifications to common LSM engines while preserving reproducibility of performance evaluation.