Stigmergy-Guided Self-Organizing Workload Allocation for Cost-Efficient Operation of Cloud-Native Enterprise Data Platforms

Abstract

Cloud-native enterprise data platforms increasingly host a heterogeneous mix of transactional, analytical, and machine learning workloads that operate under diverse performance and cost constraints. These platforms run on elastic cloud infrastructure where pricing models vary across instance families, storage tiers, and data transfer paths. As organizations consolidate data processing into shared platforms, workload allocation strategies strongly influence infrastructure expenditure and service-level adherence. Traditional centralized schedulers rely on global state and frequent recomputation of placement decisions, which becomes challenging under rapid workload arrivals, fluctuating prices, and partial observability of resource conditions. Self-organizing approaches offer an alternative in which coordination emerges from the local interactions of simple decision rules. This paper investigates a stigmergy-guided mechanism for workload allocation in cloud-native enterprise data platforms, inspired by indirect coordination processes observed in social systems where agents communicate by modifying their environment. The platform is modeled as a collection of nodes that maintain local cost and load signals analogous to pheromone fields, while workloads act as agents that choose target nodes according to these signals and application-specific heuristics. The resulting algorithm operates without centralized coordination and adapts to price changes and workload variability. The study develops a linear cost model for infrastructure consumption, integrates it with local stigmergic signals, and discusses how the resulting mechanism can be implemented in modern container-based data platforms. The behavior of the approach is examined through a conceptual evaluation focusing on cost efficiency, robustness, and operational considerations.