October 26, 2022 . 3 MIN READ
As Elastic continues to expand its offerings with solutions like Observability, Security, and Search, the Elastic Cloud user base has grown well beyond traditional operations teams. Today, data engineers, security professionals, and consultants also rely heavily on the platform. As an Elastic support engineer, I’ve had the opportunity to work with a wide variety of users and real-world use cases.
With this broader audience, one topic comes up again and again: resource allocation. In particular, users often ask how to troubleshoot allocation health issues and avoid circuit breakers. That’s completely understandable. When I first started working with Elasticsearch, I had the same questions. It was my introduction to JVM heap management, time-series shards, and scaling infrastructure on my own.
When I joined Elastic, I appreciated how quickly I could get up to speed thanks to detailed documentation, blogs, and tutorials. Still, during my first month, I struggled to connect theory with the real error messages coming through my support queue. Over time, like many support engineers, I realized that most reported issues were symptoms of underlying allocation problems. In fact, a familiar set of resources and explanations solved the majority of cases.
In this guide, I’ll walk through:
Core allocation and memory management concepts
The most common symptoms we see in support tickets
Where and how to adjust configurations to resolve these issues
Elasticsearch is a Java application, which means it relies on JVM heap memory allocated from the system’s physical RAM. Best practice is to allocate up to 50% of available RAM to the heap, with a hard cap at 32 GB. Larger heaps are typically required for expensive queries or large data volumes.
The parent circuit breaker is set to 95% by default, but once usage consistently exceeds 85%, it’s usually time to scale resources rather than push the heap further.
Helpful background reading includes:
A Heap of Trouble
Heap Sizing and Swapping
By default, Elasticsearch automatically sizes the heap based on node role and available memory. If needed, you can override this in several ways:
Directly in jvm.options
Set both -Xms (initial heap) and -Xmx (max heap) to the same value.
Using environment variables (e.g., Docker)
Define heap size via ES_JAVA_OPTS.
Elastic Cloud Hosted
Adjust deployment memory in the UI; roughly half is allocated to the heap.
Most performance problems fall into a few categories:
Configuration issues
Undersized master nodes
Missing or misconfigured ILM policies
Load-related issues
High request volume
Overlapping or expensive queries and writes
These issues typically surface through shard allocation problems or memory pressure.
Indices are split into shards, which consume heap during maintenance and query execution. A good rule of thumb is to keep shard sizes under 50 GB.
If cluster health is anything other than green, inspect:
Unassigned shards
Initializing or relocating shards
Unassigned primary shards result in a red cluster state, while unassigned replicas produce a yellow state. Replica shards are critical for resilience and recovery.
Using allocation explain APIs helps pinpoint why shards cannot be assigned—whether due to tier mismatches, node constraints, or temporary failures. In some cases, a cluster reroute can resolve transient issues.
When heap usage approaches its limits, Elasticsearch may trigger circuit breaker exceptions, leading to failed or timed-out requests. Monitoring heap.percent across nodes is the fastest way to identify this problem.
If circuit breakers are triggered:
Temporarily increasing heap can provide short-term relief
Logs should be reviewed for:
Expensive aggregations
Large bucket sizes
Inefficient mappings
High request concurrency
If memory pressure remains high over time—especially if JVM memory pressure stays near or above recommended thresholds—it’s time to scale.
Long-term solutions include:
Adding nodes or increasing node memory
Reducing shard counts
Archiving or deleting old data
Moving data to warm or cold tiers
Disabling replicas for non-critical data
Frequent or long garbage collection events are another strong indicator that scaling or workload reduction is necessary.
From a support perspective, most Elasticsearch issues trace back to the same root causes: shard allocation problems, uneven heap usage, circuit breakers, and garbage collection pressure. These are all symptoms of resource allocation challenges.
With a solid understanding of the underlying theory and the right diagnostic steps, these issues become far easier to identify and resolve. Hopefully, this overview helps you do exactly that.
Reference:
https://www.elastic.co/blog/managing-and-troubleshooting-elasticsearch-memory
