Knowledge Base

Showing results for 
Search instead for 
Did you mean: 

Events Service cluster stuck in red state with unallocated shards



The Events Service cluster is stuck in a RED state due to unassigned shards. 


Screen Shot 2018-06-28 at 11.52.11 AM.png





Use the following command to find the root cause of the issue:

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED



Some of the reasons for a shard to be in an unassigned state as given by elasticsearch:



Unassigned as a result of an API creation of an index.


Unassigned as a result of a full cluster recovery.


Unassigned as a result of opening a closed index.


Unassigned as a result of importing a dangling index.


Unassigned as a result of restoring into a new index.


Unassigned as a result of restoring into a closed index.


Unassigned as a result of explicit addition of a replica.


Unassigned as a result of a failed allocation of the shard.


Unassigned as a result of the node hosting it leaving the cluster.


Unassigned as a result of explicit cancel reroute command.


When a shard moves from started back to initializing, for example, with shadow replicas.


A better replica location is identified and causes the existing replica allocation to be canceled.



Use the following health check, shards, and indices command to determine the number of shards and which shards are in RED state:


curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v

Output from the above health check commands:
"events-service-api-store / elasticsearch-singlenode-module" : {
    "healthy" : false,
    "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [3], relocating shards: [0], initializing shards: [0], unassigned shards: [12], timed out: [false]"

appdynamics_meters_v2 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_meters_v2 1 p UNASSIGNED CLUSTER_RECOVERED appdynamics_api_keys_v1 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_api_keys_v1 1 p UNASSIGNED CLUSTER_RECOVERED appdynamics_accounts 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_accounts 1 p UNASSIGNED CLUSTER_RECOVERED event_type_metadata 0 p UNASSIGNED CLUSTER_RECOVERED event_type_metadata 1 p UNASSIGNED CLUSTER_RECOVERED event_type_extracted_fields 0 p UNASSIGNED CLUSTER_RECOVERED event_type_extracted_fields 1 p UNASSIGNED CLUSTER_RECOVERED





The solution is to manually allocate the shards using the reroute API available from elasticsearch.


In the above output/example, there are 10 unassigned shards so each shard must be separately allocated.
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ 
"commands" : [ { 
"allocate" : { 
"index" : "appdynamics_accounts", 
"shard" : 0, 
"node" : "node-12345798-7867-4ea1-b546-557a6bc546afg", 
"allow_primary" : true 




Thumbs Up Icon Kudos Screen Shot 2017-12-11 at 12.22.35 PM.pngWas this helpful? Give it kudos [click on the thumbs up icon at the end of the article, bottom left corner] to let others know too!

Do you have a question or need clarification? [Post Your Comment] Below. 


Version history
Revision #:
5 of 11
Last update:
‎07-24-2018 08:40 AM
Updated by:
Labels (1)
Tags (1)

Found this article helpful? Click the Thumbs Up button.
Have an additional comment? Post it below.
0 Kudos

Here is a powershell equivalent if you are running on Windows and don't want to install curl...




	"commands": [{
		"allocate": {
			"index": "appdynamics_accounts",
			"shard": 0,
			"node": "node-fa635229-8114-4ff8-a3b6-eee4449fkdsaaaa",
			"allow_primary": true

Invoke-WebRequest -Uri http://localhost:9200/_cluster/reroute -ContentType "application/json" -Method POST -Body $postParams

Don't forget. You may need to enable the http property ( in the "" file under events-service/processor/conf to allow port 9200 activity.