Knowledge Base

cancel
Showing results for 
Search instead for 
Did you mean: 

Events Service cluster stuck in red state with unallocated shards

Problem

 

The Events Service cluster is stuck in a RED state due to unassigned shards. 

 

Screen Shot 2018-06-28 at 11.52.11 AM.png

 

 

Troubleshooting

 

Use the following command to find the root cause of the issue:

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

 

 

Some of the reasons for a shard to be in an unassigned state as given by elasticsearch:

 

INDEX_CREATED

Unassigned as a result of an API creation of an index.

CLUSTER_RECOVERED

Unassigned as a result of a full cluster recovery.

INDEX_REOPENED

Unassigned as a result of opening a closed index.

DANGLING_INDEX_IMPORTED

Unassigned as a result of importing a dangling index.

NEW_INDEX_RESTORED

Unassigned as a result of restoring into a new index.

EXISTING_INDEX_RESTORED

Unassigned as a result of restoring into a closed index.

REPLICA_ADDED

Unassigned as a result of explicit addition of a replica.

ALLOCATION_FAILED

Unassigned as a result of a failed allocation of the shard.

NODE_LEFT

Unassigned as a result of the node hosting it leaving the cluster.

REROUTE_CANCELLED

Unassigned as a result of explicit cancel reroute command.

REINITIALIZED

When a shard moves from started back to initializing, for example, with shadow replicas.

REALLOCATED_REPLICA

A better replica location is identified and causes the existing replica allocation to be canceled.

 

 

Use the following health check, shards, and indices command to determine the number of shards and which shards are in RED state:

 

curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v


Output from the above health check commands:
 
"events-service-api-store / elasticsearch-singlenode-module" : {
    "healthy" : false,
    "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [3], relocating shards: [0], initializing shards: [0], unassigned shards: [12], timed out: [false]"
  }
}

appdynamics_meters_v2 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_meters_v2 1 p UNASSIGNED CLUSTER_RECOVERED appdynamics_api_keys_v1 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_api_keys_v1 1 p UNASSIGNED CLUSTER_RECOVERED appdynamics_accounts 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_accounts 1 p UNASSIGNED CLUSTER_RECOVERED event_type_metadata 0 p UNASSIGNED CLUSTER_RECOVERED event_type_metadata 1 p UNASSIGNED CLUSTER_RECOVERED event_type_extracted_fields 0 p UNASSIGNED CLUSTER_RECOVERED event_type_extracted_fields 1 p UNASSIGNED CLUSTER_RECOVERED

 

 

Solution

 

The solution is to manually allocate the shards using the reroute API available from elasticsearch.

 

In the above output/example, there are 10 unassigned shards so each shard must be separately allocated.
 
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ 
"commands" : [ { 
"allocate" : { 
"index" : "appdynamics_accounts", 
"shard" : 0, 
"node" : "node-12345798-7867-4ea1-b546-557a6bc546afg", 
"allow_primary" : true 
} 
}
]}'

 

 

 ___________________________________________________________________________

Thumbs Up Icon Kudos Screen Shot 2017-12-11 at 12.22.35 PM.pngWas this helpful? Give it kudos [click on the thumbs up icon at the end of the article, bottom left corner] to let others know too!

Do you have a question or need clarification? [Post Your Comment] Below. 

 

Version history
Revision #:
5 of 9
Last update:
‎07-24-2018 08:40 AM
Updated by:
 
Labels (1)
Tags (1)
0 Kudos
Comments

Here is a powershell equivalent if you are running on Windows and don't want to install curl...

 

 

 

$postParams=@'
{
	"commands": [{
		"allocate": {
			"index": "appdynamics_accounts",
			"shard": 0,
			"node": "node-fa635229-8114-4ff8-a3b6-eee4449fkdsaaaa",
			"allow_primary": true
		}
	}]
}
'@

Invoke-WebRequest -Uri http://localhost:9200/_cluster/reroute -ContentType "application/json" -Method POST -Body $postParams

Don't forget. You may need to enable the http property (ad.es.node.http.enabled=true) in the "events-service-api-store.properties" file under events-service/processor/conf to allow port 9200 activity.