Knowledge Base

cancel
Showing results for 
Search instead for 
Did you mean: 

What should I do when the EUM Server fails to start due to unassigned shards?

Symptoms

EUM Server will fail to start if the sync between EUM and Events Service or Analytics Server fails during the startup process.

 

Users should check the EUM Server and Events Service logs for the following error messages:

 

EUM Server log snippet

------start synchronize eum acct to analytics acct!---------
24 Mar 2017 11:25:50.112 +0100  main                  AD.AnalyticsAccountManage  INFO    ************** Starting Registration of Analytics Account for EUM account="xyz" **************
24 Mar 2017 11:25:50.476 +0100  main                  AD.AnalyticsAccountManage  ERROR   
 |    CreateAccount : account="xyz", Error Code [500], Exception [RestException(statusCode=500, code=Unexpected.Exception, HttpResponseProxy{HTTP/1.1 500 Internal Server Error [Date: Fri, 24 Mar 2017 10:25:50 GMT, X-Content-Security-Policy: default-src 'self', X-XSS-Protection: 0, X-Content-Type-Options: nosniff, Frame-Options: DENY, X-Frame-Options: DENY, Content-Type: application/json, Transfer-Encoding: chunked] ResponseEntityProxy{[Content-Type: application/json,Chunked: true]}}
24 Mar 2017 11:25:50.551 +0100  main                  AD.AnalyticsAccountManage  ERROR   
 |    failed bulk updating BrowserRecord
 |    java.lang.RuntimeException: bulk update event type=BrowserRecord failed!!!  url=http:/xyz:9080/v2/events/BrowserRecord/_bulk, key=...
 |    {"statusCode":500,"code":"Unexpected.Exception",


 Events Service logs:

[2017-03-24T11:12:59,937+01:00]  [INFO ]  [dw-83 - POST /v1/account]  [c.a.a.p.a.resource.AccountResource]  Account Service Request: [POST v1/account] - request [[{"accountName":"customer1""accessKey":"REDACTED","expirationDate":"2017-05-20T23:59Z","eumAccountName":"XYZ","licensingConfigurations":[]}]]
[2017-03-24T11:12:59,941+01:00]  [ERROR]  [dw-83 - POST /v1/account]  [c.a.a.p.a.ElasticsearchAccountManager]  Error occurred while upserting account configuration for accounts [customer1]
com.appdynamics.analytics.processor.exception.UnavailableException: Failed to find accounts [customer1]

 

Diagnosis

Review the output of the following health check commands to find if the shards are in an unassigned state.

 

curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v
curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

 

Output from the above health check commands:

 

"events-service-api-store / elasticsearch-singlenode-module" : {
    "healthy" : false,
    "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [3], relocating shards: [0], initializing shards: [0], unassigned shards: [12], timed out: [false]"
  }
}

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED


appdynamics_meters_v2                0 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_meters_v2                1 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_api_keys_v1              0 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_api_keys_v1              1 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_accounts                 0 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_accounts                 1 p UNASSIGNED CLUSTER_RECOVERED 
event_type_metadata                  0 p UNASSIGNED CLUSTER_RECOVERED 
event_type_metadata                  1 p UNASSIGNED CLUSTER_RECOVERED 
event_type_extracted_fields          0 p UNASSIGNED CLUSTER_RECOVERED 
event_type_extracted_fields          1 p UNASSIGNED CLUSTER_RECOVERED 
event_type_metadata_hidden_fields_v1 0 p UNASSIGNED CLUSTER_RECOVERED 
event_type_metadata_hidden_fields_v1 1 p UNASSIGNED CLUSTER_RECOVERED 

  

Check the events-service-api-store.log file (found in <Events Service home directory>/logs) for an error message that includes: Failed to execute phase [query], all shards failed. 

 

[2017-03-24T10:33:00,445+01:00]  [ERROR]  [dw-83 - POST /v1/account]  [c.a.a.p.a.ElasticsearchAccountManager]  Error occurred while upserting account configuration for accounts []
com.appdynamics.analytics.processor.exception.UnavailableException: Failed to find accounts ]
at com.appdynamics.analytics.processor.elasticsearch.exception.ElasticSearchExceptionUtils.propagateAsEventException(ElasticSearchExceptionUtils.java:147) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.elasticsearch.exception.ElasticSearchExceptionUtils.propagateAsEventException(ElasticSearchExceptionUtils.java:117) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.propagateAsException(ElasticsearchAccountManager.java:324) ~[analytics-processor.jar:na]
at 

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:62) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:52) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:100) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1112) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) ~[elasticsearch-1.7.3.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.getAllQueryMatches(ElasticsearchAccountManager.java:252) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.findAccountConfigurationsInElasticsearch(ElasticsearchAccountManager.java:222) ~[analytics-processor.jar:na]
... 54 common frames omitted

Solution

1. Manually allocate the unassigned shards using the reroute API.

 

In the following example, there are 12 unassigned shards so each shard must be separately allocated.

 

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ 
"commands" : [ { 
"allocate" : { 
"index" : "appdynamics_meters_v2", 
"shard" : 1, 
"node" : "node-12345798-7867-4ea1-b546-557a6bc546afg", 
"allow_primary" : true 
} 
}
]}'

 

2. Verify if the cluster health state is now a green state using the following curl commands. 

 

curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v

 

 3. Start the EUM Server to sync with the Events Service or Analytics Server.

Version history
Revision #:
6 of 6
Last update:
‎09-05-2018 04:20 PM
Updated by:
 
Labels (1)
0 Kudos