cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Not a customer? Start a free trial

Click the Start a free trial link to start a 15-day SaaS trial of our product and join our community as a trial user. If you are an existing customer do not start a free trial.

AppDynamics customers and established members should click the sign in button to authenticate.

Knowledge Base

What should I do when the EUM Server fails to start due to unassigned shards?

Symptoms

EUM Server will fail to start if the sync between EUM and Events Service or Analytics Server fails during the startup process.

 

Users should check the EUM Server and Events Service logs for the following error messages:

 

EUM Server log snippet

------start synchronize eum acct to analytics acct!---------
24 Mar 2017 11:25:50.112 +0100  main                  AD.AnalyticsAccountManage  INFO    ************** Starting Registration of Analytics Account for EUM account="xyz" **************
24 Mar 2017 11:25:50.476 +0100  main                  AD.AnalyticsAccountManage  ERROR   
 |    CreateAccount : account="xyz", Error Code [500], Exception [RestException(statusCode=500, code=Unexpected.Exception, HttpResponseProxy{HTTP/1.1 500 Internal Server Error [Date: Fri, 24 Mar 2017 10:25:50 GMT, X-Content-Security-Policy: default-src 'self', X-XSS-Protection: 0, X-Content-Type-Options: nosniff, Frame-Options: DENY, X-Frame-Options: DENY, Content-Type: application/json, Transfer-Encoding: chunked] ResponseEntityProxy{[Content-Type: application/json,Chunked: true]}}
24 Mar 2017 11:25:50.551 +0100  main                  AD.AnalyticsAccountManage  ERROR   
 |    failed bulk updating BrowserRecord
 |    java.lang.RuntimeException: bulk update event type=BrowserRecord failed!!!  url=http:/xyz:9080/v2/events/BrowserRecord/_bulk, key=...
 |    {"statusCode":500,"code":"Unexpected.Exception",


 Events Service logs:

[2017-03-24T11:12:59,937+01:00]  [INFO ]  [dw-83 - POST /v1/account]  [c.a.a.p.a.resource.AccountResource]  Account Service Request: [POST v1/account] - request [[{"accountName":"customer1""accessKey":"REDACTED","expirationDate":"2017-05-20T23:59Z","eumAccountName":"XYZ","licensingConfigurations":[]}]]
[2017-03-24T11:12:59,941+01:00]  [ERROR]  [dw-83 - POST /v1/account]  [c.a.a.p.a.ElasticsearchAccountManager]  Error occurred while upserting account configuration for accounts [customer1]
com.appdynamics.analytics.processor.exception.UnavailableException: Failed to find accounts [customer1]

 

Diagnosis

Review the output of the following health check commands to find if the shards are in an unassigned state.

 

curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v
curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

 

Output from the above health check commands:

 

"events-service-api-store / elasticsearch-singlenode-module" : {
    "healthy" : false,
    "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [3], relocating shards: [0], initializing shards: [0], unassigned shards: [12], timed out: [false]"
  }
}

curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED


appdynamics_meters_v2                0 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_meters_v2                1 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_api_keys_v1              0 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_api_keys_v1              1 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_accounts                 0 p UNASSIGNED CLUSTER_RECOVERED 
appdynamics_accounts                 1 p UNASSIGNED CLUSTER_RECOVERED 
event_type_metadata                  0 p UNASSIGNED CLUSTER_RECOVERED 
event_type_metadata                  1 p UNASSIGNED CLUSTER_RECOVERED 
event_type_extracted_fields          0 p UNASSIGNED CLUSTER_RECOVERED 
event_type_extracted_fields          1 p UNASSIGNED CLUSTER_RECOVERED 
event_type_metadata_hidden_fields_v1 0 p UNASSIGNED CLUSTER_RECOVERED 
event_type_metadata_hidden_fields_v1 1 p UNASSIGNED CLUSTER_RECOVERED 

  

Check the events-service-api-store.log file (found in <Events Service home directory>/logs) for an error message that includes: Failed to execute phase [query], all shards failed. 

 

[2017-03-24T10:33:00,445+01:00]  [ERROR]  [dw-83 - POST /v1/account]  [c.a.a.p.a.ElasticsearchAccountManager]  Error occurred while upserting account configuration for accounts []
com.appdynamics.analytics.processor.exception.UnavailableException: Failed to find accounts ]
at com.appdynamics.analytics.processor.elasticsearch.exception.ElasticSearchExceptionUtils.propagateAsEventException(ElasticSearchExceptionUtils.java:147) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.elasticsearch.exception.ElasticSearchExceptionUtils.propagateAsEventException(ElasticSearchExceptionUtils.java:117) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.propagateAsException(ElasticsearchAccountManager.java:324) ~[analytics-processor.jar:na]
at 

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:62) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:52) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:100) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1112) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) ~[elasticsearch-1.7.3.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.getAllQueryMatches(ElasticsearchAccountManager.java:252) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.findAccountConfigurationsInElasticsearch(ElasticsearchAccountManager.java:222) ~[analytics-processor.jar:na]
... 54 common frames omitted

Solution

1. Manually allocate the unassigned shards using the reroute API.

 

In the following example, there are 12 unassigned shards so each shard must be separately allocated.

 

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ 
"commands" : [ { 
"allocate" : { 
"index" : "appdynamics_meters_v2", 
"shard" : 1, 
"node" : "node-12345798-7867-4ea1-b546-557a6bc546afg", 
"allow_primary" : true 
} 
}
]}'

 

2. Verify if the cluster health state is now a green state using the following curl commands. 

 

curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v

 

 3. Start the EUM Server to sync with the Events Service or Analytics Server.

Version history
Last update:
‎09-05-2018 04:20 PM
Updated by:
Labels (1)
By replying you agree to the Terms and Conditions of the AppDynamics Community.