Symptoms
EUM Server will fail to start if the sync between EUM and Events Service or Analytics Server fails during the startup process.
Users should check the EUM Server and Events Service logs for the following error messages:
EUM Server log snippet
------start synchronize eum acct to analytics acct!---------
24 Mar 2017 11:25:50.112 +0100 main AD.AnalyticsAccountManage INFO ************** Starting Registration of Analytics Account for EUM account="xyz" **************
24 Mar 2017 11:25:50.476 +0100 main AD.AnalyticsAccountManage ERROR
| CreateAccount : account="xyz", Error Code [500], Exception [RestException(statusCode=500, code=Unexpected.Exception, HttpResponseProxy{HTTP/1.1 500 Internal Server Error [Date: Fri, 24 Mar 2017 10:25:50 GMT, X-Content-Security-Policy: default-src 'self', X-XSS-Protection: 0, X-Content-Type-Options: nosniff, Frame-Options: DENY, X-Frame-Options: DENY, Content-Type: application/json, Transfer-Encoding: chunked] ResponseEntityProxy{[Content-Type: application/json,Chunked: true]}}
24 Mar 2017 11:25:50.551 +0100 main AD.AnalyticsAccountManage ERROR
| failed bulk updating BrowserRecord
| java.lang.RuntimeException: bulk update event type=BrowserRecord failed!!! url=http:/xyz:9080/v2/events/BrowserRecord/_bulk, key=...
| {"statusCode":500,"code":"Unexpected.Exception",
Events Service logs:
[2017-03-24T11:12:59,937+01:00] [INFO ] [dw-83 - POST /v1/account] [c.a.a.p.a.resource.AccountResource] Account Service Request: [POST v1/account] - request [[{"accountName":"customer1""accessKey":"REDACTED","expirationDate":"2017-05-20T23:59Z","eumAccountName":"XYZ","licensingConfigurations":[]}]]
[2017-03-24T11:12:59,941+01:00] [ERROR] [dw-83 - POST /v1/account] [c.a.a.p.a.ElasticsearchAccountManager] Error occurred while upserting account configuration for accounts [customer1]
com.appdynamics.analytics.processor.exception.UnavailableException: Failed to find accounts [customer1]
Diagnosis
Review the output of the following health check commands to find if the shards are in an unassigned state.
curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v
curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
Output from the above health check commands:
"events-service-api-store / elasticsearch-singlenode-module" : {
"healthy" : false,
"message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [3], relocating shards: [0], initializing shards: [0], unassigned shards: [12], timed out: [false]"
}
}
curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
appdynamics_meters_v2 0 p UNASSIGNED CLUSTER_RECOVERED
appdynamics_meters_v2 1 p UNASSIGNED CLUSTER_RECOVERED
appdynamics_api_keys_v1 0 p UNASSIGNED CLUSTER_RECOVERED
appdynamics_api_keys_v1 1 p UNASSIGNED CLUSTER_RECOVERED
appdynamics_accounts 0 p UNASSIGNED CLUSTER_RECOVERED
appdynamics_accounts 1 p UNASSIGNED CLUSTER_RECOVERED
event_type_metadata 0 p UNASSIGNED CLUSTER_RECOVERED
event_type_metadata 1 p UNASSIGNED CLUSTER_RECOVERED
event_type_extracted_fields 0 p UNASSIGNED CLUSTER_RECOVERED
event_type_extracted_fields 1 p UNASSIGNED CLUSTER_RECOVERED
event_type_metadata_hidden_fields_v1 0 p UNASSIGNED CLUSTER_RECOVERED
event_type_metadata_hidden_fields_v1 1 p UNASSIGNED CLUSTER_RECOVERED
Check the events-service-api-store.log file (found in <Events Service home directory>/logs) for an error message that includes: Failed to execute phase [query], all shards failed.
[2017-03-24T10:33:00,445+01:00] [ERROR] [dw-83 - POST /v1/account] [c.a.a.p.a.ElasticsearchAccountManager] Error occurred while upserting account configuration for accounts []
com.appdynamics.analytics.processor.exception.UnavailableException: Failed to find accounts ]
at com.appdynamics.analytics.processor.elasticsearch.exception.ElasticSearchExceptionUtils.propagateAsEventException(ElasticSearchExceptionUtils.java:147) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.elasticsearch.exception.ElasticSearchExceptionUtils.propagateAsEventException(ElasticSearchExceptionUtils.java:117) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.propagateAsException(ElasticsearchAccountManager.java:324) ~[analytics-processor.jar:na]
at
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:62) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:52) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:100) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1112) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) ~[elasticsearch-1.7.3.jar:na]
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) ~[elasticsearch-1.7.3.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.getAllQueryMatches(ElasticsearchAccountManager.java:252) ~[analytics-processor.jar:na]
at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.findAccountConfigurationsInElasticsearch(ElasticsearchAccountManager.java:222) ~[analytics-processor.jar:na]
... 54 common frames omitted
Solution
1. Manually allocate the unassigned shards using the reroute API.
In the following example, there are 12 unassigned shards so each shard must be separately allocated.
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
"commands" : [ {
"allocate" : {
"index" : "appdynamics_meters_v2",
"shard" : 1,
"node" : "node-12345798-7867-4ea1-b546-557a6bc546afg",
"allow_primary" : true
}
}
]}'
2. Verify if the cluster health state is now a green state using the following curl commands.
curl http://<events_service_machine>:9081/healthcheck?pretty=true
curl http://<events_service_machine>:9200/_cat/shards?v
curl http://<events_service_machine>:9200/_cat/indices?v
3. Start the EUM Server to sync with the Events Service or Analytics Server.
