Not a customer? Click the 'Start a free trial' link to begin a 30-day SaaS trial of our product and to join our community.
Existing Cisco AppDynamics customers should click the 'Sign In' button to authenticate to access the community
on 05-09-2017 10:31 AM - edited on 09-05-2018 04:20 PM by Nina.Wolinsky
EUM Server will fail to start if the sync between EUM and Events Service or Analytics Server fails during the startup process.
Users should check the EUM Server and Events Service logs for the following error messages:
EUM Server log snippet ------start synchronize eum acct to analytics acct!--------- 24 Mar 2017 11:25:50.112 +0100 main AD.AnalyticsAccountManage INFO ************** Starting Registration of Analytics Account for EUM account="xyz" ************** 24 Mar 2017 11:25:50.476 +0100 main AD.AnalyticsAccountManage ERROR | CreateAccount : account="xyz", Error Code [500], Exception [RestException(statusCode=500, code=Unexpected.Exception, HttpResponseProxy{HTTP/1.1 500 Internal Server Error [Date: Fri, 24 Mar 2017 10:25:50 GMT, X-Content-Security-Policy: default-src 'self', X-XSS-Protection: 0, X-Content-Type-Options: nosniff, Frame-Options: DENY, X-Frame-Options: DENY, Content-Type: application/json, Transfer-Encoding: chunked] ResponseEntityProxy{[Content-Type: application/json,Chunked: true]}} 24 Mar 2017 11:25:50.551 +0100 main AD.AnalyticsAccountManage ERROR | failed bulk updating BrowserRecord | java.lang.RuntimeException: bulk update event type=BrowserRecord failed!!! url=http:/xyz:9080/v2/events/BrowserRecord/_bulk, key=... | {"statusCode":500,"code":"Unexpected.Exception", Events Service logs: [2017-03-24T11:12:59,937+01:00] [INFO ] [dw-83 - POST /v1/account] [c.a.a.p.a.resource.AccountResource] Account Service Request: [POST v1/account] - request [[{"accountName":"customer1""accessKey":"REDACTED","expirationDate":"2017-05-20T23:59Z","eumAccountName":"XYZ","licensingConfigurations":[]}]] [2017-03-24T11:12:59,941+01:00] [ERROR] [dw-83 - POST /v1/account] [c.a.a.p.a.ElasticsearchAccountManager] Error occurred while upserting account configuration for accounts [customer1] com.appdynamics.analytics.processor.exception.UnavailableException: Failed to find accounts [customer1]
Review the output of the following health check commands to find if the shards are in an unassigned state.
curl http://<events_service_machine>:9081/healthcheck?pretty=true curl http://<events_service_machine>:9200/_cat/shards?v curl http://<events_service_machine>:9200/_cat/indices?v
curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
Output from the above health check commands:
"events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : false, "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [3], relocating shards: [0], initializing shards: [0], unassigned shards: [12], timed out: [false]" } } curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED appdynamics_meters_v2 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_meters_v2 1 p UNASSIGNED CLUSTER_RECOVERED appdynamics_api_keys_v1 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_api_keys_v1 1 p UNASSIGNED CLUSTER_RECOVERED appdynamics_accounts 0 p UNASSIGNED CLUSTER_RECOVERED appdynamics_accounts 1 p UNASSIGNED CLUSTER_RECOVERED event_type_metadata 0 p UNASSIGNED CLUSTER_RECOVERED event_type_metadata 1 p UNASSIGNED CLUSTER_RECOVERED event_type_extracted_fields 0 p UNASSIGNED CLUSTER_RECOVERED event_type_extracted_fields 1 p UNASSIGNED CLUSTER_RECOVERED event_type_metadata_hidden_fields_v1 0 p UNASSIGNED CLUSTER_RECOVERED event_type_metadata_hidden_fields_v1 1 p UNASSIGNED CLUSTER_RECOVERED
Check the events-service-api-store.log file (found in <Events Service home directory>/logs) for an error message that includes: Failed to execute phase [query], all shards failed.
[2017-03-24T10:33:00,445+01:00] [ERROR] [dw-83 - POST /v1/account] [c.a.a.p.a.ElasticsearchAccountManager] Error occurred while upserting account configuration for accounts [] com.appdynamics.analytics.processor.exception.UnavailableException: Failed to find accounts ] at com.appdynamics.analytics.processor.elasticsearch.exception.ElasticSearchExceptionUtils.propagateAsEventException(ElasticSearchExceptionUtils.java:147) ~[analytics-processor.jar:na] at com.appdynamics.analytics.processor.elasticsearch.exception.ElasticSearchExceptionUtils.propagateAsEventException(ElasticSearchExceptionUtils.java:117) ~[analytics-processor.jar:na] at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.propagateAsException(ElasticsearchAccountManager.java:324) ~[analytics-processor.jar:na] at Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to execute phase [query], all shards failed at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:160) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:62) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:52) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:100) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:98) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:338) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.search.SearchRequestBuilder.doExecute(SearchRequestBuilder.java:1112) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:91) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:65) ~[elasticsearch-1.7.3.jar:na] at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.getAllQueryMatches(ElasticsearchAccountManager.java:252) ~[analytics-processor.jar:na] at com.appdynamics.analytics.processor.account.ElasticsearchAccountManager.findAccountConfigurationsInElasticsearch(ElasticsearchAccountManager.java:222) ~[analytics-processor.jar:na] ... 54 common frames omitted
1. Manually allocate the unassigned shards using the reroute API.
In the following example, there are 12 unassigned shards so each shard must be separately allocated.
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ { "allocate" : { "index" : "appdynamics_meters_v2", "shard" : 1, "node" : "node-12345798-7867-4ea1-b546-557a6bc546afg", "allow_primary" : true } } ]}'
2. Verify if the cluster health state is now a green state using the following curl commands.
curl http://<events_service_machine>:9081/healthcheck?pretty=true curl http://<events_service_machine>:9200/_cat/shards?v curl http://<events_service_machine>:9200/_cat/indices?v
3. Start the EUM Server to sync with the Events Service or Analytics Server.
Thank you! Your submission has been received!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form