Discussion Feed

Users trying to view a breakdown of the API timing for each resource within a browser snapshot may see the following error and be unable to view information on cross-origin resource domains: Cross Origin Resource   The solution for this issue is to  allow timing information to be collected from your CDN or other domain.  Add the Timing-Allow-Origin header to all requests served by the CDN in your app code.   The value of the header can either be * (allow timing from requests from any domain) or a list of domains within quotations with spaces separating each one.   Example: "Timing-Allow-Origin" "*" "Timing-Allow-Origin" "http://foo.com http://bar.com"   After the header has been added to the app code, users should be able to see the breakdown of the timing information within a browser snapshot.    Example: Cross domain Resource Breakdown Timing Info If the problem persists, capture a HAR file and verify if the header is seen in the response. If not, the header may be stripped off by a proxy or other hardware.   Example:  Header set for Adrum.js resource   Timing-Allow-Origin ... View more

As of version 4.3.2, the Events Service port 9200 is disabled by default. This port is used for debugging purposes and will need to be turned on while troubleshooting.   Example:  Using the following commands to find the status of shards/indices: curl http://localhost:9200/_cat/shards?v curl http://localhost:9200/_cat/indices?v  will result in the following output: curl: (7) Failed to connect to localhost port 9200: Connection refused   To enable port 9200, locate the events-service-api-store.properties file within the conf directory of your events-service home directory ( <ES_HOME>/conf/ events-service-api-store.properties ).   Change the value of the  property ad.es.node.http.enabled  to true, save, and restart the Events Service server. # To run this service in a fully, self contained mode, without # the need for dedicated master nodes, this should be set to 'true'. # If there are dedicated elasticsearch masters, then this flag # should be set to 'false'. ad.es.node.master=true ad.es.node.http.enabled=true   Related Links: 4.3.2 Release Notes ... View more

The Events Service cluster may not function correctly and its health state will be listed as red if there are unwanted indices either in the unassigned or initializing state.   Example: In a browser or terminal, run the following: curl http://localhost:9081/healthcheck?pretty=true The output of the healthcheck command will show cluster state [RED] . "events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : true, "message" : "Current [appdynamics-events-service-cluster] cluster state: [RED], data nodes: [1], nodes: [1], active shards: [48], relocating shards: [0], initializing shards: [0], unassigned shards: [5], timed out: [false]" },   These unwanted indices are created as a result of the Nessus Vulnerability Scanner that gets run on the cluster port (default 9200).   Diagnosis:   Run the following command on the Events Service machine or in a browser to verify if there are unwanted indices.   http://<events_service_machine>:9200/_cat/shards?v [in Browser] OR curl http://<events_service_machine>:9200/_cat/shards?v [in the terminal of the Events Service machine]   Output example:   index shard prirep state docs store ip node vrvsgs1.html 4 p STARTED 0 114b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 4 r UNASSIGNED 0 168b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 0 r UNASSIGNED 0 791b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 0 p STARTED 0 125b 192.x.y.z node-xyzxyzxyz vrvsgs1.html 3 r STARTED 0 105b 192.x.y.z node-xyzxyzxyz gvanws1.html 4 p STARTED 0 125b 192.x.y.z node-xyzxyzxyz gvanws1.html 4 r STARTED 0 128b 192.x.y.z node-xyzxyzxyz gvanws1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz gvanws1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz gvanws1.html 3 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz oapssw1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz oapssw1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz oapssw1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 2 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz nrawos1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz nrawos1.html 0 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz spipe 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz spipe 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz spipe 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz spipe 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz psaoon1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz psaoon1.html 2 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz psaoon1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz psaoon1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 0 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz arenon1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 3 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz sprreo1.html 2 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz vepper1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz vepper1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz vepper1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz perl 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz perl 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz perl 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz gaowsa1.html 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz greprs1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz greprs1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz enoson1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz enoson1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz enoson1.html 0 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz vpnwwg1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz vpnwwg1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 4 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz arenon1.html 4 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz soaeap1.html 3 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz soaeap1.html 1 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 1 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 2 p STARTED 0 115b 192.x.y.z node-xyzxyzxyz pwvwgg1.html 2 r STARTED 0 144b 192.x.y.z node-xyzxyzxyz enawae1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz enawae1.html 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz sawmill6cl.exe 4 p UNASSIGNED 0 115b 192.x.y.z node-xyzxyzxyz sawmill6cl.exe 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz rogwsw1.html 4 r STARTED 0 108b 192.x.y.z node-xyzxyzxyz   Solution:   Delete these unwanted indices using the following commands:   Delete all unwanted indices using the following curl command in the machine terminal: curl -XDELETE http://<events-service_hostname>:9200/<index_name> Example: curl -XDELETE http://localhost:9200/rogwsw1.html Verify if unwanted indices have been removed using the following commands: curl http://<events_service_machine>:9200/_cat/shards?v curl http://<events_service_machine>:9200/_cat/indices?v Work with a network or security team to configure the Nessus Vulnerability Scanner toexclude scanning the Events Service cluster port (9200-9300).     ... View more

Instrument the Events Service process for performance tuning using the following steps:    Configure the Java Agent via the controller-info.xml file located in /AppD/AppServerAgent-4.x.x/ver4.x.x.0/conf . Add the following option to the  <Events Service home directory>/conf/events-service.vmoptions  file. -javaagent:/AppD/AppServerAgent-4.x.x/ver4.x.x.0/javaagent.jar in conf/events-service.vmoptions Restart the Events Service process. Verify from the Controller UI that you are able to see the Events Service business transactions have registered successfully. Example:       ... View more

  Use the Analytics Agent health check URL in order to check the flow of business transactions and logs, as well as the connection status from the server where the Analytic Agent is hosted to the Events Service.   Using curl on the command line, enter the health check URL  and a ppend the string  ?pretty=true  to format the output for readability.   Example: curl http://localhost:9091/healthcheck?pretty=true Note: Replace "localhost" with your public IP/DNS if you are accessing it from a different machine where the Analytic Agent is installed. The default Analytics Agent admin port is  9091.   Sample output:   curl http://localhost:9091/healthcheck?pretty=true { "analytics-agent / Build information" : { "healthy" : true, "message" : "buildName=108-4.3.0.next-build, buildHash=8992cf933916e227b6e7ac82a2f25bb15e6ef8d1, buildTimestamp=2017-03-09T01:20:14+0000, buildNumber=7902, jobName=build-analytics-shared, version=4.3.0.0" }, "analytics-agent / Configuration properties" : { "healthy" : true, "message" : "Dynamic properties: []" }, "analytics-agent / Connection to [https://analytics.api.appdynamics.com:443]" : { "healthy" : true }, "analytics-agent / Log Sources" : { "healthy" : true, "message" : "" }, "analytics-agent / http:eventType:publish:bt" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "analytics-agent / http:eventType:publish:log" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "analytics-agent / http:receive:bt[[/v1/sinks/bt/*, /v2/sinks/bt/*]]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "analytics-agent / pipelines" : { "healthy" : true, "message" : "[16] pipeline(s). bt-publish-0: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-1: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-2: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-3: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-4: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-5: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-6: [MessageSourceStage: 1, EventServicePublishStage: 0], bt-publish-7: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-7: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-6: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-5: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-4: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-3: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-2: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-1: [MessageSourceStage: 1, EventServicePublishStage: 0], log-publish-0: [MessageSourceStage: 1, EventServicePublishStage: 0]" }, "analytics-agent / queues" : { "healthy" : true, "message" : "[2] queues [[biz_txn_v1] ratio: [0.00], size: [0], capacity: [128], [log_v1] ratio: [0.00], size: [0], capacity: [128]]" }, "deadlocks" : { "healthy" : true }     Elements  Details analytics-agent / Build information Top level, if healthy indicates all sub-components are healthy as well. Contains the version number as well. "analytics-agent / Connection to [https://analytics.api.appdynamics.com:443]" Where is the Analytics Connecting to and in this case its SAAS events-service API analytics-agent / http:eventType:publish:bt This is a method which processes the incoming business transaction analytics data and sends it to the Events Service. 0.0000000 means no activity. 1.000000 means maxed out. Average over last five minutes. analytics-agent / http:receive:bt[/v1/sinks/bt/*] This is a method to which data is received from server agents configured to send business transaction analytics data. Average over last five minutes. analytics-agent / pipelines 16 pipelines total (current default). These processes the messages to be sent to the publish service. analytics-agent / queues Queue size default is 128. biz_txn_v1 refers to the queue for business transaction analytics data. If the queue is full, could be indicative of network and auth issues.           Note: If there's an authentication issue while connecting to Events Service,  the output would show as below:   }, "analytics-agent / Connection to [https://analytics.api.appdynamics.com:443]" : { "healthy" : false, "message" : "Status code: [401], Message: The supplied auth information is incorrect or not authorized."   To resolve the authentication issue, check that the event name, account name, and access key within the Analytic Agent properties file are set up correctly. Refer to the following link for details: Installing Agent Side Components.   Related Links: Troubleshooting Analytics Agent Issues   ... View more

There is a known issue in which the Events Service does not start after rolling back to version 4.2.8 after an unsuccessful upgrade to 4.3.    Log snippet: [2017-06-01T00:34:41,146+01:00] [ERROR] [main] [o.e.gateway.local.state.shards] [node-98f903f1-6e90-4d56-8da0-5ce85ce7de02] failed to read local state (started shards), exiting... org.elasticsearch.ElasticsearchException: unexpected field in shard state [index_uuid] at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:153) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.gateway.local.state.meta.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:297) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardStateInfo(LocalGatewayShardsState.java:188) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.loadShardsStateInfo(LocalGatewayShardsState.java:173) ~[elasticsearch-1.7.3.jar:na] at org.elasticsearch.gateway.local.state.shards.LocalGatewayShardsState.<init>(LocalGatewayShardsState.java:66)    This is due to corruption of the state files on all indices. The following steps will remove the corrupted or stale state files.     Windows:   Run command prompt as Administrator: cd /d D:\AppDynamics\events_service\data\appdynamics-events-service-cluster\nodes\0\indices To find all state files dir state-*.st /a /s   To delete all state files del state-*.st /a /s     Unix:   cd /AppDynamics/events_service/data/appdynamics-events-service-cluster/nodes/0/indices To find all the state files grep -r state-*.st or find . -name state-*.st -print To delete the corrupted state files rm -rf state-*.st       Relevant Links: Github issue - "Exception: Could not find a state file to recover from" ... View more

Use the health check URL on the Events Service admin port in order to get the status (green, yellow, or red) of the Events Service cluster.   In the command line using curl, enter the <localhost:9081>/healthcheck URL and append the string ?pretty=true to format the output for readability.     Example: curl http://localhost:9081/healthcheck?pretty=true Note: Replace "localhost" with your public IP/DNS. The default admin port is 9081.   Output example:    curl http://localhost:9081/healthcheck?pretty=true { "deadlocks" : { "healthy" : true }, "events-service-api-store / BizOutcome [stage.extract]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]" }, "events-service-api-store / BizOutcome [stage.filter]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) accepted events: [0.000000], discarded events: [0.000000], acceptance ratio: [0.000000]" }, "events-service-api-store / Build information" : { "healthy" : true, "message" : "buildName=123-4.3.0.next-build, buildHash=d0e4fc0167eff6b5fa974d1f9a8180e5a1e89044, buildTimestamp=2017-03-17T22:06:29+0000, buildNumber=3709, jobName=build-events-service, version=4.3.0.1" }, "events-service-api-store / Chunking.AllChunksTimer [CatchAllChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, "events-service-api-store / Chunking.AllChunksTimer [SeriesChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, "events-service-api-store / Chunking.AllChunksTimer [TimeRangeChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, "events-service-api-store / Chunking.Counts [CatchAllChunkingStrategy]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "events-service-api-store / Chunking.Counts [SeriesChunkingStrategy]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "events-service-api-store / Chunking.Counts [TimeRangeChunkingStrategy]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "events-service-api-store / Chunking.FirstChunkTimer [CatchAllChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, "events-service-api-store / Chunking.FirstChunkTimer [SeriesChunkingStrategy]" : { "healthy" : true, "message" : "Duration in seconds percentile distribution, 50%: [0.000000], 75%: [0.000000], 95%: [0.000000], 99%: [0.000000], mean: [0.000000], count: [0.000000]" }, ............................... "events-service-api-store / Resource [v3/events/{eventType}/usage - documents]" : { "healthy" : true, "message" : "Rates (Avg per second. Avg of last 5 min) success: [0.000000], user error: [0.000000], timeout: [0.000000], error: [0.000000]" }, "events-service-api-store / Rolling index management" : { "healthy" : true, "message" : "Healthcheck for [Rolling index management]: Is running flag is: [false], last execution successful flag is: [true]" }, "events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : true, "message" : "Current [appdynamics-events-service-cluster] cluster state: [GREEN], data nodes: [1], nodes: [1], active shards: [11], relocating shards: [0], initializing shards: [0], unassigned shards: [0], timed out: [false]" }, "events-service-api-store / pipelines" : { "healthy" : true, "message" : "[1] pipeline(s). biz-outcome: [MessageSourceStage: 1, FilterStage: 0, ExtractStage: 0, UpsertStage: 0]" }, "events-service-api-store / queues" : { "healthy" : true, "message" : "[1] queues [[biz-outcome-incoming-events] ratio: [0.00], size: [0], capacity: [1000]]" }   The above example shows the cluster state is green, which means fully functional with no unassigned, relocating or initialising shards.   "events-service-api-store / elasticsearch-singlenode-module" : { "healthy" : true, "message" : "Current [appdynamics-events-service-cluster] cluster state: [GREEN], data nodes: [1], nodes: [1], active shards: [11], relocating shards: [0], initializing shards: [0], unassigned shards: [0], timed out: [false]" },       To check the connectivity, use the ping URL on the Events Service default port (9080). A "pong" message means success.    curl http://localhost:9080/_ping _pong         ... View more

Symptoms Users who are configuring a snapshot repository to take backups of the Events Service cluster, as outlined in the documentation (Back Up Events Service Data), within a Windows environment may encounter the following error message:   Command to configure snapshot repository:   D:\events-service>bin\events-service.exe snapshot-configure-fs -p conf\events-service-api-store.properties Output of the above command:   [2017-05-10T16:36:39,389+04:00] Error occurred [Error executing request to event service. Status code: [500]. Exception message: ["Unknown server error: [[\\localhost\\appdynamics-events-service-backup] location [\\\\localhost\\appdynamics-events-service-backup\\appdynamics-events-service-cluster] doesn't match any of the locations specified by path.repo]"]. For more information, view the event service logs.]   Examples for property ad.es.backupmanager.path.repo in the events-service-api-store.properties file that all yield the same error: \\localhost\appdynamics-events-service-backup \\\\localhost\\appdynamics-events-service-backup {BACKUP_HOME} as an environment variable Diagnosis The following prerequisites need to be completed before configuring an Events Service snapshot repository within a Windows environment:   Map the directory as a writable network drive. Create a network share with the user running the snapshot commands and give the administrator read/write access. Verify that the network share is accessible from ALL the nodes of the Events Service cluster.  The value of the  ad.es.backupmanager.path.repo property must be the same between all nodes.   Solution Update the ad.es.backupmanager.path.repo=\\\\localhost\\appdynamics-events-Service-backup  on all nodes and restart them all before attempting the snapshot command.   Example: ... View more