How do I verify that the Analytics Dynamic Service is enabled, running, and forwarding BT events to Analytics Agent?

[Mohammed.Rayan]

The instructions in this document are specific to AppDynamics versions 4.2.x and 4.3.x App Agent, Controller. and Analytics Agent. The instructions assume that Transaction Analytics has been enabled, like so:

Analytics is enabled

Problem:

If the analytic plugin (analytics-dynamic-service-enabled) in the App Agent fails to connect to the Analytics Agent due to connectivity issues, or if the Analytics Agent is not running, then the plugin will keep attempting to connect to the Analytics Agent a number of times.

Until that time, the plugin will try to hold Transaction BT details in buffer. If the plugin cannot connect to the Analytics Agent, and the buffer holding the BT events is full, the buffer will drop the BT event details.

The following error message will be displayed in the logs:

ERROR Analytics - Transaction [/app/Health:92477754-7573-469c-8a4e-a37c7554db30] could not be saved. Internal buffers appear to be full

This means that when the Analytic Agent is not available, the App Agent will show error messages and the analytic plugin will assume that there is an issue with the Analytics Agent and shut off the component responsible for pushing the BT events data.

Log example: 

[appdynamics-analytics-writer1]   WARN HttpSink - Error occurred while attempting to send data to [http://localhost:9090/v2/sinks/bt]. Retry attempts remaining [1177]
org.apache.http.conn.HttpHostConnectException: Connect to localhost:9090 [localhost/127.0.0.1] failed: Connection refused

[appdynamics-analytics-writer0] 09 Oct 2017 22:19:18,480 FATAL SinkWriterTask - Stopping abruptly due to error
org.apache.http.conn.HttpHostConnectException: Connect to localhost:9090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)

Solution:

1. Make sure the Analytic Agent is running.
2. Refer to the following article for further troubleshooting: How to use the Analytics Agent health check URL
3. Execute one of the following options to recover the plugin from the shutdown state:
   • Reset the agent from the UI.
   • Switch analytics config off and on again from the UI.
   • Disable & Enable it back
   • Restart the appserver (App Agent), if that's an option.
4. This will cause the analytic plugin) to start collecting data and forward the data to the Analytic Agent.

Note: In version 4.2.x, there is another option to reset the plugin. It is possible to toggle off & on the plugin property (analytics-dynamic-service-enabled) from the UI. This option is not available in version 4.3.x.

Steps to toggle the plugin property for version 4.2.x:

1. Login to Controller UI.
2. Select the Applications tab.
3. Select the application in question.
4. Select Tiers & Nodes on the left.
5. Double-click on one of the nodes in question.
6. The new screen should populate information about the node you double-clicked. In the top of the page, there should be a sub-tab called Agents, click this.
7. From there you will want to make sure the sub-tab App Server Agent is selected and then click the Configure button.
8. A new window will appear, please select the Application, Tier, or Node which you want to set this property for (it will turn blue and pull up a list of properties on the right).
9. Click the plus sign (+) to create the property using the following information:
   • Name: analytics-dynamic-service-enabled
   • Description: [your choice]
   • Type: Boolean Value: true
10. Click Save in the new property pop-up.
11. Click Save once more in the Agent configuration screen.
    • This will refresh the App Agent's configuration. Expect to see this property enabled via the App Agent log files.
12. Verify the change from the App Agent logs.

Log Example:

[Thread-0] 04 Oct 2017 08:48:55,383  INFO AnnotationPropertyListenerManager - Propagating analytics-dynamic-service-enabled value true

Why isn't the above option available in version 4.3.x?

Version 4.2 and earlier App Server Agents use the analytics-dynamic-service-enabled node property to enable Transaction Analytics. Using the node property directly allowed for disabling and enabling the Analytics Dynamic Service at the application, tier, and node levels. The Analytics Configuration UI allows for setting this property only at the Application level. Therefore, Java Agents 4.3 and higher are not able to disable or enable the Analytics Dynamic Service at the tier and node levels.

Although the Analytics Dynamic Service configurations can be directly changed through the application import and the Analytics Dynamic Service import REST API, manipulating the Analytics Dynamic Service configurations through either of those mechanisms only updates the Analytics Dynamic Service configurations and not the corresponding analytics-dynamic-service-enabled property. Therefore, if you use one of these import mechanisms to enable or disable the Analytics Dynamic Service and you have a mixture of old (pre 4.3) and new (4.3 and higher) agents, you must manually set the associated property to match the Analytics Dynamic Service configuration value.

The analytics-dynamic-service-enabled node property can also be set as part of the application import and has its own import REST API (accessed at POST /setAgentProperty). See Import Application Analytics Dynamic Service Configuration at Configuration Import and Export API for more information.