This article outlines how to troubleshoot the Java Agent when Transaction Analytics is not receiving any data.
Table of Contents
How do I start troubleshooting missing transaction data?
How do I test for communication issues between agents?
How do I start troubleshooting missing transaction analytics data?
If you aren't seeing transaction analytics data under Analytics, start by troubleshooting from the origin of the events, following the steps below:
- Check if Analytics is enabled for the Business Transactions (BT) of the application:
- Go to Analytics > Configuration > Transaction Analytics > Configure Analytics for Application.
- Make sure the Enable Analytics Data Collection for <Application> checkbox is selected and that the BT is present in the list on the left-hand side under Configure Which Transactions Report Analytics Data.
- Open the Application Agent log to look for the
analytics-dynamics-service-enabled property, and make sure it's set to true.
- Then, look for log entries similar to the ones shown below. This ensures the Dynamic Services for Analytics Data Collection have started and that the Java Agent is now sending the analytics-enabled BT events to the Analytics Agent.
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,278 INFO AnalyticsDynamicServiceConfigListener - Analytics Dynamic Service configuration listener received new properties from the Controller, starting refresh operation.
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,278 INFO AnalyticsDynamicServiceConfigListener - ********** Started applying Analytics Dynamic Service properties from Controller *************
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,279 INFO Analytics - Analytics Dynamic Service config [ENABLED] = true
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,302 INFO WriterMaster - Initialized with writer parameters [65536 (appdynamics.analytics.bufferPool.maxCount), 65536 (appdynamics.analytics.tlsCharBuffer.maxLength), 128 (appdynamics.analytics.unflushedQueue.maxLength), 2 (appdynamics.analytics.numOfSinkWriterTasks), 102400 (appdynamics.analytics.message.maxSizeBytes), 10000] and sink parameters ['http://localhost:9090/v2/sinks/bt' (appdynamics.analytics.agent.url), 30000 (appdynamics.analytics.agent.connection.timeoutMillis), 1200 (appdynamics.analytics.agent.send.attempt.max), 15000 (appdynamics.analytics.agent.send.attempt.pauseMillis), 16 (appdynamics.analytics.agent.send.batch.items.max), 60000 (appdynamics.analytics.throughputCollectionIntervalMillis)]
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,304 INFO WriterMaster - Starting
[appdynamics-analytics-writer0] 17 Oct 2018 23:34:46,319 INFO SinkWriterTask - Starting
[appdynamics-analytics-writer1] 17 Oct 2018 23:34:46,320 INFO SinkWriterTask - Starting
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,321 INFO WriterMaster - Started
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,324 INFO AnalyticsCollectorFactory - Analytics configured with new delegate: [com.singularity.ee.service.analytics.Analytics]
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,324 INFO Analytics - Started [Analytics] collector
[AD Thread Pool-Global0] 17 Oct 2018 23:34:46,324 INFO AnalyticsDynamicServiceConfigListener - --------- Finished applying Analytics Dynamic Service properties from Controller ------------
- Check to see if the
analytics-dynamics-service-enabled property is set to true.
If it is true, but the later logs never confirm WriterMaster - Started or Analytics - Started [Analytics] collector, it's possible that the analytics-dynamics-service.jar file under javaagent/ver4.x.x.xxxxx/external-service/analytics is not present.
In this case, please download a valid Java Agent installer package, which has analytics-dynamics-service.jar under javaagent/ver4.x.x.xxxxx/external-service/analytics. Then instrument the application again.
- Check if the
analytics-dynamics-service.jar file is present under the javaagent/ver4.x.x.xxxxx/external-service/analytic directory.
If present, open javaagent/ver4.x.x.xxxxx/conf/app-agent-config.xml and look for the following:
<agent-service name="DynamicServiceManager"
implementation-class-name="com.singularity.ee.agent.appagent.kernel.DynamicServiceManager"
enabled="true">
<implementation-class-name>
com.singularity.ee.agent.appagent.kernel.DynamicServiceManager
</implementation-class-name>
<configuration-properties>
<property name="external-service-directory" value="external-services"/>
</configuration-properties>
</agent-service>
If it's not present, please add it.
- If the above config is present, open the latest
javaagent/ver4.x.x.xxxxx/logs/dynamic-service*.log and look for any error in loading Analytics Dynamics Services.
Check for an an error similar to the following:
[AD Thread Pool-Global0] 18 Oct 2018 13:02:56,585 INFO com.singularity.dynamicservice.DynamicServiceManager - Service class name com.singularity.ee.service.analytics.Analytics
[AD Thread Pool-Global0] 18 Oct 2018 13:02:56,590 ERROR com.singularity.dynamicservice.DynamicServiceManager - Error occurred while polling for new services
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_102]
at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_102]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_102]
at java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:1528) ~[?:1.8.0_102]
at java.lang.ClassLoader.getParent(ClassLoader.java:1374) ~[?:1.8.0_102]
at com.singularity.ee.agent.appagent.kernel.classloader.AgentClassLoader.a(AgentClassLoader.java:432) ~[appagent-boot.jar:?]
at com.singularity.ee.agent.appagent.kernel.classloader.AgentClassLoader.c(AgentClassLoader.java:415) ~[appagent-boot.jar:?]
at com.singularity.ee.agent.appagent.kernel.classloader.d.b(d.java:69) ~[appagent-boot.jar:?]
at com.singularity.ee.agent.appagent.kernel.classloader.AgentClassLoader.loadClass(AgentClassLoader.java:305) ~[appagent-boot.jar:?]
at java.lang.ClassLoader.loadClass(ClassLoader.java:411) ~[?:1.8.0_102]
at com.singularity.ee.util.c.loadClass(c.java:320) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.agent.appagent.kernel.c.loadClass(c.java:84) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_102]
at com.singularity.ee.service.analytics.Analytics.<init>(Analytics.java:70) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_102]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_102]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_102]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_102]
at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_102]
at com.singularity.ee.agent.util.reflect.l.f(l.java:1693) ~[appagent-boot.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.agent.appagent.kernel.DynamicServiceManager.pollDirectoryForNewServices(DynamicServiceManager.java:476) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.agent.appagent.kernel.DynamicServiceManager.pollForNewServices(DynamicServiceManager.java:329) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.agent.appagent.kernel.DynamicServiceManager.run(DynamicServiceManager.java:272) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.util.javaspecific.scheduler.n.run(n.java:122) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_102]
at com.singularity.ee.util.javaspecific.scheduler.z.e(z.java:335) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.util.javaspecific.scheduler.a.b(a.java:152) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.util.javaspecific.scheduler.b.a(b.java:119) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.util.javaspecific.scheduler.b.b(b.java:206) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.util.javaspecific.scheduler.b.run(b.java:236) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.util.javaspecific.scheduler.i.a(i.java:694) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at com.singularity.ee.util.javaspecific.scheduler.i.run(i.java:726) ~[appagent.jar:Server Agent #4.5.0.23604 v4.5.0 GA compatible with 4.4.1.0 r88b53d319a45abc58963be9d2d0824499a2c23dd release/4.5.0-release]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
If there is such an error, you need to provide the permission in the java.policy file under <JAVA_HOME>/lib/security/java.policy as follows:
grant codeBase "file:/opt/appdynamics/-" {
permission java.security.AllPermission;
permission java.lang.management.ManagementPermission "monitor";
permission java.io.FilePermission "read";
};
This should fix the Analytics Dynamics Service startup issue and start reporting events to the Analytics Agent, which is responsible for forwarding these events further to the Events Service.
- In the Java agent log, are we seeing the following WARN message? It tells us that your Analytics data is being dropped because you have reached the limit.
[pool-63794-thread-3] 21 Jul 2019 19:56:07,004 ERROR AnalyticsCollectorFactory - Bounds exceeded creating analytics collector. Disabling Analytics Collection.
The Analytics collector is attached to each thread that is spawned so that it can detect and publish the analytics information. When the number of threads goes beyond 2200, it shuts off the Analytics component from the Agent.
To get past this error, please add the following to the <agent-services> section of the app-agent-config.xml file for the Agent:
<agent-service name="Analytics" enabled="true" minimum-java-version="1.5">
<configuration-properties>
<property name="analytics-collectors-limit" value="5000,WARN_IF_BOUND_EXCEEDED"/>
</configuration-properties>
</agent-service>
After adding this, restart the app agent once.
This will let the collection grow without bound, but will log every time something is added beyond the bound. No data will be dropped.
How do I test for communication issues between agents?
Once the above issues are fixed, you can expect events to be forwarded to the Analytics Agent. However, if you still don't see data in Analytics, you should look for communication issues between the Analytics Agent and the Java Agent.
- Check if the Java Agent is configured to send the events to the correct Analytics Agent endpoint.
By default, the Java Agent sends events to the Analytics Agent running at http://localhost:9090. If this is not the case, you have to explicitly set the JVM parameter to talk to the remote Analytics Agent. This is discussed in detail in the Installing Agent-Side Components document.
- If for some reason (e.g., maintenance or communication configuration issues) the Java Agent is not able to talk to Analytics Agent for a very long period of time, the Java Agent will queue the events on its end, ready for the Analytics Agent communication to be back up. However, the communication is not available before the buffer's memory is exceeded, you will see a WARNING in the Java Agent log that says "Internal buffers appear to be full."
After multiple failed attempts, the Java Agent will disable the Analytics Service within itself and stop sending data to the Analytics Agent. To resume sending data, go to Analytics > Configuration > Transaction Analytics > Configure Analytics for Application and disable and re-enable the Enable Analytics Data Collection for <Application> checkbox.
If this does not work, try a JVM restart to resume the service.
