Java Agent seems started successfully for jetty application but not report any analytics data

[cskingslayer]

Hi,

I met some issues trying to use AppDynamics Java agent, I made some search but not able to find similar case, so I raised it here to seek for help. My app is running on jetty9.4.4 server, Java8

I've configured applicationName, nodeName and tierName, followed the instructions of jetty configuration to start the application. I could see logs that indicated Appdynamics Java Agent is successful and in console, "Agent Connected" is green and "waiting for data" didn't ends, but after waiting for quite a while, there's still no statistics data.

Starting Jetty: Full Agent Registration Info Resolver found system property [appdynamics.agent.applicationName] for application name [bank-portal]
Full Agent Registration Info Resolver found system property [appdynamics.agent.tierName] for tier name [portal]
Full Agent Registration Info Resolver found system property [appdynamics.agent.nodeName] for node name [bank-portal]
Full Agent Registration Info Resolver using selfService [true]
Full Agent Registration Info Resolver using selfService [true]
Full Agent Registration Info Resolver using application name [bank-portal]
Full Agent Registration Info Resolver using tier name [portal]
Full Agent Registration Info Resolver using node name [bank-portal]
Install Directory resolved to[/opt/appdynamics/appagent]
[Thread-0] Mon Aug 14 13:48:32 AEST 2017[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver is running
[Thread-0] Mon Aug 14 13:48:32 AEST 2017[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found system property [appdynamics.agent.applicationName] for application name [bank-portal]
[Thread-0] Mon Aug 14 13:48:32 AEST 2017[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found system property [appdynamics.agent.tierName] for tier name [portal]
[Thread-0] Mon Aug 14 13:48:32 AEST 2017[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found system property [appdynamics.agent.nodeName] for node name [bank-portal]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [true]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [true]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using application name [bank-portal]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using tier name [portal]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using node name [bank-portal]
log4j:WARN No appenders could be found for logger (com.singularity.SelfServiceNameResolver).
log4j:WARN Please initialize the log4j system properly.
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver finished running
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: AgentInstallManager - Agent runtime directory set to [/opt/appdynamics/appagent/ver4.3.5.0]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: AgentInstallManager - Agent node directory set to [bank-portal]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: JavaAgent - Using Java Agent Version [Server Agent v4.3.5.0 GA #15366 r429232a0b601543e1fdf4e1fb8fa041efa136556 2835-4.3.5.next-analytics]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: JavaAgent - Running IBM Java Agent [No]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: JavaAgent - Java Agent Directory [/opt/appdynamics/appagent/ver4.3.5.0]
[Thread-0] Mon Aug 14 13:48:33 AEST 2017[INFO]: JavaAgent - Java Agent AppAgent directory [/opt/appdynamics/appagent/ver4.3.5.0]
Agent Logging Directory [/opt/appdynamics/appagent/ver4.3.5.0/logs/bank-portal]
Running obfuscated agent
. . . . Registered app server agent with Node ID[88307] Component ID[42943] Application ID [177519]
. Started AppDynamics Java Agent Successfully.

In /opt/appdynamics/appagent/ver4.3.5.0/logs/bank-portal/agent.2017_08_14__13_48_33.0.log file, it seems got some error:

......
[AD Thread Pool-Global0] 14 Aug 2017 13:49:49,800 INFO JMXMetricRepository - Added new JMX Rule [JMXMetricRule [ MBeanQuery [MBeanQuery [ domain [kafka.producer], mbeanPattern [kafka.producer:type=producer-metrics,*], queryLogicalOperator [null], queryExpressions []]], metricCategory [ProducerMetrics], beanName [null], metricPath [null], instanceName [null], instanceIdentifier [client-id], name [Kafka_Producer], domain [kafka.producer] Attribute Definitions [ JMX Attribute Definition [mbeanAttributeName [response-rate], metricName [Response Rate], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [request-rate], metricName [Request Rate], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [request-latency-avg], metricName [Average Request Latency], metricTimeRollupType [CURRENT], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [OBSERVATION], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [outgoing-byte-rate], metricName [Outgoing Byte Rate], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [io-wait-time-ns-avg], metricName [Average IO Wait], metricTimeRollupType [CURRENT], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [OBSERVATION], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [record-error-rate], metricName [Record Error Rate], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [waiting-threads], metricName [Waiting Threads], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [requests-in-flight], metricName [Requests in Flight], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [network-io-rate], metricName [Network IO Rate], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ] ] ]]
[AD Thread Pool-Global0] 14 Aug 2017 13:49:49,800 INFO JMXMetricRepository - Added new JMX Rule [JMXMetricRule [ MBeanQuery [MBeanQuery [ domain [kafka.consumer], mbeanPattern [kafka.consumer:type=consumer-fetch-manager-metrics,*], queryLogicalOperator [null], queryExpressions []]], metricCategory [ConsumerMetrics], beanName [null], metricPath [null], instanceName [null], instanceIdentifier [client-id], name [Kafka_Consumer], domain [kafka.consumer] Attribute Definitions [ JMX Attribute Definition [mbeanAttributeName [records-lag-max], metricName [Max Records Lag], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [OBSERVATION], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [bytes-consumed-rate], metricName [Bytes Consumption Rate], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [fetch-rate], metricName [Fetch Rate], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [records-consumed-rate], metricName [Records Consumption Rate], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [null] ]JMX Attribute Definition [mbeanAttributeName [fetch-latency-max], metricName [Max Fetch Latency], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [OBSERVATION], metricHoleType [null], getterChain [null] ] ] ]]
[AD Thread-Metric Reporter0] 14 Aug 2017 13:50:35,118 ERROR ServerMBeanManager - Failed to discover any domains. Consider increasing MBeanFinderDelay
[AD Thread Pool-Global0] 14 Aug 2017 13:50:35,246 INFO TransformationManager - Class java/util/logging/Logger is scheduled for re-transformation
[AD Thread Pool-Global0] 14 Aug 2017 13:50:35,301 INFO TransformationManager - Class jdk/Exported is scheduled for re-transformation
[AD Thread Pool-Global0] 14 Aug 2017 13:50:35,323 INFO TransformationManager - Class transformations will not take place in a background thread
[AD Thread Pool-Global0] 14 Aug 2017 13:50:49,528 INFO MBeanRegistrationManager - MBean com.singularity.ee.agent.appagent.services.bciengine.BCIEngineInfoProvider@26a8f44f with ObjectName AppDynamics:type=BCIEngineInfoProvider has been registered
[AD Thread Pool-Global0] 14 Aug 2017 13:50:49,529 INFO MBeanRegistrationManager - MBean com.singularity.ee.agent.appagent.services.snapshot.SnapshotInfoProvider@4a6d650d with ObjectName AppDynamics:type=SnapshotInfoProvider has been registered
[AD Thread Pool-Global0] 14 Aug 2017 13:50:49,531 INFO MBeanRegistrationManager - MBean com.singularity.ee.agent.appagent.services.transactionmonitor.TransactionMonitorInfoProvider@3bcfeb97 with ObjectName AppDynamics:type=TransactionMonitorInfoProvider has been registered
[AD Thread Pool-Global1] 14 Aug 2017 13:50:50,078 INFO AnalyticsDynamicServiceConfigListener - Analytics Dynamic Service configuration listener received new properties from the Controller, starting refresh operation.
[AD Thread Pool-Global1] 14 Aug 2017 13:50:50,078 INFO AnalyticsDynamicServiceConfigListener - ********** Started applying Analytics Dynamic Service properties from Controller *************
[AD Thread Pool-Global1] 14 Aug 2017 13:50:50,079 INFO Analytics - Analytics Dynamic Service config [ENABLED] = false
[AD Thread Pool-Global1] 14 Aug 2017 13:50:50,079 INFO Analytics - [Analytics] collector was not running. Stop routine will not be performed
[AD Thread Pool-Global1] 14 Aug 2017 13:50:50,079 INFO AnalyticsDynamicServiceConfigListener - --------- Finished applying Analytics Dynamic Service properties from Controller ------------

I am not sure if I missed anything, does anyone have any clue about this? Thanks

Kind Regards,

Denny

[srinirand]

I am facing the same issue with jetty.

Jetty server starts up informing "Started AppDynamics Java Agent Successfully". But the controller does not report the application and also the AppServer Agents are empty in the controller.

For me, even the logs under app agent folder is empty.

[Peter.Holditch]

Does the user-id running jetty have permission to write to the AGENTDIR/v4.x.y.z/logs/ directory?

What happens if you log in as this user and run:

java -javaagent:AGENTDIR/javaagent.jar -Dappdynamics.agent.nodeName=test -Dappdynamics.agent.tierName=test -Dappdynamics.agent.applicationName=test -version

[Peter.Holditch]

Do you see the JVM reporting if you look at the 'Tiers and Nodes' dashboard within your application?

Is there any load running though your application?

Given that you're running in jetty, I assume your load is handled by servlets.  Is that true?

If you see the node and there is load running then it is possible that you need to configure non-standard entry points, this is very likely to be necessary if your application is not implemented using the servlet API

[srinirand · Accepted solution]

Setting 

<node-name>mylocalserver</node-name>

in <java-agent-dir>\ver4.3.5.0\conf\controller-info.xml

resolved my issue and controller started showing load.