Java (Java Agent, Installation, JVM, and Controller Installation)

cancel
Showing results for 
Search instead for 
Did you mean: 

Agents unable to send data to the server

A.A
Wanderer

Agents unable to send data to the server

Similar to what is described at https://community.appdynamics.com/t5/Java-Java-Agent-Installation-JVM/Java-agent-issue-Metric-Report..., neither machine agent nor app agent (Java) is able to send data to the AD server. There is no activity in the logs until the local buffer is full and got the error [AD Thread-Metric Reporter1] 04 Jun 2018 14:48:47,998 ERROR ManagedMonitorDelegate - Error sending metrics - will requeue for later transmission com.singularity.ee.agent.commonservices.metricgeneration.metrics.MetricSendException: Connection back off limitation in effect: /controller/instance/97149/metrics at com.singularity.ee.agent.commonservices.metricgeneration.AMetricSubscriber.publish(AMetricSubscriber.java:345) at com.singularity.ee.agent.commonservices.metricgeneration.MetricReporter.run(MetricReporter.java:111) at com.singularity.ee.util.javaspecific.scheduler.AgentScheduledExecutorServiceImpl$SafeRunnable.run(AgentScheduledExecutorServiceImpl.java:122) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask$Sync.innerRunAndReset(ADFutureTask.java:335) at com.singularity.ee.util.javaspecific.scheduler.ADFutureTask.runAndReset(ADFutureTask.java:152) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.access$101(ADScheduledThreadPoolExecutor.java:119) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.runPeriodic(ADScheduledThreadPoolExecutor.java:206) at com.singularity.ee.util.javaspecific.scheduler.ADScheduledThreadPoolExecutor$ADScheduledFutureTask.run(ADScheduledThreadPoolExecutor.java:236) at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.runTask(ADThreadPoolExecutor.java:694) at com.singularity.ee.util.javaspecific.scheduler.ADThreadPoolExecutor$Worker.run(ADThreadPoolExecutor.java:726) at java.lang.Thread.run(Thread.java:745) [AD Thread-Metric Reporter0] 04 Jun 2018 14:48:57,998 WARN ManagedMonitorDelegate - Metric Reporter Queue full. Dropping metrics. The fact is the server is reachable by telnet, nc or any other network tool: $ telnet xx.xx.xx.xx 80 Trying xx.xx.xx.xx... Connected to xx.xx.xx.xx. Escape character is '^]'. get /? Error report

HTTP Status 501 - Method get is not defined in RFC 2068 and is not supported by the Servlet API


type Status report

messageMethod get is not defined in RFC 2068 and is not supported by the Servlet API

descriptionThe server does not support the functionality needed to fulfill this request.


Connection closed by foreign host and same for the port configured in the controller-info.xml. Inspecting the traffic see that every minute got a "HTTP/1.1 204 No content\r\n" from the server, right after a POST /controller/sim/v2/agent/machines/xxxx/metadata HTTP/1.1 the agent is sending to. Any hint/recommendation would be appreciated. Thanks in advance.
Agents unable to send data to the server
9 REPLIES 9
Moderator

Re: Agents unable to send data to the server

Can you please attach the complete set of logs from the agent logs directory?

 

I suspect there is a problem with the agent/controller communication, which would be reported early on.



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
A.A
Wanderer

Re: Agents unable to send data to the server

Hi,

thanks for your reply.

Please note since the platform does not allow to upload certain extensions, I have upload them to a public service. The link is https://probeta.net/m/d/m_0607090133526_machine-agent.log

Best regards.

Moderator

Re: Agents unable to send data to the server

 I hadn't realised you were talking about a machine agent here, but there is defintely some problem with the agent connecting to the controller:

[AD Thread Pool-Global0] 31 May 2018 10:06:05,694 ERROR SystemAgentPollingForUpdate - Fatal transport error while connecting to URL [/controller/instance/97149/systemagentpolling]: org.apache.http.conn.ConnectTimeoutException: Connect to xx.xx.xx.xx:yyyy [/xx.xx.xx.xx] failed: connect timed out

What version of the agent and controller do you have running? 



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
A.A
Wanderer

Re: Agents unable to send data to the server

I'm running 4.3.7.5-1, but also tried with 4.2.3.1-1 with same result.

The server runs 4.3.8.6.

Moderator

Re: Agents unable to send data to the server

OK, could you please restart the machine agent and provide the log for the first couple of minutes of activity, along with the log from the controller.



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
A.A
Wanderer

Re: Agents unable to send data to the server

Hi,

I've restarted the agent, but unfortunately I don't have access to the controller, so I can not provide these logs; I'll try to get them.

In the mean time, please find below the logs from the agent once it has been restarted:

[main] 07 Jun 2018 11:53:25,733  INFO FlexibleX509TrustManager - Using default keystore for SSL certificate validation.
[main] 07 Jun 2018 11:53:25,971  INFO HostIdProvider - Default Host Identifier Resolver using host name for unique host identifier [myserver.example.com]
[ConfigExecutor-0] 07 Jun 2018 11:53:26,271  INFO SecondStageFeaturesListener - Dynamic monitoring feature is disabled in configuration, hence machine agent sends all metrics irrespective of the monitoring mode requested
[system-thread-0] 07 Jun 2018 11:53:26,330  INFO SecondStageSystem - Starting main system with features Features(features=[basic], reason=Features.Reason(message=, code=))
[system-thread-0] 07 Jun 2018 11:53:26,525  INFO SystemAgent - ####################################################################################
[system-thread-0] 07 Jun 2018 11:53:26,525  INFO SystemAgent - Agent Install Directory [/opt/appdynamics/machine-agent]
[system-thread-0] 07 Jun 2018 11:53:26,525  INFO SystemAgent - Using Agent Version [Machine Agent v4.3.7.5 GA Build Date 2018-02-21 12:19:54]
[system-thread-0] 07 Jun 2018 11:53:26,525  INFO SystemAgent - JVM Runtime:
	java.home=/opt/appdynamics/machine-agent/jre
	java.vm.vendor=Oracle Corporation
	java.vm.name=Java HotSpot(TM) 64-Bit Server VM
	java.runtime.version=1.8.0_111-b14
	java.io.tmpdir=/tmp
	user.language=en
	user.country=US
	user.variant=
	Default locale=en_US
[system-thread-0] 07 Jun 2018 11:53:26,525  INFO SystemAgent - OS Runtime:
	os.name=Linux
	os.arch=amd64
	os.version=3.10.0-693.17.1.el7.x86_64
	user.name=root
	user.home=/root
	user.dir=/opt/appdynamics/machine-agent
[system-thread-0] 07 Jun 2018 11:53:26,527  INFO SystemAgent - JVM Args : -Xms4g | -Xmx4g | -Dlog4j.configuration=file:/opt/appdynamics/machine-agent/conf/logging/log4j.xml |
[system-thread-0] 07 Jun 2018 11:53:26,527  INFO SystemAgent - JVM Runtime Name: 122658@myserver.example.com
[system-thread-0] 07 Jun 2018 11:53:26,527  INFO SystemAgent - JVM PID: 122658
[system-thread-0] 07 Jun 2018 11:53:26,528  INFO SystemAgent - Machine Agent is resolving bootstrap info....
[system-thread-0] 07 Jun 2018 11:53:26,645  INFO SystemAgent - Orchestration is disabled - disabling virtualization resolvers by default.
[system-thread-0] 07 Jun 2018 11:53:26,648  INFO SystemAgent - Full Agent Registration Info Resolver found system property [appdynamics.agent.create.agent.info.if.missing] for appdynamics.agent.create.agent.info.if.missing [false]
[system-thread-0] 07 Jun 2018 11:53:26,659  INFO SystemAgent - Default Host Identifier Resolver using host name for unique host identifier [myserver.example.com]
[system-thread-0] 07 Jun 2018 11:53:26,662  INFO SystemAgent - Default IP Address Resolver found IP addresses [[0123:4567:89ab:cdef:0123:4567:89ab:cdef%eth0, zz.zz.zz.zz]]
[system-thread-0] 07 Jun 2018 11:53:26,667  INFO SystemAgent - Full Agent Registration Info Resolver using selfService [false]
[system-thread-0] 07 Jun 2018 11:53:26,667  INFO SystemAgent - Full Agent Registration Info Resolver using application name [null]
[system-thread-0] 07 Jun 2018 11:53:26,667  INFO SystemAgent - Full Agent Registration Info Resolver using tier name [null]
[system-thread-0] 07 Jun 2018 11:53:26,667  INFO SystemAgent - Full Agent Registration Info Resolver using node name [null]
[system-thread-0] 07 Jun 2018 11:53:26,671  INFO SystemAgent - XML Controller Info Resolver found controller host [xx.xx.xx.xx]
[system-thread-0] 07 Jun 2018 11:53:26,672  INFO SystemAgent - XML Controller Info Resolver found controller port [yyyy]
[system-thread-0] 07 Jun 2018 11:53:26,676  INFO SystemAgent - XML Agent Account Info Resolver using account name [customer1]
[system-thread-0] 07 Jun 2018 11:53:26,676  INFO SystemAgent - XML Agent Account Info Resolver using account access key [SJ5b2m7d1$354]
[system-thread-0] 07 Jun 2018 11:53:26,679  INFO SystemAgent - Keystore file /opt/appdynamics/machine-agent/conf/cacerts.jks was not found
[system-thread-0] 07 Jun 2018 11:53:26,688  INFO SystemAgent - Machine Agent resolved bootstrap info!
[system-thread-0] 07 Jun 2018 11:53:26,689  INFO SystemAgent - Creating machine agent scheduler, pool size: 2
[system-thread-0] 07 Jun 2018 11:53:26,697  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4
[system-thread-0] 07 Jun 2018 11:53:26,700  INFO SystemAgent - Started Agent Schedulers
[ConfigExecutor-0] 07 Jun 2018 11:53:26,702  INFO DefaultLegacyAgentRegistrationStateManager - Registered machine with machine ID [Optional.of(97149)]
[ConfigExecutor-0] 07 Jun 2018 11:53:26,702  INFO DefaultLegacyAgentRegistrationStateManager - Scheduling System-Agent start...
[system-thread-0] 07 Jun 2018 11:53:26,706  INFO SimAgentRepetitiveLoggingModule - The turnover time for the SIM agent repetitive logger is 5 minutes
[system-thread-0] 07 Jun 2018 11:53:26,706  INFO SimAgentRepetitiveLoggingModule - The cache size for the SIM agent repetitive logger is 1000
[system-thread-0] 07 Jun 2018 11:53:26,707  INFO DefaultLegacyAgentRegistrationStateManager - Starting machine agent...
[system-thread-0] 07 Jun 2018 11:53:26,708  INFO SystemAgent - Starting Machine Agent....
[system-thread-0] 07 Jun 2018 11:53:26,709  INFO ControllerTimeSkewHandler - Skew Handler is : [enabled].
[system-thread-0] 07 Jun 2018 11:53:26,807  INFO ManagedMonitorDelegate - Started Agent Metric Generation Service
[system-thread-0] 07 Jun 2018 11:53:26,815  INFO ManagedMonitorDelegate - Event Service is : [enabled].
[system-thread-0] 07 Jun 2018 11:53:26,844  INFO ManagedMonitorDelegate - Metric Service is : [enabled].
[system-thread-0] 07 Jun 2018 11:53:26,860  INFO ManagedMonitorDelegate - Started Agent Env Properties Service
[system-thread-0] 07 Jun 2018 11:53:26,863  INFO ManagedMonitorDelegate - Scheduled Continuous Task Monitor with frequency [30000]ms
[system-thread-0] 07 Jun 2018 11:53:26,877  INFO MonitorConfigReader - Reading monitor config file:/opt/appdynamics/machine-agent/monitors/HardwareMonitor/monitor.xml
[system-thread-0] 07 Jun 2018 11:53:26,894  INFO MonitorConfigReader - os name [linux] version [3.10.0-693.17.1.el7.x86_64]
[system-thread-0] 07 Jun 2018 11:53:26,903  INFO NodeMonitorManager - Initializing managed monitor [HardwareMonitor]
[system-thread-0] 07 Jun 2018 11:53:26,903  INFO ManagedMonitorDelegate - Not initializing managed monitor [HardwareMonitor] - Disabled in configuration file.
[system-thread-0] 07 Jun 2018 11:53:26,904  INFO MonitorConfigReader - Reading monitor config file:/opt/appdynamics/machine-agent/monitors/JavaHardwareMonitor/monitor.xml
[system-thread-0] 07 Jun 2018 11:53:26,907  INFO MonitorConfigReader - os name [linux] version [3.10.0-693.17.1.el7.x86_64]
[system-thread-0] 07 Jun 2018 11:53:26,907  INFO MonitorConfigReader - Overriding enabled tag. new value = false
[system-thread-0] 07 Jun 2018 11:53:26,908  INFO NodeMonitorManager - Initializing managed monitor [JavaHardwareMonitor]
[system-thread-0] 07 Jun 2018 11:53:26,908  INFO ManagedMonitorDelegate - Not initializing managed monitor [SigarHardwareMonitor] - Disabled in configuration file.
[system-thread-0] 07 Jun 2018 11:53:26,908  INFO MonitorConfigReader - Reading monitor config file:/opt/appdynamics/machine-agent/monitors/analytics-agent/monitor.xml
[system-thread-0] 07 Jun 2018 11:53:26,911  INFO MonitorConfigReader - os name [linux] version [3.10.0-693.17.1.el7.x86_64]
[system-thread-0] 07 Jun 2018 11:53:26,912  INFO NodeMonitorManager - Initializing managed monitor [analytics-agent]
[system-thread-0] 07 Jun 2018 11:53:26,912  INFO ManagedMonitorDelegate - Not initializing managed monitor [AppDynamics Analytics Agent] - Disabled in configuration file.
[system-thread-0] 07 Jun 2018 11:53:26,912  INFO NodeMonitorManager - Directory [/opt/appdynamics/machine-agent/monitors/unmanaged] not found, continuing.
[system-thread-0] 07 Jun 2018 11:53:26,912  INFO AgentMonitorManager - Initialized System Monitor Manager with directory [/opt/appdynamics/machine-agent/monitors]
[system-thread-0] 07 Jun 2018 11:53:26,912  INFO SystemAgent - Set up agent monitor manager
[system-thread-0] 07 Jun 2018 11:53:26,913  INFO SystemAgent - Orchestration is disabled - disabling one-way agent transport. The agent will not be able to execute workflow tasks.
[system-thread-0] 07 Jun 2018 11:53:26,918  INFO SystemAgentConfigManager - Scheduling configuration refresh at an interval of 60 seconds
[system-thread-0] 07 Jun 2018 11:53:26,919  INFO SystemAgentConfigManager - Configuration refresh task interval is 60 seconds
[system-thread-0] 07 Jun 2018 11:53:26,919  INFO SystemAgent - Configuration manager successfully configured
[system-thread-0] 07 Jun 2018 11:53:26,921  INFO RunbookHandler - Runbook Operation Execution is : [enabled].
[system-thread-0] 07 Jun 2018 11:53:26,921  INFO SystemAgent - Started AppDynamics Machine Agent Successfully.
[ExtensionStarter-AgentServer] 07 Jun 2018 11:53:56,830  INFO SystemAgent - Creating machine agent scheduler, pool size: 2
[ExtensionStarter-AgentServer] 07 Jun 2018 11:53:56,830  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4
[ExtensionStarter-AgentServer] 07 Jun 2018 11:53:56,830  INFO SystemAgent - Started Agent Schedulers
[ExtensionStarter-ServerMonitoring] 07 Jun 2018 11:53:57,396  INFO SystemAgent - Creating machine agent scheduler, pool size: 2
[ExtensionStarter-ServerMonitoring] 07 Jun 2018 11:53:57,397  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4
[ExtensionStarter-ServerMonitoring] 07 Jun 2018 11:53:57,397  INFO SystemAgent - Started Agent Schedulers
[ExtensionStarter-DockerMonitoring] 07 Jun 2018 11:53:57,633  INFO SystemAgent - Creating machine agent scheduler, pool size: 2
[ExtensionStarter-DockerMonitoring] 07 Jun 2018 11:53:57,634  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4
[ExtensionStarter-DockerMonitoring] 07 Jun 2018 11:53:57,634  INFO SystemAgent - Started Agent Schedulers
[ExtensionStarter-DockerMonitoring] 07 Jun 2018 11:53:57,643  INFO DockerMonitoringModule - Initializing Executor Service for Docker Metric Collection, pool size:  3
[ExtensionStarter-DockerMonitoring] 07 Jun 2018 11:53:57,644  INFO DockerMonitor - Not starting docker monitoring extension because it is disabled.
[ExtensionStarter-CrashGuard] 07 Jun 2018 11:53:57,677  INFO SystemAgent - Creating machine agent scheduler, pool size: 2
[ExtensionStarter-CrashGuard] 07 Jun 2018 11:53:57,677  INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4
[ExtensionStarter-CrashGuard] 07 Jun 2018 11:53:57,677  INFO SystemAgent - Started Agent Schedulers
[ConfigExecutor-0] 07 Jun 2018 11:53:57,714  INFO CrashGuardRunner - Not starting crash guard extension because it is disabled.To enable, please update the configuration enabled in CrashGuardConfig.yml.
Moderator

Re: Agents unable to send data to the server

It looks to me that the agent has connected to the controller:

[ConfigExecutor-0] 07 Jun 2018 11:53:26,702  INFO DefaultLegacyAgentRegistrationStateManager - Registered machine with machine ID [Optional.of(97149)]
[ConfigExecutor-0] 07 Jun 2018 11:53:26,702  INFO DefaultLegacyAgentRegistrationStateManager - Scheduling System-Agent start...   

 Do you see it there?  I see no configuration of app, etc.so it will likely only appear in the agents view

 



Found something helpful? Click the Accept as Solution button to help others find answers faster.
Liked something? Click the Thumbs Up button.
A.A
Wanderer

Re: Agents unable to send data to the server

No, I can't see them in the view, and logs keep quiet.

A.A
Wanderer

Re: Agents unable to send data to the server

I think there is a networking issue:

As I've said before, we can reach the server by using standard network tools:

$ telnet xx.xx.xx.xx yyyy
Trying xx.xx.xx.xx...
Connected to xx.xx.xx.xx.
Escape character is '^]'.

But trying to send traffic to the controller, no packets were received:

echo "asfas" | nc -v xx.xx.xx.xx yyyy
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to xx.xx.xx.xx:yyyy.
Ncat: 6 bytes sent, 0 bytes received in 0.08 seconds.