Java (Java Agent, Installation, JVM, and Controller Installation)

cancel
Showing results for 
Search instead for 
Did you mean: 

Machine agent stopping without any error after starting successfully.

Adventurer

Machine agent stopping without any error after starting successfully.

Machine agent stopping without any error after starting successfully.

 

Why, any idea??

 

Below is the log file.

 

 

[main] 22 Mar 2017 14:25:22,170 INFO SystemAgent - ####################################################################################
[main] 22 Mar 2017 14:25:22,171 INFO SystemAgent - Agent Install Directory [/opt/appdynamics/MA]
[main] 22 Mar 2017 14:25:22,171 INFO SystemAgent - Using Agent Version [Machine Agent v4.0.0.0 GA Build Date 2014-12-11 23:35:01]
[main] 22 Mar 2017 14:25:22,172 INFO SystemAgent - JVM Runtime:
java.home=/opt/jdk1.7.0_79/jre
java.vm.vendor=Oracle Corporation
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.runtime.version=1.7.0_79-b15
java.io.tmpdir=/tmp
[main] 22 Mar 2017 14:25:22,172 INFO SystemAgent - OS Runtime:
os.name=Linux
os.arch=amd64
os.version=3.10.0-514.10.2.el7.x86_64
user.name=root
user.home=/root
user.dir=/opt/appdynamics/MA
[main] 22 Mar 2017 14:25:22,174 INFO SystemAgent - JVM Args :
[main] 22 Mar 2017 14:25:22,182 INFO SystemAgent - JVM Runtime Name: 28059@575234-www2
[main] 22 Mar 2017 14:25:22,182 INFO SystemAgent - JVM PID: 28059
[main] 22 Mar 2017 14:25:22,182 INFO SystemAgent - Machine Agent is resolving bootstrap info....
[main] 22 Mar 2017 14:25:22,324 INFO SystemAgent - Orchestration is disabled - disabling virtualization resolvers by default.
[main] 22 Mar 2017 14:25:22,331 INFO SystemAgent - Full Agent Registration Info Resolver found system property [appdynamics.agent.create.agent.info.if.missing] for appdynamics.agent.create.agent.info.if.missing [f alse]
[main] 22 Mar 2017 14:25:22,340 INFO SystemAgent - Default Host Identifier Resolver using host name for unique host identifier [575234-www2]
[main] 22 Mar 2017 14:25:22,343 INFO SystemAgent - Default IP Address Resolver found IP addresses [[fe80:0:0:0:250:56ff:feb1:1897%3, 10.241.120.175, fe80:0:0:0:250:56ff:feb1:28c2%2, 172.21.255.12]]
[main] 22 Mar 2017 14:25:22,347 INFO SystemAgent - Full Agent Registration Info Resolver using selfService [false]
[main] 22 Mar 2017 14:25:22,347 INFO SystemAgent - Full Agent Registration Info Resolver using application name [www2-hal]
[main] 22 Mar 2017 14:25:22,347 INFO SystemAgent - Full Agent Registration Info Resolver using tier name [null]
[main] 22 Mar 2017 14:25:22,347 INFO SystemAgent - Full Agent Registration Info Resolver using node name [www2-hal]
[main] 22 Mar 2017 14:25:22,356 INFO SystemAgent - XML Controller Info Resolver found controller host [ha.saas.appdynamics.com]
[main] 22 Mar 2017 14:25:22,356 INFO SystemAgent - XML Controller Info Resolver found controller port [443]
[main] 22 Mar 2017 14:25:22,357 INFO SystemAgent - XML Controller Info Resolver found valid config [Controller Host: ha.saas.appdynamics.com; Controller Port: 443; ssl-enabled: true]
[main] 22 Mar 2017 14:25:22,361 INFO SystemAgent - XML Agent Account Info Resolver using account name [HA]
[main] 22 Mar 2017 14:25:22,361 INFO SystemAgent - XML Agent Account Info Resolver using account access key [e4698bb4427d]
[main] 22 Mar 2017 14:25:22,363 INFO SystemAgent - Keystore file /opt/appdynamics/MA/conf/cacerts.jks was not found
[main] 22 Mar 2017 14:25:22,553 INFO SystemAgent - Trying secure protocol:TLSv1.2
[main] 22 Mar 2017 14:25:22,638 INFO SystemAgent - Machine Agent resolved bootstrap info!
[main] 22 Mar 2017 14:25:22,638 INFO SystemAgent - Creating machine agent scheduler, pool size: 2
[main] 22 Mar 2017 14:25:22,651 INFO SystemAgent - Creating machine agent monitor scheduler, pool size: 4
[main] 22 Mar 2017 14:25:22,655 INFO SystemAgent - Started Agent Schedulers
[main] 22 Mar 2017 14:25:22,655 INFO SystemAgent - Scheduling Machine Agent Registration ....
[Agent-Scheduler-1] 22 Mar 2017 14:25:22,664 INFO RegistrationChannel - using controller host [ha.saas.appdynamics.com]; controller port [443]
[Agent-Scheduler-1] 22 Mar 2017 14:25:22,677 INFO RegistrationChannel - setting unique host information Host Name [575234-www2]
[Agent-Scheduler-1] 22 Mar 2017 14:25:22,677 INFO RegistrationChannel - setting agent version [Machine Agent v4.0.0.0 GA Build Date 2014-12-11 23:35:01]
[Agent-Scheduler-1] 22 Mar 2017 14:25:22,678 INFO RegistrationChannel - setting agent install dir [/opt/appdynamics/MA]
[Agent-Scheduler-1] 22 Mar 2017 14:25:22,678 INFO RegistrationChannel - setting agent type [MACHINE_AGENT]
[Agent-Scheduler-1] 22 Mar 2017 14:25:22,678 INFO RegistrationChannel - setting agent application [www2-hal]
[Agent-Scheduler-1] 22 Mar 2017 14:25:22,678 INFO RegistrationChannel - setting agent node name [www2-hal]
[Agent-Scheduler-1] 22 Mar 2017 14:25:22,678 INFO RegistrationChannel - Sending Registration request
[Agent-Scheduler-1] 22 Mar 2017 14:25:23,291 INFO SystemAgent - Full certificate chain validation performed using default certificate file
[Agent-Scheduler-1] 22 Mar 2017 14:25:23,828 INFO SystemAgent - Registered Machine Agent with machine ID [268364]
[Agent-Scheduler-1] 22 Mar 2017 14:25:23,828 INFO SystemAgent - Starting Machine Agent....
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,016 INFO ManagedMonitorDelegate - Started Agent Metric Generation Service
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,038 INFO ManagedMonitorDelegate - Started Agent Env Properties Service
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,039 INFO ManagedMonitorDelegate - Scheduled Continuous Task Monitor with frequency [30000]ms
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,047 INFO MonitorConfigReader - Reading monitor config file:/opt/appdynamics/MA/monitors/JavaHardwareMonitor/monitor.xml
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,053 INFO MonitorConfigReader - os name [linux] version [3.10.0-514.10.2.el7.x86_64]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,063 INFO NodeMonitorManager - Initializing Managed Monitor [JavaHardwareMonitor]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,064 INFO ManagedMonitorDelegate - Initializing managed monitor [SigarHardwareMonitor]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,064 INFO ManagedMonitorDelegate - Executing managed monitor [SigarHardwareMonitor], task name [null]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,064 INFO ManagedMonitorDelegate - Task [null] for monitor [SigarHardwareMonitor] is SCHEDULED
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,068 INFO JavaTaskCreator - Task classpath not specified,System classpath will be used.
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,133 WARN SigarConfig - No task file found. Will use default configuration. /opt/appdynamics/MA/monitors/JavaHardwareMonitor/task.xml
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,133 INFO SigarDelegate - Operating system is supported [Linux]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,660 INFO SigarAppAgentMonitor - Full Agent Registration Info Resolver using selfService [true]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,661 INFO SigarAppAgentMonitor - Full Agent Registration Info Resolver using application name [HAL New Production]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,661 INFO SigarAppAgentMonitor - Full Agent Registration Info Resolver using tier name [Web Tier]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,661 INFO SigarAppAgentMonitor - Full Agent Registration Info Resolver using node name [www2-hal]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,694 INFO MonitorConfigReader - Reading monitor config file:/opt/appdynamics/MA/monitors/analytics-agent/monitor.xml
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,697 INFO MonitorConfigReader - os name [linux] version [3.10.0-514.10.2.el7.x86_64]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,698 INFO NodeMonitorManager - Initializing Managed Monitor [analytics-agent]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,698 INFO ManagedMonitorDelegate - Not initializing managed monitor [AppDynamics Analytics Agent] - Disabled in configuration file.
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,699 INFO MonitorConfigReader - Reading monitor config file:/opt/appdynamics/MA/monitors/HardwareMonitor/monitor.xml
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,702 INFO MonitorConfigReader - os name [linux] version [3.10.0-514.10.2.el7.x86_64]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,704 INFO NodeMonitorManager - Initializing Managed Monitor [HardwareMonitor]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,704 INFO ManagedMonitorDelegate - Not initializing managed monitor [HardwareMonitor] - Disabled in configuration file.
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,704 INFO NodeMonitorManager - No unmanaged Monitors installed for this Agent.
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,704 INFO AgentMonitorManager - Initialized System Monitor Manager with directory [/opt/appdynamics/MA/monitors]
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,704 INFO SystemAgent - Set up agent monitor manager
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,709 INFO SystemAgent - Orchestration is disabled - disabling one-way agent transport. The agent will not be able to execute workflow tasks.
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,710 INFO SystemAgent - Set up agent re-registration task
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,714 INFO SystemAgentConfigManager - Scheduling configuration refresh at an interval of 60 seconds
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,715 INFO SystemAgentConfigManager - Configuration refresh task interval is 60 seconds
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,715 INFO SystemAgent - Configuration manager successfully configured
[Agent-Scheduler-1] 22 Mar 2017 14:25:24,717 INFO SystemAgent - Started AppDynamics Machine Agent Successfully.
[Thread-1] 22 Mar 2017 14:30:03,001 INFO SystemAgent - Stopping Machine Agent....
[Thread-1] 22 Mar 2017 14:30:03,001 INFO SystemAgent - Shutting down scheduler
[Thread-1] 22 Mar 2017 14:30:03,002 INFO SystemAgent - Stopped scheduler
[Thread-1] 22 Mar 2017 14:30:03,002 INFO SystemAgent - Shutting down agent scheduler manager
[Thread-1] 22 Mar 2017 14:30:03,003 INFO SystemAgent - Shutting down agent scheduler manager
[Thread-1] 22 Mar 2017 14:30:03,003 INFO SystemAgent - Shutting down agent monitor manager
[Thread-1] 22 Mar 2017 14:30:03,003 INFO ManagedMonitorDelegate - shutdown metric generation service.
[Thread-1] 22 Mar 2017 14:30:03,003 INFO SystemAgent - Stopped agent monitor manager
[Thread-1] 22 Mar 2017 14:30:03,003 INFO SystemAgent - Machine Agent Stopped!

Machine agent stopping without any error after starting successfully.
4 REPLIES 4
Architect

Re: Machine agent stopping without any error after starting successfully.

Do you see anything in the nohup.out file that is generated?

Re: Machine agent stopping without any error after starting successfully.

Hi,

 

We are seeing something similar after upgrading machine agents from 4.1.7.1 to 4.2.15.0. Our systems are AIX 7.1 and we have not yet identified what is happening. One thing you could do is to enable debug logging for the machine agent with below instructions. Of course you could let the agent run until it stops unexpectedly instead of only 5 minutes.

 

We are going to try this to see if we find anything and I'll update here if we find the cause. We are starting the agents with the same command that was used with v4.1 agent and that might cause some issues because we had set some maximum Java heap size in the command.

 

Steps for enabling machine agent debug mode:

  • in log4j.xml file located in <MachineAgent-installation-dir>\conf\logging
  • change the below logger to "debug"

<logger name="com.singularity" additivity="false"> 
<level value="debug"/> 
<appender-ref ref="FileAppender"/> 
</logger>

  • allow the machine agent to run for at-least 5 minutes
  • zip the entire folder after 5 minutes of starting the machine agent- "<MachineAgent-installation-dir>\logs" and attach to this ticket
  • Revert the logging level to "info" in log4j.xml

 

-Antti

Highlighted

Re: Machine agent stopping without any error after starting successfully.

The sigar library that we use to get the OS related metrics is not supported on AIX 7.1.

 

So as a workaround can you try this.

 

In the MachineAgent/lib folder create a symbolic link for AIX 5 version of the library to work as AIX 7 version as well. Ensure same file permissions.
ln -s libsigar-ppc64-aix-5.so libsigar-ppc64-aix-7.so

 

Another option:

 

  1. Stop the Machine Agent

  2. Switch to Hardware Monitor. Here are the steps to switch to Hardware monitor:

a. Edit monitor.xml from <machine_agent_install_directory>/monitors/HardwareMonitor/
b. Change <enabled>false</enabled> to <enabled>true</enabled>
c. Save the file
d. Edit monitor.xml from <machine_agent_install_directory>/monitors/JavaHardwareMonitor/
e. Change <enabled>true</enabled> to <enabled>false</enabled>
f. Save the file

 

   3. Restart Machine Agent

Re: Machine agent stopping without any error after starting successfully.

Hi,

 

Actually I'm not sure about the AIX level currently but I think they are 7.1.

 

Anyway, the problem was very simple - we forgot to use "nohup" in front of the command when starting from shell without using the machine-agent script. Normally these agents are started from inittab during system reboot and we just copied the command from there. The working form is:

"nohup <commandFromInittab> &"

 

Without nohup the agents stop when the user logs off. This wasn't needed on all servers as some servers have probably the WAS user always logged on and that's why I was thinking there is something else wrong now.

 

-Antti