Knowledge Base

cancel
Showing results for 
Search instead for 
Did you mean: 

How do I capture thread dumps when the EUM Server Process becomes unresponsive or stuck?

The purpose of this article is to show you how to capture thread dumps on different OS like Windows & Unix when the EUM Server Processor becomes unresponsive or gets hung for some reason.

 

So how do we know at the first place that the EUM Server Processor is indeed hung/Unresponsive?

 

1. If the OS is unix,verify if eum server is running using the command  [ps -ef | grep -i eum] and you should see something like below if the process exist.

 

ps -ef | grep -i eum

APPD 46167 1 5 Nov18 ? 04:27:34 /opt/appdynamics/eum/eum-processor/../jre/bin/java -server -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=50 -XX:+HeapDumpOnOutOfMemoryError -XX:NewRatio=1 -Xms1024m -Xmx4096m -DEUM_COMPONENT=processor -Dlogback.configurationFile=bin/logback.xml -Dcom.mchange.v2.c3p0.cfg.xml=bin/c3p0.xml -Dorg.owasp.esapi.resources=bin -classpath /opt/appdynamics/eum/eum-processor/lib/* com.appdynamics.eumcloud.EUMProcessorServer

 

2. On Windows,verify it using the taskmanager/process explorer to know if the process exist for the eum server or not.

 

3. Then verify using the netstat command as given below to see if the EUM Server Process shows up in LISTEN state

 

   On Unix:

netstat -an | grep 7001

Example:
$ netstat -an | grep 7001

tcp6       0      0 :::7001                 :::*                    LISTEN 

 

 


netstat -an | grep 7002
tcp6       0      0 :::7002                 :::*                    LISTEN

 

  On windows:Use the below command

netstat -an | findstr 7001

netstat -an | findstr 7002 

 

4. Finally, try pinging the eum-server using the below url either from a browser or using curl utility.

 

http://<EUM_SERVER_HOST>:7001/eumaggregator/ping 
http://<EUM_SERVER_HOST>:7001/eumcollector/ping

OR

curl -kv http://<EUM_SERVER_HOST>:7001/eumcollector/ping
For https/SSL

https://<EUM_SERVER_HOST>:7002/eumaggregator/ping
https://<EUM_SERVER_HOST>:7002/eumcollector/ping

OR

curl -kv https://<EUM_SERVER_HOST>:7002/eumcollector/ping

 

5. If you get a "ping" response in the browser window or"200 HTTP status code OK" using curl, it means you are able to connect to your EUM Server successfully and its not hung and if there's no response or browser keeps spinning and eventually the connection gets reset,then it indicates that the eum server process is not responding to your request and hence we need to capture few set of thread dumps( 3 to 5) with a frequency of  30 secs to 60 secs apart,so that we can later compare the successive thread dumps and verify if any threads of the eum server process was hung.

 

 

If your On-Prem EUM Server Processor is running on windows,then below are the available options to capture thread dumps.

 

1. Ctrl+Break key combination:

On Windows, the combination of pressing the Ctrl key and the Break key at the application console (standard input) causes the VM to print a thread dump to the EUM Server's standard output.
Note: This is helpful when your eum processor server is running in the foreground but not in the background.


2. JStack utility:

jstack is available only in JDK . It is not available in the JRE and hence we have to install the JDK in order to use jstack.
jstack prints the stack traces of all Java threads for a given Java process.

jstack [option] pid

Reference:http://download.oracle.com/javase/6/docs/technotes/tools/share/jstack.html


3. Third-Party utility:
http://www.latenighthacking.com/projects/2003/sendSignal/

 
If your On-Prem EUM Server Processor is running on Unix,then below are some of the available options to capture thread dumps.

 

1. kill command

Using the kill command and the output will be stored or redirected to the stdout of the process and usually,it will be nohup.out file.

kill -3 <pid> or kill QUIT <pid>
2. JStack utility:

jstack is available only in JDK . It is not available in the JRE and hence we have to install the JDK in order to use jstack.
jstack prints the stack traces of all Java threads for a given Java process.

jstack [option] pid or
jstack <pid> > threadump1.txt [Redirecting the output to threadump1.txt]

Reference:http://download.oracle.com/javase/6/docs/technotes/tools/share/jstack.html


 

 

 Thread Dump Location:

 

1. If the thread dump was collected using kill -3 <pid>,then the dump would be in stdout as mentioned earlier

 

For example:

 

/home/appdynamics/EUM/eum-processor/bin/nohup.out

 

2. if the thread dump was colledted using Jstack <pid> > output.txt,then the output.txt will contain the necessary dump.

 

Along with the thread dumps collected, You need to upload the EUM Processor logs and its properties file to your support case for the AppD Support team to review and find out the reason for the unresponsiveness.

 

Log Location: 

/home/appdynamics/EUM/logs/eum-processor.log

 

Properties Location:

/home/appdynamics/EUM/eum-processor/bin/eum.properties

 

  

Version history
Revision #:
6 of 6
Last update:
‎09-05-2018 04:09 PM
Updated by:
 
Labels (1)
Tags (2)
0 Kudos