The purpose of this article is to show you how to capture thread dumps on different OS like Windows & Unix when the EUM Server Processor becomes unresponsive or gets hung for some reason.
So how do we know at the first place that the EUM Server Processor is indeed hung/Unresponsive?
1. If the OS is unix,verify if eum server is running using the command [ps -ef | grep -i eum] and you should see something like below if the process exist.
ps -ef | grep -i eum
APPD 46167 1 5 Nov18 ? 04:27:34 /opt/appdynamics/eum/eum-processor/../jre/bin/java -server -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=50 -XX:+HeapDumpOnOutOfMemoryError -XX:NewRatio=1 -Xms1024m -Xmx4096m -DEUM_COMPONENT=processor -Dlogback.configurationFile=bin/logback.xml -Dcom.mchange.v2.c3p0.cfg.xml=bin/c3p0.xml -Dorg.owasp.esapi.resources=bin -classpath /opt/appdynamics/eum/eum-processor/lib/* com.appdynamics.eumcloud.EUMProcessorServer
2. On Windows,verify it using the taskmanager/process explorer to know if the process exist for the eum server or not.
3. Then verify using the netstat command as given below to see if the EUM Server Process shows up in LISTEN state
On Unix:
netstat -an | grep 7001
Example:
$ netstat -an | grep 7001
tcp6 0 0 :::7001 :::* LISTEN
netstat -an | grep 7002
tcp6 0 0 :::7002 :::* LISTEN
On windows:Use the below command
netstat -an | findstr 7001
netstat -an | findstr 7002
4. Finally, try pinging the eum-server using the below url either from a browser or using curl utility.
http://<EUM_SERVER_HOST>:7001/eumaggregator/ping
http://<EUM_SERVER_HOST>:7001/eumcollector/ping
OR
curl -kv http://<EUM_SERVER_HOST>:7001/eumcollector/ping
For https/SSL
https://<EUM_SERVER_HOST>:7002/eumaggregator/ping
https://<EUM_SERVER_HOST>:7002/eumcollector/ping
OR
curl -kv https://<EUM_SERVER_HOST>:7002/eumcollector/ping
5. If you get a "ping" response in the browser window or"200 HTTP status code OK" using curl, it means you are able to connect to your EUM Server successfully and its not hung and if there's no response or browser keeps spinning and eventually the connection gets reset,then it indicates that the eum server process is not responding to your request and hence we need to capture few set of thread dumps( 3 to 5) with a frequency of 30 secs to 60 secs apart,so that we can later compare the successive thread dumps and verify if any threads of the eum server process was hung.
If your On-Prem EUM Server Processor is running on windows,then below are the available options to capture thread dumps.
1. Ctrl+Break key combination:
On Windows, the combination of pressing the Ctrl key and the Break key at the application console (standard input) causes the VM to print a thread dump to the EUM Server's standard output.
Note: This is helpful when your eum processor server is running in the foreground but not in the background.
2. JStack utility:
jstack is available only in JDK . It is not available in the JRE and hence we have to install the JDK in order to use jstack.
jstack prints the stack traces of all Java threads for a given Java process.
jstack [option] pid
Reference:http://download.oracle.com/javase/6/docs/technotes/tools/share/jstack.html
3. Third-Party utility:
http://www.latenighthacking.com/projects/2003/sendSignal/
If your On-Prem EUM Server Processor is running on Unix,then below are some of the available options to capture thread dumps.
1. kill command
Using the kill command and the output will be stored or redirected to the stdout of the process and usually,it will be nohup.out file.
kill -3 <pid> or kill QUIT <pid>
2. JStack utility:
jstack is available only in JDK . It is not available in the JRE and hence we have to install the JDK in order to use jstack.
jstack prints the stack traces of all Java threads for a given Java process.
jstack [option] pid or
jstack <pid> > threadump1.txt [Redirecting the output to threadump1.txt]
Reference:http://download.oracle.com/javase/6/docs/technotes/tools/share/jstack.html
Thread Dump Location:
1. If the thread dump was collected using kill -3 <pid>,then the dump would be in stdout as mentioned earlier
For example:
/home/appdynamics/EUM/eum-processor/bin/nohup.out
2. if the thread dump was colledted using Jstack <pid> > output.txt,then the output.txt will contain the necessary dump.
Along with the thread dumps collected, You need to upload the EUM Processor logs and its properties file to your support case for the AppD Support team to review and find out the reason for the unresponsiveness.
Log Location:
/home/appdynamics/EUM/logs/eum-processor.log
Properties Location:
/home/appdynamics/EUM/eum-processor/bin/eum.properties
