failed to attach java agent to the running VM

[jackl3]

I'm following the document to attach java agent to running VM.

But i get java exception while doing it, below is the output for my command:

[root@tropo-102 ~]# ps -ef | grep java | grep -v grep| awk '{print $2}'
10592
[root@tropo-102 ~]#

[root@tropo-102 ~]# /opt/voxeo/prism/jre/bin/java -Xbootclasspath/a:/opt/voxeo/prism/jre/lib/tools.jar -jar /root/appdynamics/javaagent.jar 10592
Attaching to VM [10592]
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:754)
Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
    at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
    at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
    at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
    ... 5 more
Exception in thread "main" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:768)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:754)
Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded
    at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)
    at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)
    at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
    ... 5 more

Any one aware of this issue? my Operation system is centOS 6.7

BR

jack

[jackl3]

BTW, i have change the owern of the agent to my application user in order to elimiate permission problem

[Peter.Holditch]

What JVM version are you using?  The symptoms look similar to those described here.

[jackl3]

Hi, 

   Thanks for your feedback. My JDK is 1.8.0. Looks like the workaround is not work in my issue.

[root@tropo-102 ~]# /opt/voxeo/prism/jre/bin/java -version

java version "1.8.0_101"

Java(TM) SE Runtime Environment (build 1.8.0_101-b13)

Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

[root@tropo-102 ~]# ps -ef | grep java

voxeo    10592 10573  8 Oct20 ?        06:08:12 /opt/voxeo/prism/jre/bin/java -Dwrapper.key=OuYamULl31LbphFW -Dwrapper.port=32000 -Dvoxeo.config-dir=/opt/voxeo/prism/conf -Dproduct-name=prism -Dcom.sun.management.jmxremote -Djava.endorsed.dirs=/opt/voxeo/prism/common/endorsed -Djava.library.path=/opt/voxeo/prism/lib -Dkernel.home=/opt/voxeo/prism -Dcom.voxeolabs.aes.key=/opt/voxeo/prism/conf/aeskey -Dwrapper.native_library=wrapper -Dwrapper.version=3.2.3 -Dlog4j.ignoreTCL=true -Dfile.encoding=utf-8 -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/voxeo/prism/conf/logging.properties -Dorg.tanukisoftware.wrapper.WrapperSimpleApp.waitForStartMain=TRUE -Dsun.net.inetaddr.ttl=60 -Dsun.net.client.defaultReadTimeout=60000 -Dsun.net.client.defaultConnectTimeout=60000 -Djava.rmi.server.RMIClassLoaderSpi=com.micromethod.sipmethod.server.SIPMethodRMIClassLoaderSpi -Djava.rmi.server.useCodebaseOnly=false -Dlog4j.configurationFile=/opt/voxeo/prism/conf/log4j2.xml -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -DAsyncLogger.WaitStrategy=Sleep -DAsyncLogger.RingBufferSize=262144 -Dcom.micromethod.sipmethod.server.cluster.logmc=false -Dh2.bindAddress=127.0.0.1 -Dcom.voxeo.sipmethod.server.launcher.main.block=true -Dnet.sf.ehcache.enableShutdownHook=true -Ddns.defaultQueryTimeout=10000 -Djava.security.egd=file:/dev/./urandom -Dcom.sun.phobos.javascript.useInterpreter=true -Dcom.voxeolabs.moho.suppressEarlyMedia=true -Dcom.voxeolabs.moho.copyHeadersForContinueRouting=false -classpath /opt/voxeo/prism/lib/jars/bcpkix-jdk15on-149b13.jar:/opt/voxeo/prism/lib/jars/bcprov-ext-jdk15on-149b13.jar:/opt/voxeo/prism/lib/jars/common-api.jar:/opt/voxeo/prism/lib/jars/commons-codec-1.4.jar:/opt/voxeo/prism/lib/jars/disruptor-3.3.5.jar:/opt/voxeo/prism/lib/jars/dom4j-1.6.1.jar:/opt/voxeo/prism/lib/jars/kernel-launcher.jar:/opt/voxeo/prism/lib/jars/kernel-utils.jar:/opt/voxeo/prism/lib/jars/log4j-1.2-api-2.6.2.jar:/opt/voxeo/prism/lib/jars/log4j-api-2.6.2.jar:/opt/voxeo/prism/lib/jars/log4j-core-2.6.2.jar:/opt/voxeo/prism/lib/jars/protobuf-java-2.6.0.jar:/opt/voxeo/prism/lib/jars/server-launcher.jar:/opt/voxeo/prism/lib/jars/server-tools.jar:/opt/voxeo/prism/lib/jars/server-utils.jar:/opt/voxeo/prism/lib/jars/servlet-api.jar:/opt/voxeo/prism/lib/jars/tropo-lic.jar:/opt/voxeo/prism/lib/jars/utils-apache.jar:/opt/voxeo/prism/lib/jars/vlib-core-12.2.25.jar:/opt/voxeo/prism/lib/jars/voxeolabs-util.jar:/opt/voxeo/prism/lib/jars/vsm-lib.jar:/opt/voxeo/prism/lib/jars/wrapper.jar:/opt/voxeo/prism/lib/jars/xmppservlet-api.jar:/opt/voxeo/prism/jre/lib/tools.jar:/opt/voxeo/prism/conf -server -Xms2G -Xmx2G -XX:+UseG1GC -XX:NewRatio=3 -XX:SurvivorRatio=8 -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m -Xloggc:/tropo_logs/gclog.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:ErrorFile=/tropo_logs/java_error_pid%p.log -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tropo_logs/java_heap_pid%p.hprof -XX:+PrintAdaptiveSizePolicy -XX:+ParallelRefProcEnabled org.tanukisoftware.wrapper.WrapperSimpleApp com.micromethod.sipmethod.server.Launcher -start

[root@tropo-102 ~]# /opt/voxeo/prism/jre/bin/java -XX:+StartAttachListener -Xbootclasspath/a:/opt/voxeo/prism/jre/lib/tools.jar -jar /root/appdynamics/javaagent.jar 10592

Attaching to VM [10592]

java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:754)

Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)

at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)

at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)

... 5 more

Exception in thread "main" java.lang.RuntimeException: java.lang.reflect.InvocationTargetException

at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:768)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.singularity.ee.agent.appagent.AgentEntryPoint.main(AgentEntryPoint.java:754)

Caused by: com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file: target process not responding or HotSpot VM not loaded

at sun.tools.attach.LinuxVirtualMachine.<init>(LinuxVirtualMachine.java:106)

at sun.tools.attach.LinuxAttachProvider.attachVirtualMachine(LinuxAttachProvider.java:63)

at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)

... 5 more

BR

[Peter.Holditch]

Looking at the above command line, it seems that the java process you are targetting for instrumentation is the Tanuki simple wrapper.

By default, this will wait 2 seconds and then exit, after starting the actual application program which is is wrapping, so I wonder if it has already exitted by the time you run the attach?

I also wonder whether instrumenting the wrapper with AppDynamics is what you want to do?   You are probably better instrumenting the process that the wrapper starts (or amending the wrapper.app.n properties to add in the -javaagent option to start the wrapped app together with the agent as it boots)