Appdynamic agent causes java crash on Weblogic server

[Erik.Pahlberg]

Hello, when using Appdynamic agent (ver-4.2.1.7) on Weblogic (ver 12.2.1.0.0) nodes I get random java crashes. It is happening on all nodes, but some nodes are affected more often than others and it always happens during work time, when load is high.

I will add java version info and logs:

java version "1.8.0_172"

Java(TM) SE Runtime Environment (build 1.8.0_172-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
# Problematic frame:
# V [libjvm.so+0x46e720] ClassLoaderData::oops_do(OopClosure*, KlassClosure*, bool)+0xa0

VM Arguments:
jvm_args: -Xms256m -Xmx512m -XX:CompileThreshold=8000 -Dweblogic.Name=ecs2 -Djava.security.policy=/opt/bea/wls1221/wlserver/server/lib/weblogic.policy -XX:+UnlockCommercialFeatures -XX:+ResourceManagement -XX:+UseG1GC -Djava.net.preferIPv4Stack=true -Dweblogic.system.BootIdentityFile=/opt/bea/wls1221/user_projects/domains/icislivetoll/servers/ecs2/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.nmservice.RotationEnabled=true -Xms2048m -Xmx4096m -Djava.awt.headless=true -Donion.federation.conf=file:///opt/bea/wls1221/user_projects/domains/icislivetoll/onion3_ecs2_federation.xml -Denable.all.timers=true -Dcodelist.service.version=2 -Dlog4j.debug=true -DUseSunHttpHandler=true -Dcoal.PERMIT_WS_URL=http://muff.mta:7032/icis-webservices-bridge/PermitServiceWS -javaagent:/opt/bea/Appdynamics/appagent/javaagent.jar -Dappdynamics.agent.applicationName=ECS2_LIVE -Dappdynamics.agent.tierName=ecs2 -Dappdynamics.agent.nodeName=ecs2 -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -javaagent:/opt/bea/wls1221/wlserver/server/lib/debugpatch-agent.jar -da -Dwls.home=/opt/bea/wls1221/wlserver/server -Dweblogic.home=/opt/bea/wls1221/wlserver/server -Dweblogic.management.server=http://icislive.mta:7002
java_command: weblogic.Server
java_class_path (initial): /opt/java/jdk1.8.0_172/lib/tools.jar:/opt/bea/wls1221/wlserver/server/lib/weblogic.jar:/opt/bea/wls1221/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/opt/bea/wls1221/wlserver/modules/features/oracle.wls.common.nodemanager.jar:/opt/bea/wls1221/wlserver/server/lib/mbeantypes/wlManagementMBean.jar:/opt/bea/wls1221/wlserver/server/lib/mbeantypes/wlManagementImplSource.jar:/opt/bea/wls1221/user_projects/domains/icislivetoll/lib/onion-oaas-4.0.0.0.jar:/opt/bea/wls1221/wlserver/common/derby/lib/derbyclient.jar:/opt/bea/wls1221/wlserver/common/derby/lib/derby.jar:/opt/java/jdk1.8.0_172/lib/tools.jar::/opt/bea/Appdynamics/appagent/javaagent.jar:/opt/bea/wls1221/wlserver/server/lib/debugpatch-agent.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/opt/java/jdk1.8.0_172
JRE_HOME=/opt/java/jdk1.8.0_172/jre
CLASSPATH=/opt/java/jdk1.8.0_172/lib/tools.jar:/opt/bea/wls1221/wlserver/server/lib/weblogic.jar:/opt/bea/wls1221/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/opt/bea/wls1221/wlserver/modules/features/oracle.wls.common.nodemanager.jar:/opt/bea/wls1221/wlserver/server/lib/mbeantypes/wlManagementMBean.jar:/opt/bea/wls1221/wlserver/server/lib/mbeantypes/wlManagementImplSource.jar:/opt/bea/wls1221/user_projects/domains/icislivetoll/lib/onion-oaas-4.0.0.0.jar:/opt/bea/wls1221/wlserver/common/derby/lib/derbyclient.jar:/opt/bea/wls1221/wlserver/common/derby/lib/derby.jar:/opt/java/jdk1.8.0_172/lib/tools.jar:
PATH=/opt/bea/wls1221/user_projects/domains/icislivetoll/bin:/opt/bea/wls1221/wlserver/server/bin:/opt/bea/wls1221/wlserver/../oracle_common/modules/org.apache.ant_1.9.2/bin:/opt/java/jdk1.8.0_172/jre/bin:/opt/java/jdk1.8.0_172/bin:/opt/java/jdk1.8.0_172/jre/bin:/opt/java/jdk1.8.0_172/bin:/opt/bea/bin:/usr/local/bin:/usr/bin:/bin:/usr/games
LD_LIBRARY_PATH=:/opt/bea/wls1221/wlserver/server/native/linux/x86_64:/opt/bea/wls1221/wlserver/server/native/linux/x86_64/oci920_8
SHELL=/bin/bash
DISPLAY=localhost:10.0
HOSTTYPE=x86_64
OSTYPE=linux
MACHTYPE=x86_64-suse-linux

[Peter.Holditch]

A JVM crash like this is always the result of hitting a JVM bug.

Of course, with the agent present, different behaviour is experienced within the JVM (specifically, class retransformations), which often means that these JVM bugs are hit only in the presence of the agent.

Update 172 of java 8, which you are using, was released April 17th 2018, so it's not super old, but there have been 3 releases since then, which include several fixes for JVM crashes (I saw 7 on a quick scan of the subsequent release notes)

The first step to root-causing this is for sure to upgrade to the latest available build of java 8

Please let us know if this solves the issue.

[Erik.Pahlberg]

Hello, thank you for reply!

I actually updated Java version to:
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b33)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b33, mixed mode)

in test environment. 

[Erik.Pahlberg]

Hello, thank you for reply!

I actually updated Java version to:
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b33)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b33, mixed mode)

in test environment. 

[Erik.Pahlberg]

One possible solution that we are trying in production environment is lowering "Error Detection" levels, so Appdynamics won't go and look for errors in log's, because there is myriad of errors in log files. Can you confirm, that this could also cause crashes?

[Peter.Holditch]

Erik,

The AppDynamics java agent never looks in logs for errors, it intercepts them being logged by instrumenting logging frameworks.

I think its quite unlikely that changing the error detection rules would have an impact on JVM crash behaviour.

Did you try upgrading the JVM?

[Erik.Pahlberg]

I updated java version as said before. But only in test environment. Can't update it randomly in production environment, especially when im not sure, if it even helps.

[Peter.Holditch]

Ah, sorry.  I had missed your earlier comment.

Often customers find that the security fixes contained within each update release as a good motivator for justifying upgrades the producton JVM estate.

That said, I understand how frustrating it is to need to push through a production JVM upgrade without absolute certainty that it will fix a crash issue.  For what it's worth, in my experience, that is the outcome in the vast majority of cases,

Warm regards,

Peter.

[Erik.Pahlberg]

Hopefully I will be able to upgrade Java in production environment next week.

[Erik.Pahlberg]

I stumbled upon "https://community.appdynamics.com/t5/Java-Java-Agent-Installation-JVM/About-Dappdynamics-agent-disab..." and I thought, that maybe I can also try this? Adding this option won't require any restart?