Not a customer? Click the 'Start a free trial' link to begin a 30-day SaaS trial of our product and to join our community.
Existing Cisco AppDynamics customers should click the 'Sign In' button to authenticate to access the community
12-03-2018 10:44 PM - edited 12-03-2018 10:45 PM
Hello, when using Appdynamic agent (ver-4.2.1.7) on Weblogic (ver 12.2.1.0.0) nodes I get random java crashes. It is happening on all nodes, but some nodes are affected more often than others and it always happens during work time, when load is high.
I will add java version info and logs:
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)
# Problematic frame:
# V [libjvm.so+0x46e720] ClassLoaderData::oops_do(OopClosure*, KlassClosure*, bool)+0xa0
VM Arguments:
jvm_args: -Xms256m -Xmx512m -XX:CompileThreshold=8000 -Dweblogic.Name=ecs2 -Djava.security.policy=/opt/bea/wls1221/wlserver/server/lib/weblogic.policy -XX:+UnlockCommercialFeatures -XX:+ResourceManagement -XX:+UseG1GC -Djava.net.preferIPv4Stack=true -Dweblogic.system.BootIdentityFile=/opt/bea/wls1221/user_projects/domains/icislivetoll/servers/ecs2/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.nmservice.RotationEnabled=true -Xms2048m -Xmx4096m -Djava.awt.headless=true -Donion.federation.conf=file:///opt/bea/wls1221/user_projects/domains/icislivetoll/onion3_ecs2_federation.xml -Denable.all.timers=true -Dcodelist.service.version=2 -Dlog4j.debug=true -DUseSunHttpHandler=true -Dcoal.PERMIT_WS_URL=http://muff.mta:7032/icis-webservices-bridge/PermitServiceWS -javaagent:/opt/bea/Appdynamics/appagent/javaagent.jar -Dappdynamics.agent.applicationName=ECS2_LIVE -Dappdynamics.agent.tierName=ecs2 -Dappdynamics.agent.nodeName=ecs2 -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -javaagent:/opt/bea/wls1221/wlserver/server/lib/debugpatch-agent.jar -da -Dwls.home=/opt/bea/wls1221/wlserver/server -Dweblogic.home=/opt/bea/wls1221/wlserver/server -Dweblogic.management.server=http://icislive.mta:7002
java_command: weblogic.Server
java_class_path (initial): /opt/java/jdk1.8.0_172/lib/tools.jar:/opt/bea/wls1221/wlserver/server/lib/weblogic.jar:/opt/bea/wls1221/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/opt/bea/wls1221/wlserver/modules/features/oracle.wls.common.nodemanager.jar:/opt/bea/wls1221/wlserver/server/lib/mbeantypes/wlManagementMBean.jar:/opt/bea/wls1221/wlserver/server/lib/mbeantypes/wlManagementImplSource.jar:/opt/bea/wls1221/user_projects/domains/icislivetoll/lib/onion-oaas-4.0.0.0.jar:/opt/bea/wls1221/wlserver/common/derby/lib/derbyclient.jar:/opt/bea/wls1221/wlserver/common/derby/lib/derby.jar:/opt/java/jdk1.8.0_172/lib/tools.jar::/opt/bea/Appdynamics/appagent/javaagent.jar:/opt/bea/wls1221/wlserver/server/lib/debugpatch-agent.jar
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/opt/java/jdk1.8.0_172
JRE_HOME=/opt/java/jdk1.8.0_172/jre
CLASSPATH=/opt/java/jdk1.8.0_172/lib/tools.jar:/opt/bea/wls1221/wlserver/server/lib/weblogic.jar:/opt/bea/wls1221/wlserver/../oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/opt/bea/wls1221/wlserver/modules/features/oracle.wls.common.nodemanager.jar:/opt/bea/wls1221/wlserver/server/lib/mbeantypes/wlManagementMBean.jar:/opt/bea/wls1221/wlserver/server/lib/mbeantypes/wlManagementImplSource.jar:/opt/bea/wls1221/user_projects/domains/icislivetoll/lib/onion-oaas-4.0.0.0.jar:/opt/bea/wls1221/wlserver/common/derby/lib/derbyclient.jar:/opt/bea/wls1221/wlserver/common/derby/lib/derby.jar:/opt/java/jdk1.8.0_172/lib/tools.jar:
PATH=/opt/bea/wls1221/user_projects/domains/icislivetoll/bin:/opt/bea/wls1221/wlserver/server/bin:/opt/bea/wls1221/wlserver/../oracle_common/modules/org.apache.ant_1.9.2/bin:/opt/java/jdk1.8.0_172/jre/bin:/opt/java/jdk1.8.0_172/bin:/opt/java/jdk1.8.0_172/jre/bin:/opt/java/jdk1.8.0_172/bin:/opt/bea/bin:/usr/local/bin:/usr/bin:/bin:/usr/games
LD_LIBRARY_PATH=:/opt/bea/wls1221/wlserver/server/native/linux/x86_64:/opt/bea/wls1221/wlserver/server/native/linux/x86_64/oci920_8
SHELL=/bin/bash
DISPLAY=localhost:10.0
HOSTTYPE=x86_64
OSTYPE=linux
MACHTYPE=x86_64-suse-linux
12-04-2018 01:38 AM
A JVM crash like this is always the result of hitting a JVM bug.
Of course, with the agent present, different behaviour is experienced within the JVM (specifically, class retransformations), which often means that these JVM bugs are hit only in the presence of the agent.
Update 172 of java 8, which you are using, was released April 17th 2018, so it's not super old, but there have been 3 releases since then, which include several fixes for JVM crashes (I saw 7 on a quick scan of the subsequent release notes)
The first step to root-causing this is for sure to upgrade to the latest available build of java 8
Please let us know if this solves the issue.
12-04-2018 02:23 AM
Hello, thank you for reply!
I actually updated Java version to:
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b33)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b33, mixed mode)
in test environment.
12-04-2018 02:23 AM
Hello, thank you for reply!
I actually updated Java version to:
java version "1.8.0_192"
Java(TM) SE Runtime Environment (build 1.8.0_192-b33)
Java HotSpot(TM) 64-Bit Server VM (build 25.192-b33, mixed mode)
in test environment.
12-06-2018 01:10 AM
One possible solution that we are trying in production environment is lowering "Error Detection" levels, so Appdynamics won't go and look for errors in log's, because there is myriad of errors in log files. Can you confirm, that this could also cause crashes?
12-06-2018 02:31 AM
Erik,
The AppDynamics java agent never looks in logs for errors, it intercepts them being logged by instrumenting logging frameworks.
I think its quite unlikely that changing the error detection rules would have an impact on JVM crash behaviour.
Did you try upgrading the JVM?
12-06-2018 04:09 AM
I updated java version as said before. But only in test environment. Can't update it randomly in production environment, especially when im not sure, if it even helps.
12-06-2018 04:20 AM
Ah, sorry. I had missed your earlier comment.
Often customers find that the security fixes contained within each update release as a good motivator for justifying upgrades the producton JVM estate.
That said, I understand how frustrating it is to need to push through a production JVM upgrade without absolute certainty that it will fix a crash issue. For what it's worth, in my experience, that is the outcome in the vast majority of cases,
Warm regards,
Peter.
12-06-2018 04:24 AM - edited 12-06-2018 04:25 AM
Hopefully I will be able to upgrade Java in production environment next week.
12-10-2018 05:03 AM - edited 12-11-2018 11:53 PM
I stumbled upon "https://community.appdynamics.com/t5/Java-Java-Agent-Installation-JVM/About-Dappdynamics-agent-disab..." and I thought, that maybe I can also try this? Adding this option won't require any restart?
Thank you! Your submission has been received!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form