Oracle Weblogic Server 12.2.1.3 does not start up with Java Agent in the startup argument

[Anil.Anjaneya]

oracle weblogic server 12.2.1.3 is not starting up after I set up Java Agent (ver4.5.7.24863) in the start up arguments. Does anybody know how to resolve the below error and load up the Java Agent successfully inside Oracle Weblogic Server 12.2.1.3 ?

Initially I can see the Agent starts up successfully with below entries in the apps server logs:

Full Agent Registration Info Resolver using selfService [true]

Full Agent Registration Info Resolver using selfService [true]

Full Agent Registration Info Resolver using application name [null] Full Agent Registration Info Resolver using tier name [unknown]

[Thread-2] Wed Feb 20 00:31:29 EST 2019[INFO]: JavaAgent - Started AppDynamics Java Agent Successfully. Then apps server (weblogic) shuts down because of the error below: 

weblogic.coherence.api.internal.CoherenceException: (Wrapped) Illegal field modifiers in class com/tangosol/net/Invocable: 0x1A
at weblogic.cacheprovider.coherence.CoherenceClusterManager.processException(CoherenceClusterManager.java:837)

[Umervali.Niyamathulla]

hey 

We recently successfully instrumented weblogic in startweblogic.sh file along with Appdynamics java agent parameters ....can you share where did you input appdynamcis parameters?

[Umervali.Niyamathulla]

also refer https://docs.appdynamics.com/display/PRO45/Oracle+WebLogic+Startup+Settings

[Anil.Anjaneya]

We need the instrumentation on Managed Servers. So we have added the java agent to the custom startup file which gets loaded by the setDomainEnv.sh file.

[Anil.Anjaneya]

Here are the step followed:

1. installed Java Agent on the llinux machine. (version: ver4.5.7.24863)

2. Added permissions in the weblogi.policy file as below:

grant codeBase "file:/u01/app/AppServerAgent/-"
{ permission java.security.AllPermission; };

3. Added the Java Agent argument in the custom startup file which loads up and shows up as below:

/u01/app/oracle/product/fmw/12.2.1/osb/jdk1.8.0_172/bin/java -server -javaagent:/u01/app/AppServerAgent/javaagent.jar -Xms8192m -Xmx8192m -cp /u01/app/oracle/product/fmw/12.2.1/osb/wlserver/server/lib/weblogic-launcher.jar -Dlaunch.use.env.classpath=true -Dweblogic.Name=osb_server1 -Djava.security.policy=/u01/app/oracle/product/fmw/12.2.1/osb/wlserver/server/lib/weblogic.policy -Dweblogic.system.BootIdentityFile=/u01/app/oracle/config/domains/12.2.1/osbdomain/servers/osb_server1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.nmservice.RotationEnabled=true -Djava.system.class.loader=com.oracle.classloader.weblogic.LaunchClassLoader -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|com.bea.wli.sb.resources.url -Doracle.mds.filestore.preferred= -Dadf.version=12.2.1.3.0 -Dopss.version=12.2.1.3 -Digf.arisidbeans.carmlloc=/u01/app/oracle/config/domains/12.2.1/osbdomain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/app/oracle/config/domains/12.2.1/osbdomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/app/oracle/config/domains/12.2.1/osbdomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/app/oracle/config/domains/12.2.1/osbdomain/servers/osb_server1/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/app/oracle/product/fmw/12.2.1/osb/oracle_common/modules/oracle.ossoiap,/u01/app/oracle/product/fmw/12.2.1/osb/oracle_common/modules/oracle.oamprovider,/u01/app/oracle/product/fmw/12.2.1/osb/oracle_common/modules/oracle.jps -Dweblogic.jdbc.remoteEnabled=true -Dcommon.components.home=/u01/app/oracle/product/fmw/12.2.1/osb/oracle_common -Djrf.version=12.2.2 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/app/oracle/config/domains/12.2.1/osbdomain -Doracle.server.config.dir=/u01/app/oracle/config/domains/12.2.1/osbdomain/config/fmwconfig/servers/osb_server1 -Doracle.domain.config.dir=/u01/app/oracle/config/domains/12.2.1/osbdomain/config/fmwconfig -Doracle.ess.home=/u01/app/oracle/product/fmw/12.2.1/osb/oracle_common -Doracle.osb.home=/u01/app/oracle/product/fmw/12.2.1/osb/osb -javaagent:/u01/app/oracle/product/fmw/12.2.1/osb/wlserver/server/lib/debugpatch-agent.jar -da -Dwls.home=/u01/app/oracle/product/fmw/12.2.1/osb/wlserver/server -Dweblogic.home=/u01/app/oracle/product/fmw/12.2.1/osb/wlserver/server -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder -Dem.oracle.home=/u01/app/oracle/product/fmw/12.2.1/osb/em -Dweblogic.configuration.schemaValidationEnabled=false -Dums.oracle.home=/u01/app/oracle/product/fmw/12.2.1/osb/oracle_common -Dweblogic.management.server=http://admint1.oracle.com:9001 -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:MaxTenuringThreshold=6 -XX:NewSize=3584m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+UseGCLogFileRotation -XX:GCLogFileSize=1m -XX:NumberOfGCLogFiles=20 -Xloggc:/u01/app/oracle/ha_osb/domains/12.2.1/osbdomain/gclogs/osb_server1.log -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/u01/app/oracle/ha_osb/domains/12.2.1/osbdomain/heapdumps/osb_server1.hprof -Dcoherence.mode=dev -Dcoherence.edition=EE -Dcoherence.cacheconfig=/u01/app/oracle/ha_osb/domains/12.2.1/osbdomain/cacheconfigurationfiles/defaultservice-near-distributed-coherence-cache-config.xml -Dweblogic.management.server=http://admint1.oracle.com:9001 -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=/u01/app/oracle/ha_osb/domains/12.2.1/osbdomain/flightrecords,maxage=3h,dumponexit=true,dumponexitpath=/u01/app/oracle/ha_osb/domains/12.2.1/osbdomain/flightrecords -Dweblogic.SAFTXTimeout=250 -Dweblogic.jms.saf.RefreshInterval=900000 -Dweblogic.debug.DebugJMSSAF=true -Dweblogic.debug.DebugJMSMessagePath=true -Dweblogic.debug.DebugJMSFrontEnd=true -Dweblogic.debug.DebugJMSMessagePath=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=19002 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.access.file=/u01/app/oracle/ha_osb/domains/12.2.1/osbdomain/monitoringcreds/jms.access -Dcom.sun.management.jmxremote.password.file=/u01/app/oracle/ha_osb/domains/12.2.1/osbdomain/monitoringcreds/jms.password -Djava.util.logging.manager=oracle.core.ojdl.logging.ODLLogManager weblogic.Server

[Peter.Holditch]

Anil,

I have not seen this error before, but I suspect you can supress it by excluding instrumentation of the com.tangosol.net package.

If you look in /u01/app/AppServerAgent/ver4.5.7.24863/

you will find a file called app-agent-config.xml

In there, you will see a line saying:

<agent-service name="BCIEngine" enabled="true">

beneath this, you will see

<bci-processing-excludes>

and under there lines like

<custom-exclude filter-type="STARTSWITH" filter-value="io/netty/util/concurrent/SingleThreadEventExecutor"/>

if you add this:

<custom-exclude filter-type="STARTSWITH" filter-value="com/tangosol/net/"/>

and restart, you should find the process starts fine.

Please let me know if this helps.

Warm regards,
Peter

[Parameswaran.Perinkulam Sankaranarayanan]

Hello,

I received the below error when trying to restart the Weblogic instance after adding the AppDynamics arguments to the start-up script of managedWeblogic server.

Admin Server Start Exception: Error occurred while performing nmStart : Error Starting server AdminServer : Received error message from Node Manager Server: [Server start command for WebLogic server 'AdminServer' failed due to: [Server failed to start up but Node Manager was not aware of the reason]. Please check Node Manager log and/or server 'AdminServer' log for detailed information.]. Please check Node Manager log for details.

Use dumpStack() to view the full stacktrace :

Also added the below to the weblogic.policy file , still did not come up.

grant codeBase "file:/apps/appdynamics/-" { permission java.security.AllPermission; };

However , the jvm comes up fine if the AppDynamics arguments are removed.

Please suggest a fix/solution for this.

Thanks,

Parameswaran