java.lang.NoClassDefFoundError: com/singularity/ee/agent/appagent/entrypoint/bciengine/FastMethodInt

[Tim.Dallmann]

I have seen this referenced in several other posts related to other servers, like JBoss, but nothing that seems to help me.  

We have a system called Guidewire BillingCenter, which runs on Tomcat 8.  I have 2 servers that have the Java agent applied and are running fine, but one machine gets the exception at startup:
java.lang.NoClassDefFoundError: com/singularity/ee/agent/appagent/entrypoint/bciengine/FastMethodInterceptorDelegatorBoot  

From what I can tell this is a classpath issue, right?  If so, what would allow one server to work fine, while another fails?  I have compared both the working and non-working servers as best I can and I can't find any differences that would explain this behavior.  

[Peter.Holditch]

This ususally appears in scenarios where there is an OSGi container in place, and boot delegation has not been set up for com.singularity classes.

I am not, however, aware that Guidewire on tomcat uses OSGi and the fact that you have it working in 2/3 apparently identical environments is suspicious too.

Instrumenting Guidewire often requires the following additions (in blue) to the app-agent-config.xml:

<agent-service name="BCIEngine" enabled="true">
            <optional-service-dependencies>Agent16Service</optional-service-dependencies>
            <configuration-properties>
                <property name="exclude.classes.from.class.loader.1" value="com.guidewire.pl.system.gosu.GosuPluginContainer" />
                <property name="exclude.classes.from.class.loader.2" value="gw.internal.gosu.compiler.GosuClassLoader" />
                <property name="exclude.classes.from.class.loader.3" value="com.guidewire.pl.system.integration.plugins.Standard" />
                <property name="exclude.classes.from.class.loader.4" value="com.guidewire.pl.system.integration.plugins.StandardClassLoader" />
            </configuration-properties>

If applying this does not help, can you post the full agent log so we can see the context in which this exception appears?

Warm regards,
Peter.

[Tim.Dallmann]

Thanks for the response Peter.  We had already setup the first three exclusions you provided, so I added the fourth - still getting the same error.  

I'm attaching the log folder for the agent.  I purged the old logs and then went through tomcat startup so it should only contain relevant data.  

As it turns out, the other 2 servers that I thought were working are not - I had an invalid character in app-agent-config.xml, which meant that it allowed the BillingCenter process to start, but did not start the AppDynamics agent.  So I suspect all three servers will have the same issue now (but I can't re-start it to validate, as people are actively testing in that environment right now).  

Related to your comment about Guidewire and OSGI - I found some references on the web that implied Guidewire was moving in that direction.  Since we recently upgraded BillingCenter to version 9.x, it's very possible that it does use OSGI.  

Thanks for you assistance! 

[Julia.Leyton]

Hey there,

did you fix this issue? If so, could you please share how did you do it? We are having the same problem and can't find any solution documented or side notes in the documentation.

Thanks!

Julia

[andrew.sumner]

I've got the same issue so if you've found a solution I'd be very gratefull if you can post it here thanks.

[Julia.Leyton]

Hi,

in our case we updated <base-liferay-folder>/tomcat-8.0.32/webapps/ROOT/WEB-INF/classes/portal-ext.properties adding:

#appdynamics agent override
module.framework.properties.org.osgi.framework.bootdelegation=__redirected,com.singularity.*,com.liferay.aspectj,com.liferay.aspectj.*,com.liferay.portal.servlet.delegate,com.liferay.portal.servlet.delegate*,com.sun.ccpp,com.sun.ccpp.*,com.sun.crypto.*,com.sun.image.*,com.sun.jmx.*,com.sun.jna,com.sun.jndi.*,com.sun.mail.*,com.sun.management.*,com.sun.media.*,com.sun.msv.*,com.sun.org.*,com.sun.syndication,com.sun.tools.*,com.sun.xml.*,com.yourkit.*,org.eclipse.persistence.internal.jaxb,org.eclipse.persistence.internal.jaxb.*,sun.*

and restarted the app.

Best!

Julia

[andrew.sumner]

Thanks for the reply.  We ended up using this setup to get Guidewire Running with AppDynamics (Tomcat 8.0 on Windows)

Edit
C:\apache-tomcat-8.0.44\webapps\<app>\modules\configuration\config\security\serialization-whitelist.lst
Add
org.osgi.framework.bootdelegation=com.singularity.*
Save

Edit
  C:\apache-tomcat-8.0.44\webapps\<app>\modules\configuration\config\config.xml
Change
  <param name="SerializationWhitelistEnabled" value="false" />
To
  <param name="SerializationWhitelistEnabled" value="true" />
Save

NOTE: Will require a Guidewire with full upgrade mode because of this config change.

Unzip the appdynamics agent to C:\appdynamics
If not pre-configured with appdynamics server details then follow this guide
https://docs.appdynamics.com/display/PRO41/Install+the+Java+Agent

Edit
  C:\appdynamics\appserveragent\ver4.3.8.0\conf\app-agent-config.xml

Add the inner items to the following sections:

<app-agent-configuration>
  <configuration-properties>

    <!-- Guidewire -->
    <property name="exclude.classes.from.class.loader.1" value="com.guidewire.pl.system.gosu.GosuPluginContainer" />
    <property name="exclude.classes.from.class.loader.2" value="gw.internal.gosu.compiler.GosuClassLoader" />
    <property name="exclude.classes.from.class.loader.3" value="com.guidewire.pl.system.integration.plugins.Standard" />
    <property name="exclude.classes.from.class.loader.4" value="com.guidewire.pl.system.integration.plugins.StandardClassLoader" />

  </configuration-properties>

  <agent-service name="BCIEngine" enabled="true">
    <configuration>
      <bci-processing-excludes>

        <!-- Guidewire -->
        <custom-exclude filter-type="STARTSWITH" filter-value="gw.api/" />
        <custom-exclude filter-type="STARTSWITH" filter-value="org.apache.xml.dtm.ref/" />
        <custom-exclude filter-type="STARTSWITH" filter-value="com/guidewire/"/>

      </bci-processing-excludes>
    </configuration>
  </agent-service>

  <agent-service name="TransactionMonitoringService" enabled="true">
    <configuration>
      <fork-config>

        <!-- Guidewire -->
        <excludes filter-type="STARTSWITH" filter-value="com.guidewire.pl.cluster.internal"/>
      </fork-config>
    </configuration>
  </agent-service>

</app-agent-configuration>
Save

Parameters (to add) for TomcCat - most of this can be done via configuration file, not sure about javaagent entry yet
-javaagent:C:\appdynamics\appserveragent\ver4.3.3.8\javaagent.jar
# -Dorg.osgi.framework.bootdelegation=singularity.* IS THIS ONE REQUIRED?
-Dappdynamics.agent.applicationName=<application_name>
-Dappdynamics.agent.tierName=<tier_name>
-Dappdynamics.agent.nodeName=<hostname>-<tier_name>

[Sumit.Khaitan]

I am receiving similar error on (WebSphere Application Server 18.0.0.3/wlp-1.0.22.cl180320180905-2337) on IBM J9 VM, version 8.0.5.22

[ERROR ] CWWKE0701E: FrameworkEvent ERROR Bundle:com.ibm.ws.kernel.service.location(id=6) org.osgi.framework.BundleException: Exception in com.ibm.ws.kernel.service.location.internal.Activator.start() of bundle com.ibm.ws.kernel.service.location.
at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:800)
at [internal classes]
Caused by: java.lang.NoClassDefFoundError: com/singularity/ee/agent/appagent/entrypoint/bciengine/FastMethodInterceptorDelegatorBoot
at com.ibm.ws.kernel.service.location.internal.SymbolRegistry$1.<init>(SymbolRegistry.java:39)
... 1 more
Event:org.osgi.framework.FrameworkEvent[source=com.ibm.ws.kernel.service.location_1.1.22.cl180320180905-2337 [6]]
[ERROR ] CWWKE0701E: FrameworkEvent ERROR Bundle:com.ibm.ws.config(id=10) org.osgi.framework.BundleException: Exception in com.ibm.ws.config.internal.WSConfigActivator.start() of bundle com.ibm.ws.config.
at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:800)
at [internal classes]
Caused by: java.lang.NoClassDefFoundError: com/singularity/ee/agent/appagent/entrypoint/bciengine/FastMethodInterceptorDelegatorBoot
at com.ibm.ws.config.admin.internal.UpdateQueue$Queue.<init>(UpdateQueue.java:41)
... 1 more
Event:org.osgi.framework.FrameworkEvent[source=com.ibm.ws.config_1.0.22.cl180320180905-2337 [10]]

Kindly help

[Peter.Holditch]

The WebSphere issue here is that the classes being instrumented are loaded by some OSGi framework.

You need to configure boot delegation as documented here.

Warm regards,

Peter