Cisco AppDynamics Community

pclark · ‎02-02-2015

AppDynamics provides out-of-the-box configuration of JMX/PMI metrics based on the MBeans exposed by many commonly used app servers.

The MBean Browser is accessed on the Node dashboard from the JMX tab. The browser is used to look at MBean metric values for short-term troubleshooting.

For long-term continuous monitoring, you can create a metric based on the MBean statistics exposed by your app server. You do this using the Configure -> Instrumentation -> JMX tab. For details, see Configure JMX Metrics from MBeans.

When the app server starts up, the associated MBean server starts and the MBeans are discovered. The timing of these activities can vary by app server and by configuration.

The AppDynamics agent discovers the MBean server associated with running app servers in the application environment and reads the exposed MBeans to get all JMX related statistics.

The path for JMX statistics is embedded in the MBean configuration which is set internally when the server registers those MBeans to the MBean server during startup. The AppDynamics agent uses MBean specific APIs to get the MBeans, their object pattern and path out of those registered MBeans. If the server does not expose the MBeans, the AppDynamics agent can't see them.

When you don't see the MBeans you expect, use the following flowchart and techniques to troubleshoot the issue.

1. Enable MBeans

The MBeans need to be enabled and exposed by your JVM/app server.

Issue: MBeans Not Exposed

One reason that you might not see MBeans in the AppDynamics MBean Browser is that they are not enabled on your app server or not exposed by the app server and therefore the AppD agents are not able to get the data related to those statistics.

The JMX agent (also known as the MBean server) needs to be enabled on the JVM or app server. And the specific statistics that you want to see need to be exposed via Mbean monitoring by that appserver.

Solution: Confirm MBean Availability

To verify that the MBeans are available, use an independent tool such as JConsole. If the MBean is visible under one of those tools then move to the next issue and confirm that there is enough time for the agent to discover the MBeans.

If the MBeans are not visible in JConsole, then AppD agent cannot get the data either. View the documentation for your server to enable the MBeans and review which JMX statistics are being exposed for use.

See Oracle's documentation for JConsole here: JConsole.

Issue: Specific MBean Not Visible

Sometimes an app server does not automatically expose the information that you want to see.

Solution: Enable MBean

Use the app server documentation to locate the statistics that you want to monitor and then use the app server's admin console to enable the statistics. Use a tool such as JConsole to confirm that the MBean statistics that you want are visible.

2. Agent Discovery of MBeans

The AppDynamics agent tries to capture the MBean domains and JMX/PMI statistics within two minutes of app server start or restart, assuming all the domains and MBeans can be discovered within that time. If the MBean server (associated with your app server) is not started in that time frame or not started at all, the MBeans can not be discovered.

Issue: App Server Start time

When an app server starts up, the associated MBean server starts and the MBeans are discovered. The timing of these activities varies by app server and by configuration. If this activity is not completed in the time that the AppD agent is expecting to discover the MBeans, then the MBean Browser will not show them.

Solution: Delay Discovery

You can delay the discovery of MBeans to make sure that agent discovers all the domains after complete start up of the app server. The default delay for AppD agent is two minutes. You can increase this time using the jmx-appserver-mbean-finder-delay-in-seconds node property.

To use the node property:
1. Register the jmx-appserver-mbean-finder-delay-in-seconds node property from the Node Dashboard. Use these steps: Add a Node Property.

2. Enter a value, such as "300". Set the delay to a time which is 1.5 times of your app server 's startup time.

3. Restart the JVM/app server.

Solution: Trigger Rediscovery

You can trigger the rediscovery of MBeans to make sure that the agent discovers all the domains after complete start up of the app server by using the jmx-rediscover-mbean-servers node property.

1. Register the jmx-rediscover-mbean-servers node property from the Node Dashboard. Use these steps: Add a Node Property.
2. Enter the value "true".
3. Restart the JVM/server.

3. Other Troubleshooting

Issue: MBean Limits

There are two limits: Per Domain and Attribute.

Per Domain Limit

With some app servers, it is possible to exceed the MBean count limit for a domain. The limit is controlled by the jmx-max-mbeans-to-load-per-domain node property. The default value is 1000.

Attribute Limit

With some app servers, it is possible to exceed the MBean attribute limit. The limit is controlled by the jmx-max-mbean-attributes-to-load node property. The default value is 1000.

Solution: Increase the Limit

1. Register the appropriate limit node property from the Node Dashboard. Use these steps: Add a Node Property.
2. Enter an Integer value greater than the default value.
3. Restart the JVM/server.

Issue: Last Resort

If none of the previous techniques have solved your problem, you can get an XML file that shows all the MBeans in each domain. Sometimes, the object name patterns can change between app server releases and reviewing the XML file might help you debug special corner cases.

Solution: Generate and Review XML for the MBeans

Setting the discover-mbeans node property to true causes the agent to discover all MBeans in a JVM/app server and generate XML files in the $AGENT_RUNTIME_DIR/conf/nodeDir/discovered=mbeans directory.

For each MBean domain a corresponding XML file containing all MBeans for that domain is created.

For example: for two domains such as "java.lang" and "Catalina", the XML files are named:

Catalina-jmx-config.xml
java.lang-jmx.config.xml

You can examine the XML file for your MBean.

To use the discover-mbeans node property:

==========
1. Register the discover-mbeans node property rom the Node Dashboard. Use these steps: Add a Node Property.
2. Enter the value "true".
3. Restart the JVM/server.
==========

Examining Logs for MBean Information

MBean and JMX Metric Logs

The following log entries show the MBean finder is initialized with the two minute delay described above.

Thread-0] 03 Aug 2014 00:15:44,189 INFO ServerMBeanManagerVersion2 - 
Initialized MBean Finder with delay=120 secs
[Thread-0] 03 Aug 2014 00:15:44,191 INFO JMXService - Server JMX metric 
collection initialized with update interval [60] seconds

The following log entries show that no MBean server was detected by the agent. In this example, the application being monitoring did not contain any of these servers.

[AD Thread Pool-Global0] 07 Aug 2014 22:09:52,204 DEBUG BaseMBeanServerReporterVersion2 - No mbean servers exist for domain [jboss.web]. No metrics will be reported
[AD Thread Pool-Global0] 07 Aug 2014 22:09:52,204 DEBUG BaseMBeanServerReporterVersion2 - No mbean servers exist for domain [WebSpherePMI]. No metrics will be reported
[AD Thread Pool-Global0] 07 Aug 2014 22:09:52,204 DEBUG BaseMBeanServerReporterVersion2 - No mbean servers exist for domain [WebSpherePMI]. No metrics will be reported
[AD Thread Pool-Global0] 07 Aug 2014 22:09:52,204 DEBUG BaseMBeanServerReporterVersion2 - No mbean servers exist for domain [org.apache.cassandra.net]. No metrics will be reported