Why can't I see MBean metrics in the Metric Browser?

[pclark]

This article covers some reasons that a configured MBean metric might not show up in the Metric Browser.

Confirm a Persistent Metric is Configured

Not all MBeans are configured as persistent metrics in AppDynamics. First confirm that the information you want is exposed as a MBean and that the MBean attributes have been configured as an AppDynamics JMX metric.

1. Determine if your app server exposes the information that you want to see using a tool such as JConsole.
2. Use your app server documentation to find the Object name pattern for the MBean that you want to see.
3. If the MBean you want is exposed, then you can create a metric for it in AppDynamics.
4. Use this documentation to configure the Configure JMX Metrics from MBeans.

Example: WebLogic Server

The Primary Sessions and Sessions Replicas metrics from WebLogic Server are not instrumented by default in AppDynamics. Since WebLogic Server exposes the Session Replication information via JMX, you can configure a JMX metric that AppD can use for monitoring.

1. Use your app server documentation to find the specific Object Name pattern and the available attributes. For example:

Primary Sessions: Provides the number of objects that the local server hosts as primaries. From MBean Attribute Reference: ReplicationRuntimeMBean.PrimaryCount

Session Replicas:
Provides the number of objects that the local server hosts as secondaries. From MBean Attribute Reference: ReplicationRuntimeMBean.SecondaryCount.

2. Locate the "ReplicationRuntimeMBean" in the MBean Browser and create a JMX metric on the attributes, "PrimaryCount" and "SecondaryCount".

Confirm that All MBean Domains Were Discovered

If you have created the necessary JMX rule and you still don't see the metric being reported, check the appserver startup time. When the appserver takes more than two minutes to start, the agent can't discover all the domains. In such scenarios, you can use the jmx-appserver-mbean-finder-delay-in-seconds node property to delay the discovery of MBeans to make sure that agent discovers all the domains after complete startup of the appserver.

You might see logs similar to this:

[AD Thread Pool-Global1] 27 Jun 2014 10:24:06,603 WARN WebSpherePMIStatsHandlerVersion2 - Stats is NULL for statDescriptor [threadPoolModule>WebContainer]. No metrics will be reported.

Steps for using the jmx-appserver-mbean-finder-delay-in-seconds node property are here: Can Not See Expected MBeans.

Confirm that MBean Limits Were Not Hit

There are some limits associated with Mbean metrics. These limits can be adjusted using two node properties. Agent logs report when limits are being hit. For details on using node properties, see App Agent Node Properties.

MBean Browser Limit

The jmx-max-mbeans-to-load-per-domain node property controls the number of MBeans that are visible for each domain. The default value = 1000.

Metric Browser Limit

The jmx-max-metrics-to-report node property controls the total number of JMX metrics that are reported in the Metric Browser. The default = 500.

Using Logs for Debugging

The persistent JMX metrics created from MBeans follow the same life cycle as other AppDynamics monitored entities. The following logs are for an example JMX metric "jdbc/cartDS" created for Node-8003 of the ECommerce tier on the DataSource MBean using the MaxActive attribute.

Searching in the agent log using the assigned name "jdbc/cartDS", you can find log entries similar to the following examples. 

The MBean details appear in this entry:

[AD Thread Pool-Global1] 03 Oct 2013 14:30:43,646 INFO ManagedObjectFactory - Instantiated JMX Managed object for bean=JDBCConnectionPool, category=JDBC Connection Pools, instance=Catalina:class=javax.sql.DataSource,host=localhost,name="jdbc/cartDS",path=/appdynamicspilot,type=DataSource

This log snippet shows the attribute and the MBean pattern that was used to create the JMX metric. You can search on the metric name that you assigned. You can also search for "JMXMetricRepository" and it will show you the rule that was added as shown here:

[AD Thread Pool-Global1] 03 Oct 2013 19:50:46,247 INFO JMXMetricRepository - Added new JMX Rule [JMXMetricRule [ MBeanQuery [MBeanQuery [ domain [Catalina], mbeanPattern [Catalina:type=DataSource,path=/appdynamicspilot,host=localhost,class=javax.sql.DataSource,name="jdbc/cartDS"], queryLogicalOperator [AND], queryExpressions []]], metricCategory [Individual Nodes], beanName [null], metricPath [null], instanceName [null], instanceIdentifier [null], name [jdbcCart-DataSource], domain [Catalina] Attribute Definitions [ JMX Attribute Definition [mbeanAttributeName [maxActive], metricName [maxActive], metricTimeRollupType [AVERAGE], metricClusterRollupType [INDIVIDUAL], metricAggregatorType [AVERAGE], metricHoleType [null], getterChain [] ] ] ]]

This log snippet show sthe metric ID being received from controller = 3050.

[AD Thread-Metric Reporter0] 03 Oct 2013 19:52:03,440 DEBUG MetricHandler - Added for metric registration [Metric Name[Server|Component:3|JMX|Individual Nodes:"jdbc/cartDS"|maxActive]]
[AD Thread-Metric Reporter0] 03 Oct 2013 19:52:03,464 DEBUG MetricHandler - Response body for metric registration <request>
<metric id="3050" name="Server|Component:3|JMX|Individual Nodes:&quot;jdbc/cartDS&quot;|maxActive"/>

These entries show the metric value being reported to the controller at one minute intervals.

[AD Thread-Metric Reporter0] 03 Oct 2013 19:55:53,445 DEBUG MetricHandler - 3050 Server|Component:3|JMX|Individual Nodes:"jdbc/cartDS"|maxActive 100 1049826925
[AD Thread-Metric Reporter0] 03 Oct 2013 19:56:53,445 DEBUG MetricHandler - 3050 Server|Component:3|JMX|Individual Nodes:"jdbc/cartDS"|maxActive 100 1049826925
[AD Thread-Metric Reporter0] 03 Oct 2013 19:57:53,445 DEBUG MetricHandler - 3050 Server|Component:3|JMX|Individual Nodes:"jdbc/cartDS"|maxActive 100 1049826925
[AD Thread-Metric Reporter0] 03 Oct 2013 19:58:53,445 DEBUG MetricHandler - 3050 Server|Component:3|JMX|Individual Nodes:"jdbc/cartDS"|maxActive 100 1049826925