Dashboards with custom JMX metrics

[John.Gregg]

All,

The OOTB Kafka JMX configuration is incorrect.  Maybe it was correct for an older version of the Kafka client library, but it's not correct any longer.  For example, there are metrics defined using the object name match pattern "kafka.consumer:type=consumer-fetch-manager-metrics,*" such as records-lag-max.  However records-lag-max is not an attribute of any object matching that pattern.

I have fixed this problem manually and disabled the OOTB configuration.  My new configuration uses the object name match pattern "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*,topic=*" and the instance identifier "client-id,topic."  My new metrics do show up in the metric browser.  If the client id is myClientId and the topic is myTopic, the metrics show under JMX|ConsumerMetrics|myClientId|myTopic.

However I cannot seem to use the new metrics in a dashboard.  When adding a time series to a dashboard, I can choose "JMX|ProducerMetrics" or "JMX|ConsumerMetrics" as the JMX objects to include.  But when selecting the actual metric to display, I can't see any of my new metrics.  I can only see the OOTB metrics like Average IO Wait.  When selecting the JMX objects to include or when selecting the metrics to display, I cannot drill down further than ProducerMetrics or ConsumerMetrics, even though there are two levels below (corresponding to the client id and topic.)

thanks

[Ryan.Paredez]

Hi @John.Gregg,

Are you getting this info from an AppD Docs page or something within the product? The more you can share on where this data is found, I can share it with the right team.

[John.Gregg]

I'm not aware of any detailed documentation on how the OOTB JMX metrics are defined.  There is some high-level documentation but it's wrong, too.  Regardless, I think I've fixed the JMX configuration but I can't select the new metrics for inclusion in a dashboard.

[Ryan.Paredez]

Hi @John.Gregg,

Thanks for that info. I have shared that with the Docs team. Hopefully, we can also get an answer to your question too.

[Ryan.Paredez]

Hi @John.Gregg,

I have not heard back from the Docs team yet. You may want to submit this as a bug in a ticket. I sent you a PM about how your account shows up in community too.