How do I resolve an error when reaching the agent metric registration limit?

[Rajendra.Nautiyal]

What should I consider when resolving agent metric registration limits?

In this article...

• Why do I see this error and what can I do about it? AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED 
• How can we find the custom metric with an incorrect configuration? 
• How can we determine the correlation between metric size and CPU utilization?
• Can other system utilization also be affected by configuration?

Why do I see this error and what can I do about it?  AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED

When an agent tries to register more than 5000 metrics, it will fire the error below to the Controller. The default limit is 5,000 per session and once it has been met, the agent will not register new metrics. 

NOTE | A new session begins at the last restart 

This error is most often seen when users try to register Custom Metrics with incorrect configurations that consume the metric limit.

AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED

To resolve the AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED error message:

1. First, delete unwanted business transactions, information points, service endpoints, and backends.
   
   
2. If all of the metrics are still required, you can increase the agent metric registration limit using the system property below and restarting the agent.
   
   -Dappdynamics.agent.maxMetrics=<IntegerNumber>
   
   Use caution when changing a metric registration limit. A drastic increase in the agent limit can affect the Controller.
   
   It is highly recommended that you check your current CPU and memory utilization from the application side before increasing the metric registration limit.

3. Restart the agent.
   

NOTE | Identify which metrics are registered, and focus only on relevant metrics. Increasing the limits every time you encounter this error can cause serious performance issues.

How can we find the custom metric with an incorrect configuration?

There is no single place to look for an incorrect configuration of the custom metric. If you find you’re hitting the backend limit, check the backend configuration. 

Similarly, if you’re exceeding limits in SEP, BT, etc., look in those places for how you might adjust or reconfigure them to bring your numbers down.

When increasing the maximum metric size, how can we anticipate how large the increase to higher CPU utilization will be?

Increasing the maximum metric size can cause high CPU utilization, but only at a very high value. The Controller also has a metric limit of 1M per Controller application, including all its tiers and nodes. 

As a rule of thumb, things should work fine if you keep below 1M per Controller.

Can utilization of the main Controller also be affected by configuration?

If the Controller application metric limit is exceeding limits even after you've increased those limits, then the Controller level can also experience issues as a result.

[Fernando.Villava]

Regarding this matter, we have an extra large Appdynamics setup and we are finding many metric reg limit exceeded.

The question, reading your manuals is:

Which most common errors can be seen as incorrect, that produces exceeded limits?

Thanks for your kind answer.

[Rajendra.Nautiyal]

@Fernando.Villava The limit you are seeing here is the agent side limit, so the agent will send or register up to 5000 metrics to the Controller and after that, if you are trying to register any BT, Backend, SEP agent will not register those metrics to the Controller UI.
and you will see the agent limit reached message on agent logs as well on Controller UI.

Now there are so many reasons for reaching the limit

1) You are monitoring so many BT's and due to that agent is reaching the limit.
2) You are creating so many backends due to that you are reaching the limit.
3) You are registering too many mbean from the same node, and due to that you are reaching the limit.

How to resolve 

1) If you have sufficient heap space and your CPU utilization is normal you can increase the agent metric registration limit using below mentioned property

-Dappdynamics.agent.maxMetrics=<IntegerNumber>

2) You can group your BT/backend in one backend/BT so you can control the metric limit

https://docs.appdynamics.com/display/PRO45/Backend+Detection+Rules#BackendDetectionRules-all_other_t...

https://docs.appdynamics.com/display/PRO45/Business+Transactions#BusinessTransactions-RefineBusiness...

Regards,

Rajendra

10/15/23 Edits to remove extra spaces, minor typographical errors. C. Landivar

[Anonymous]

We have a large AppD installation and find that when this occurs, a business transaction detection rule is configured one more more levels too deep and is catching arguments of transactions as new transactions.

For example:

/getCustomer/customer1

/getCustomer/customer2

...

/getCustomer/customerN

shows us N business transactions when, of course, it's only 1 transaction. This is evidenced in the business transaction tab for the application showing those as the transaction names. We eliminate this problem by tweaking the configuration that catches these transactions to back up one layer or more until it only registers /getCustomer and then deleting the other ones.

[Claudia.Landivar]

@Anonymous, thanks for adding that helpful tweak ^ that resolves the BT's catching arguments of transactions as new transactions!

Claudia.Landivar

Community Manager & Editor

[Basit.Amjad]

Hi AppD community

We really need a solid solution for this. We are seeing this issue on our platform as well. The comments/answers above do not help. Can you just give us actionable steps that we can use every month as a maintenance activity if this is something that we need to do on a regular basis?

Thanks

Basit