When an agent tries to register more than 5000 metrics, it will fire the error below to the Controller. The default limit is 5,000 per session and once it has been met, the agent will not register new metrics.
NOTE: A new session begins at the last restart
This error is most often seen when users try to register Custom Metrics with incorrect configurations that consume the metric limit.
To resolve the AGENT_METRIC_BLACKLIST_REG_LIMIT_REACHED error message:
First, delete unwanted business transactions, information points, service endpoints, and backends.
If all of the metrics are still required, you can increase the agent metric registration limit using the system property below and restarting the agent.
Use caution when changing a metric registration limit. A drastic increase in the agent limit can affect the Controller.
It is highly recommended that you check your current CPU and memory utilization from the application side before increasing the metric registration limit.
Restart the agent.
Note: Identify which metrics are registered, and focus only on relevant metrics. Increasing the limits every time you encounter this error can cause serious performance issues.
How can we find the custom metric with an incorrect configuration?
There is no single place to look for an incorrect configuration of the custom metric. If you find you’re hitting the backend limit, check the backend configuration. Similarly, if you’re exceeding limits in SEP, BT, etc, look in those places for how you might adjust or reconfigure them to bring your numbers down.
When increasing the maximum metric size, how can we anticipate how large the increase to higher CPU utilization will be?
Increasing the maximum metric size can cause high CPU utilization, but only at a very high value. The Controller also has a metric limit of 1M per Controller application, including all its tiers and nodes.
As a rule of thumb, things should work fine if you keep below 1M per Controller.
Can utilization of the main Controller also be affected by configuration?
If the Controller application metric limit is exceeding limits even after you've increased those limits, then the Controller level can also experience issues as a result.