When an agent tries to register more than 5000 metrics, it will fire the error below to the Controller. The default limit is 5,000 per session and once it has been met, the agent will not register new metrics.
NOTE: A new session begins at the last restart
This error is most often seen when users try to register Custom Metrics with incorrect configurations that consume the metric limit.
Note: Identify which metrics are registered, and focus only on relevant metrics. Increasing the limits every time you encounter this error can cause serious performance issues.
There is no single place to look for an incorrect configuration of the custom metric. If you find you’re hitting the backend limit, check the backend configuration.
Similarly, if you’re exceeding limits in SEP, BT, etc., look in those places for how you might adjust or reconfigure them to bring your numbers down.
Increasing the maximum metric size can cause high CPU utilization, but only at a very high value. The Controller also has a metric limit of 1M per Controller application, including all its tiers and nodes.
As a rule of thumb, things should work fine if you keep below 1M per Controller.
If the Controller application metric limit is exceeding limits even after you've increased those limits, then the Controller level can also experience issues as a result.