What does this Controller or Agent ADD limit notification mean?
AppDynamics collects Application Diagnostic Data (ADD) with limits in place to prevent overhead. If any ADD limit is hit, either on the agent or on the Controller, a notification will appear and that particular type of ADD will not be registered. Therefore, it will not be tracked and report metrics, which can impact application visibility. It’s very important that you not ignore these notifications.
For larger deployments with hundreds or thousands of agents and varied types of applications, the default limits are often miniscule. You can adjust the limits by following the remediation steps below. Keep in mind that defining a high value for a property might add overhead, so be cautious.
Contents
Application Diagnostic Data (ADD) Limit Notifications
Following are the ADD Limit Notifications, by type. Click each notification's link below to learn about its context.
| Controller ADD Limit Notifications |
|
| Agent ADD Limit Notifications |
|
Explanations by ADD Limit Notification Type
The following explanations correspond to each of the ADD Limit notification types identified in the previous section.
CONTROLLER_ASYNC_ADD_REG_LIMIT_REACHED
There are many thread-related classes that undergo async instrumentation for multi-threaded correlation. However, there is a default limit (1,000) for the number of async threads that a Controller can accept. This notification indicates that the limit for registering async ADDs for this account has been reached and no further async ADD registration will be accepted.
If you have a highly multi-threaded environment, you could come across this issue frequently. In general, you can resolve this notification by:
- Increasing the default limit.
- Excluding the thread-related packages on the nodes from which excessive async instrumentation is done. Contact the Support team and send the zipped agent logs folder from the suspected nodes where there is heavy async instrumentation.
CONTROLLER_STACKTRACE_ADD_REG_LIMIT_REACHED
The limit for registering StackTrace ADDs for the account has been reached. No further StackTrace ADD registrations will be accepted. In other words, new call graph elements (as seen in snapshots) will NOT be registered and reported in snapshots.
These ADDs usually reach their limits due to too many errors and exceptions OR too many unique call graphs and elements being discovered. The setting name is stacktrace.ADD.registration.limit and the default value is 4,000.
CONTROLLER_SEP_ADD_REG_LIMIT_REACHED
The limit for registering Service Endpoints (SEP) ADDs for this account has been reached and no further SEP ADD registration will be accepted and reported.
The setting name is sep.ADD.registration.limit and the default value is 4,000.
AGENT_ADD_BLACKLIST_REG_LIMIT_REACHED
The Agent ADD blacklist registration limit has been reached. If the agent attempts to register an ADD above the limit, the Controller will reject the attempt and add it to a blacklist. A total of 500 unregistered ADDs can be added to the blacklist queue and this notification indicates that you have reached the limit. This is a per agent limit.
NOTE | The Agent ADD blacklist limit notification is mostly a result of one or more Controller ADD limits being reached. Therefore, after any Controller ADD limit notification, it will take a while for the Agent ADD blacklist limit to be reached. This is because it holds the unregistered ADDs in its backlist queue until it reaches its 500 ADD limit.
AGENT_ASYNC_ADD_REG_LIMIT_REACHED
The agent sync ADD registration limit has been reached. This limit may be reached faster if the applications are highly multi-threaded. The node-level agent property is max-async-task-registrations-allowed and the default value is 500.
Remediation Steps
The following remediation steps apply to:
Controller ADD Limit Notifications
When these notifications are observed, log in to the Controller's Admin Console and increase the setting that governs that particular type of ADD:
- ASYNC ADD: Increase by 1,000 (small profiles), 2,000 (medium profiles), 4,000 (large+ profiles). The property name is
async.thread.tracking.registration.limit
- STACKTRACE ADD: Increase by 25,000 (small profiles), 50,000 (medium profiles), or 100,000 (large+ profiles). The property name is
stacktrace.ADD.registration.limit
- SEP ADD: SEPs are like Business Transactions. Therefore, the data uploaded for each of the SEP is large in size. Before increasing the SEP ADD limit, the existing SEPs should be analyzed and unnecessary SEPs should be deleted and the default/auto-discovery SEP config should be fine-tuned. Once this is done, if there is further need to increase the limit, increase by 2,000 (small or medium profiles) or 4,000 (large+ profiles). The property name is
sep.ADD.registration.limit
ADD limits can be adjusted at both Controller- and account-levels. Please see How do I increase Controller and Agent ADD limits? for instructions.
Agent ADD Blacklist Limit Notification
As mentioned above, this notification is the result of one or more of the Controller's ADD type limits being reached. You’ll need to determine which one was reached for that account:
- Log in to the Controller DB and run the following SQL:
SELECT t4.name as account_name, t1.diagnostic_type as ADD_type, COUNT(*) as current_ADD_count FROM application_diagnostic_data as t1, application_component as t2, application as t3, account t4 WHERE t1.entity_id=t2.id AND t1.diagnostic_type IN ('STACK_TRACE','ERROR','SERVICE_ENDPOINT', 'THREAD_TASK') AND t2.application_id=t3.id AND t3.account_id = t4.id and t4.name != 'system' GROUP BY t4.name, t1.diagnostic_type;
- Look at the SQL results column and the
current_ADD_count value for a particular ADD type. Compare it to the corresponding setting’s limit value to confirm which ADD type has exceeded the limit.
- Follow the steps outlined above to remediate ADD limits on the Controller.
- Delete the existing notification events after applying the remediation and reset the agents that reported those blacklist notifications. It’s better to reset all the agents at once to clear the agents' cache via the AppDynamics Agents screen.
IMPORTANT NOTES | It may be the case that the SQL counts are less than the set limit. This can happen when the Controller's ADD limit is reached and the agent starts accumulating those unregistered ADDs. Then, the Controller limit is increased but that limit increase message isn’t sent to the agent. In this case, you should do an agent RESET.
After any of the actions performed (limit increased, agent RESET, SEPs cleanup etc), the existing notifications should be cleared.
AGENT_ASYNC_ADD_REG_LIMIT_REACHED
You can adjust this value by setting the node level agent property max-async-task-registrations-allowed. Since this property is an unlisted node property, it must be added as a new agent property. The default value is 500.For additional information on how to add and edit registered node properties, see App Agent Node Properties.
ADD Purger
The purger is currently only available for StackTrace ADDs, which are registered in high volume and associated with snapshots. Snapshots are retained for a limited time (default 14 days), so it's possible that many stack traces (call elements) are no longer referenced in active (non-deleted) snapshots. However, StackTrace ADDs are not deleted and remain in the system, and are thereby counted in the overall StackTrace ADD limit. Therefore, it is important that unused or unreferenced StackTrace ADDs are automatically deleted from the system to keep the overall limit in check. The ADD purger was implemented with the following governing properties:
- application.diagnostic.data.purge.enabled
(Note: the default is false, you so will need to enable it.)
- stacktrace.application.diagnostic.data.purge.frequency.in.minutes
(Note: the default 720 minutes)
If there are thousands of StackTrace ADDs, set application.diagnostic.data.purge.enabled = true from Controller's Admin Console. This change does not require a Controller restart and it should delete the unreferenced ADDs. The purger thread runs every 12 hours.
For SaaS Controllers:
- Contact Support for assistance.
- For SEP ADDs, complete the SEPs cleanup before contacting Support if possible.
Additional Information
If you are interested in learning about the many types of events and custom system notifications generated in AppDynamics, we’ve documented them in the resources below:
