AppDynamics collects Application Diagnostic Data (ADD) with limits in place to prevent overhead. If any ADD limit is hit, either on the agent or on the Controller, a notification will appear and that particular type of ADD will not be registered. Therefore, it will not be tracked and report metrics, which can impact application visibility. It’s very important that you not ignore these notifications.
For larger deployments with hundreds or thousands of agents and varied types of applications, the default limits are often miniscule. You can adjust the limits by following the remediation steps below. Keep in mind that defining a high value for a property might add overhead, so be cautious.
Following are the ADD Limit Notifications, by type. Click the notification link of each to learn about its context.
Controller ADD Limit Notifications
Agent ADD Limit Notifications
The following are explanations for each ADD Limit notification type identified in the previous section.
There are many thread-related classes that undergo async instrumentation for multi-threaded correlation. However, there is a default limit (1,000) for the number of async threads that a Controller can accept. This notification indicates that the limit for registering async ADDs for this account has been reached and no further async ADD registration will be accepted. If you have a highly multi-threaded environment, you could come across this issue frequently. In general, you can resolve this notification by:
The limit for registering StackTrace ADDs for the account has been reached. No further StackTrace ADD registrations will be accepted. In other words, new call graph elements (as seen in snapshots) will NOT be registered and reported in snapshots. These ADDs usually reach their limits due to too many errors and exceptions OR too many unique call graphs and elements being discovered. The setting name is
stacktrace.ADD.registration.limit and the default value is 4,000.
The limit for registering Service Endpoints (SEP) ADDs for this account has been reached and no further SEP ADD registration will be accepted and reported. The setting name is
sep.ADD.registration.limit and the default value is 4,000.
The Agent ADD blacklist registration limit has been reached. If the agent attempts to register an ADD above the limit, the Controller will reject the attempt and add it to a blacklist. A total of 500 unregistered ADDs can be added to the blacklist queue and this notification indicates that you have reached the limit. This is a per agent limit.
Note: The Agent ADD blacklist limit notification is mostly a result of one or more Controller ADD limits being reached. Therefore, after any Controller ADD limit notification, it will take a while for the Agent ADD blacklist limit to be reached. This is because it holds the unregistered ADDs in its backlist queue until it reaches its 500 limit.
The agent sync ADD registration limit has been reached. This limit may be reached faster if the applications are highly multi-threaded. The node-level agent property is max-async-task-registrations-allowed and the default value is 500.
The following remediation steps apply to:
When these notifications are observed, log in to the Controller's Admin Console and increase the setting that governs that particular type of ADD:
ADD limits can be adjusted at both controller- and account-levels. Please see How do I increase Controller and Agent ADD limits? for instructions.
As mentioned above, this notification is the result of one or more of the Controller's ADD type limits being reached. You’ll need to determine which one was reached for that account:
SELECT t4.name as account_name, t1.diagnostic_type as ADD_type, COUNT(*) as current_ADD_count FROM application_diagnostic_data as t1, application_component as t2, application as t3, account t4 WHERE t1.entity_id=t2.id AND t1.diagnostic_type IN ('STACK_TRACE','ERROR','SERVICE_ENDPOINT', 'THREAD_TASK') AND t2.application_id=t3.id AND t3.account_id = t4.id and t4.name != 'system' GROUP BY t4.name, t1.diagnostic_type;
current_ADD_countvalue for a particular ADD type. Compare it to the corresponding setting’s limit value to confirm which ADD type has exceeded the limit.
It may be the case that the SQL counts are less than the set limit. This can happen when the Controller's ADD limit is reached and the agent starts accumulating those unregistered ADDs. Then, the Controller limit is increased but that limit increase message isn’t sent to the agent. In this case, you should do an agent RESET.
After any of the actions performed (limit increased, agent RESET, SEPs cleanup etc), the existing notifications should be cleared.
You can adjust this value by setting the node level agent property max-async-task-registrations-allowed. Since this property is an unlisted node property, it must be added as a new agent property. The default value is 500.For additional information on how to add and edit registered node properties, see App Agent Node Properties.
The purger is currently only available for StackTrace ADDs, which are registered in high volume and associated with snapshots. Snapshots are retained for a limited time (default 14 days), so it's possible that many stack traces (call elements) are no longer referenced in active (non-deleted) snapshots. However, StackTrace ADDs are not deleted and remain in the system, and are thereby counted in the overall StackTrace ADD limit. Therefore, it is important that unused or unreferenced StackTrace ADDs are automatically deleted from the system to keep the overall limit in check. The ADD purger was implemented with the following governing properties:
If there are thousands of StackTrace ADDs, set
application.diagnostic.data.purge.enabled = true from Controller's Admin Console. This change does not require a Controller restart and it should delete the unreferenced ADDs. The purger thread runs every 12 hours.
If you are interested in learning about the many types of events and custom system notifications generated in AppDynamics, we’ve documented them in the resources below: