How do I resolve the error "[EVENT_UPSERT], you have reached the events limit of [25000000] for [biz_txn_v.*|custom_.*]"?

[#1]

Symptoms

Transaction Analytics stops getting data and you see the types of errors listed below in the analytics-agent.log file:

[2018-04-23T21:54:17,448+01:00]  [WARN ]  [pipeline-thread-14 (bt-publish-6)]  [c.a.a.a.p.e.EventServicePublishStage]  Transient error encountered due to the following cause: [Message could not be delivered because the REST resource rejected it]com.appdynamics.analytics.client.common.exceptions.NotAcceptableRestException: Status code: [406], Message: For action [EVENT_UPSERT], you have reached the events limit of [25000000]  for account [xyz_bc1420f4-b665-4393-a4bb-e866dc948682] and event type group [biz_txn_v.*|custom_.*]

[2018-04-23T21:54:17,448+01:00]  [WARN ]  [pipeline-thread-14 (bt-publish-6)]  [c.a.a.a.p.e.EventServicePublishStage]  This message was throttled [13] times: Transient error encountered due to the following cause: [Message could not be delivered because the REST resource rejected it]

com.appdynamics.analytics.client.common.exceptions.NotAcceptableRestException: Status code: [406], Message: For action [EVENT_UPSERT], you have reached the events limit of [25000000]  for account [xyz_bc1420f4-b665-4393-a4bb-e866dc948682] and event type group [biz_txn_v.*|custom_.*]

Diagnosis

As we know, Transaction Analytics are measured using a specific number of Business Transaction (BT) events. We have a daily limit on the number of events that can be reported into the Events Service (ES) and it gets reset every 24 hours, allowing a fresh set of 5 units to be ingested into Transaction Analytics. However, once this daily limit is reached, the ES will stop ingesting any further events from the agents. For more Information on analytics license entitlements, refer to this link:

https://docs.appdynamics.com/display/PRO44/License+Entitlements+and+Restrictions 

You can use the guidelines in the KB article below to find out how many Transaction Analytics events are you publishing:

https://community.appdynamics.com/t5/Knowledge-Base/How-many-business-transaction-analytic-events-ar...

Solution

To get past this issue, you need to choose one of the options below:

1) Review the traffic and remove the unimportant BTs from reporting into the Events Service. Essentially, you need to remove additional BTs from the Analytics configuration in order to bring down the total to within your daily limit. You can choose other business transactions to remove, but most effective ones to remove are the transactions that have high loads.

2) As a long term solution, you can increase the number of Transaction Analytics units in the license to allow for more events to be ingested. If you opt for this option, we suggest that you get in touch with your sales rep and work on this as this is going to be a license entitlement thing. Perhaps they can get you temporary units for a couple days to see what is the more realistic amount needed, and then you can decide how to proceed.