This article is intended for anyone who has installed the Apache Agent and has encountered the situation where one or more Nodes has stopped reporting because the BT Threshold was exceeded. This occurs when the number of BTs created within a Node exceeds the hard limit of 50. This will result in all additional BTs created on that Node being sent to the "All Other Traffic" container, and under some circumstances can result in loads not being reported as expected.
It also possible for a new BT to be within the 50 BT threshold for the Node, but to exceed the hard limit of 200 BTs for the Application (the collection of all Tiers and Nodes belonging to it), in which case the BT will also be sent to the "All Other Traffic" container for that Node, even if it is within the 50 BT/Node threshold. The BT Threshold is described in more detail in the AppDynamics Business Transaction documentation.
This situation can also occur when first installing an Apache agent to an already instrumented topology. See here if you already have an AppDynamics-instrumented set of application servers and are about to attempt adding an Apache Agent upstream of them, or, if having just done so, are now not seeing BTs (or load) being reported that was previously working. Some customers run into the BT Overflow condition (also referred to as BT Explosion), when installing an Apache Agent because, unlike other Agents, there may be no "default" Custom Match Rules set out of the box (or the default might be set to use the first two segments of the URL, which will result in fewer BTs being created), but with all segments of the URI being used, you will very quickly overflow the thresholds.
Given that there might be more than 50 unique URLs within the address space that the server (and Agent) services, it is possible for this to occur. as every "unique" URL processed by the Apache Agent will be associated with a Business Transaction (of the same URL name, by default), so while /web/images/image1 and /web/images/image2 will both be treated as the “/web/images” BT, /web/store, and /web/info will each generate a unique, separate BT. When there are only a limited number of permissible URLs within Apache's address space, this may not be an issue, but for systems where thousands of different URLs could be served, and there are no default or Custom Match Rules, this can quickly exceed the BT threshold.
There are several ways to determine if you've already exceeded the BT limit for a given Node. From the Controller UI, select "Business Transactions". Make sure to unset any previously added Filters, and that all checkboxes are empty, and especially, to uncheck the box for "Transactions with Performance Data", to insure that all BTs are correctly listed.
Set the timespan to "last 1 month", then look at the displayed count on the upper righthand corner of the screen. It will say something along the lines of "Showing N of M", where N is the number of BTs that match the (default) filter, and M is the total number of available BTs for the current timeframe. If N is greater than 50, or if M is greater than 200, you have exceeded the BT Threshold, and need to revise the custom match rules.
The best way to avoid a BT Overflow situation is to create appropriate Custom Match Rules, which allow one to group various URLs into a single BT, and to insure that there are at most no more than 50 possible BTs generated within an Apache Agent for a given Node. See Avoiding Overflow With Earlier Agents if you are using a 4.2 or older Agent.
Custom match rules can be created using regular expression search patterns, with both inclusion and exclusion attributes, and tailored to meet the needs of most circumstances. See here for more details on how to create a Custom Match Rule.
Best Practice is to create a rule that matches the first few segments of the URL only, (or some generic portion of the URL that ties a logical group of URLs together) allowing all of the URLs to be represented by a single BT.
Start by selecting the Configuration ⇒ Instrumentation menus, then select the “Transaction Detection” tab, then select the Rules subtab. Drill down on the “Web Server” entry.
This will allow you to access the Rule Editor. Create a series of Rules that will allow no more than 50 possible BTs. You can name each Rule separately, and assign it a priority. The Rules will be applied in order of priority, with the highest priority first.
Turn on Transaction Monitoring, and configure the UI to Discover Transactions automatically for all HTTP web requests monitored by the Agent.
You can also create Exclude Rules with will ignore certain URLs. Exclude Rules are executed after Include Rules, i.e. have a lower priority.
If you are using the older 4.2.x Agent, you'll access the custom match rule mechanism by going to Configuration==>Instrumentation, then selecting "Transaction Detection", then highlight the Application level (in this example it is called "MyApache2-188.8.131.52App", then selecting the "Web Server" tab on the far right.
You should then see a menu that displays a black icon with the blue character "<>" within it. Make sure that the Transaction Monitoring "Enabled" checkbox is checked, as well as the other two check boxes for "Discover Transactions automatically..." and "Configure Naming"
Next create a new rule by clicking on the "+" icon. Create a custom match rule (in the example I used "Default Catch All") that has a priority of zero, and matches any URL that begins with "/". This will match every URL that comes into the server, and send them to a BT labeled "Default Catch All".
Then create additional custom match rules, that have a higher priority (say 10) that will match specific patterns of URLs. A common approach is to use some number of URL segments (usually 2) as a starting point, then refining the match rules as you become more familiar with which URLs should be grouped together.
To reduce the number of BTs created within an Apache Node you will need to first create a series of Custom Match Rules that either group URLs into BTs and/or exclude specific URLs (see above). Once you have created the appropriate rules, you’ll need to delete those BTs that were already created (either by default or from previous rules that were scoped too broadly) and are overflowing the BT Threshold.
Starting at the Business Transactios menu, drill down into the “All Other Traffic” item for the Apache Agent Node; look for a black rectangular icon with white characters “...>” inside it. Make sure that there are no Filters turned on, so that all BTs are displayed.
Click on the “View Traffic Details” selection at the top right of the screen to see a list of BTs in the All Other Traffic view. If there are any items listed, you can highlight on the individual line items and drill down to see their FlowMap, or right click to select other options.
Having a sense of which URLs are being sent to the All Other Traffic container can help determine how best to create your Custom Match Rules.
Delete the BTs that were already (erroneously) created by selecting (highlighting) one or more BT line items in the overflow list, then right clicking, and selecting "Delete".
If a BT (listed within All Other Traffic) is instead one you’d like to keep, you can promote that BT by highlighting the item and then pressing the “Register” button (large plus sign), which will automatically register the BT, providing there is room. i.e. You haven’t exceeded the BT threshold.
If you are running on an active server that can not be cycled at the moment and new URLs are continually being registered as BTs, you might also consider enabling BT Lockdown to prevent deleted BTs from automatically re-registering. This is an application-wide parameter that prevents the creation of any "new" BTs, and instead sends them to the "All Other Traffic" container, if they aren’t already registered. URLs that are already registered (or part of previously defined BT match rule that was registered) will continue to be reported as before, but any new, unregistered URLs will be sent to All Other Traffic.
You can get to the BT Lockdown screen from the Configuration ⇒ Instrumentation menus. Select “Transaction Detection” and then “More” (version 4.3 Controller) or select the Application (version 4.2 Controller).
Be aware that enabling BT Lockdown will prevent the snapshots of BTs that are sent to All Other Traffic container; if you wish to see snapshots for all traffic, even those in overflow, and you have BT Lockdown enabled, then create a default “Catch All” Custom Match Rule (that matches all URLs), with a priority of 0 (lowest), so that any URLs that would have been sent to All Other Traffic are instead contained within it, and will include snapshots.