How do I install the Apache Agent within an existing AppDynamic-instrumented topology?

[Robb.Kane]

This article is intended for anyone who is about to install the Apache Agent as an upstream (entry point within your topology) web server. It may also prove useful for anyone who has already done so and is now unable to see Business Transactions being generated from Apache (usually due to the BT thresholds being exceeded).

Content

• Key Contepts
• Business Transaction Limits
• Setting up Health Rules
• Business Transaction Lockdown

Key Concepts

We use the Business Transaction model, which means that we track the entire user transaction through the network topology. In other words, the path that the request took, and how long it took.

It is important to remember that a BT is a "label" and represents a request or a group of associated requests. For instance, you might have a "UserLogin" BT, that is used to track requests that are associated with the "/login" URL. You might also have a "StatusRequest" BT, that is used to track any user status request URLs (perhaps you've created a Custom Match Rule regular expression that matches all URLs that begin with "/system/status"). These are examples of two BTs, but they could represent lots of URL requests the Apache server responds to, and could constitute thousands of individual requests from hundreds of users within a given timeframe.

When displayed within a flow-map, the BT will be shown as having been created by the "originating tier" (i.e. the first AppDynamics instrumented server it encountered) and is then correlated with other instrumented servers as that request migrates through the network topology.

Business Transaction Limits

There is a limit on the number of BTs that any given Node can create (50), and a limit on how many can be represented within an Application (200). After either of these limits are reached, any new BTs will be sent to the "AllOtherTraffic" container for the Node on which it occurred.

Out of the box, the Apache Agent will treat any new URL request as a new BT. This will very quickly exceed the node limit for Apache if your request stream uses lots of slightly different URL names. It can also exceed the application limit if your previously instrumented downstream servers were already at or near that threshold.

If that request was previously being captured as a BT from a downstream server, it will now originate with Apache but be sent to the "AllOtherTraffic" container. This is why when adding an Apache Agent to an existing (fully-instrumented) topology, you may sometimes stop seeing load being reported from a server downstream of it (the Apache Agent has now started reporting the URL that was previously originating at it), or not see load reported from the Apache server itself (because you exceeded the Node threshold).

What is important about that idea is that the first Agent (i.e. any server on which you have installed and enabled our APM software) will be considered the "Originating Tier," and add a Singularity Header to all subsequent (downstream) server requests. This additional header will be used by the downstream tiers to correlate the requests and display a flow-map, so if the Apache (upstream) Node has exceeded its threshold, then the previously reporting downstream tiers/nodes may stop showing that BT.

The total number of BTs in effect will be listed in the top right corner of the Business Transactions page, and given based on the time frame specified, for the Application, Tier, Node, or BT in question.  Click on the Business Transactions page for the Apache Node to see the current total number of BTs created.   This value should not exceed 50 for the Node, or 200 for the Application.

 

To avoid this situation, it is important to review the current BT levels. Take a look at the Business Transactions pages for the entire application, and then for each node of each tier, choose a time frame longer than 1 week. If, in the upper right-hand corner of each page (highlighted in red), you see a message like "showing N transactions of M" (where either N or M are values that exceed either the node or application, depending on what you are viewing) then you'll need to revise the match rules to limit the BTs created (keeping it within 50 for each Node, and 200 for the Application as a whole), delete any BTs that were created inadvertently, and promote any BTs from within the "AllOtherTraffic" container that don't belong there.

Setting up Health Rules

In a properly running system, the AllOtherTraffic containers should remain empty.   If you see BTs being sent there, then you need to reconfigure. The best way to do this (for Apache) is to stop the web server, then stop the proxy task, then from the UI, create the appropriate Custom Match Rules such that all of the BTs created (within Apache) do not exceed 50.

Select Configuration, then Instrumentation, then RulesWithin Rules, Select WebServer, then right click to edit, then Rule Configuration 

You'll also want to go into the Business Transactions page, then select "AllOtherTraffic" container, and then select the "View All Traffic" link, which will display the list of BTs within that container. Either delete or promote these items as necessary. Once you've created the match rules that cover all of the URLs you want to track, you may want to create a rule that acts as a "CatchAllOther" BT. This rule should have the lowest priority, and match any URL. Any URL request that doesn't match a prior rule will go into this BT. You can then set up a health rule alert on this BT to notify you when a new URL is discovered that doesn't match any prior rule, to decide which BT is should be included in. 

Once you've properly set up your rules, restart the proxy and the Apache server, and run load on your system. You should see BTs being created as configured, with Apache cited as the "originating tier" (this is best seen from the "Transaction Snapshots" page, selecting a line item then right clicking to "View Details", then selecting the "Waterfall View"); the downstream tiers should correlate to them.

In a similiar manner, you can check that the "AllOtherTraffic" containers are empty and that the "CatchAllOthers" BT doesn't contain anything you aren't interested in tracking. If everything doesn't report as expected, review your match rules for each tier, and repeat the start/stop steps described above. Remember that there is some propagation delay when changing configurations/match rules, and allow a few minutes for these changes to take effect across all nodes/tiers within the application.

Business Transaction Lockdown 

If you'd rather not use a catch-all BT, try enabling the "Business Transaction Lockdown" checkbox. This feature prevents any new BTs from being created by default. Having this set prevents unexpected BTs from exceeding the thresholds, and prevents any new BTs from being created by default. To find this feature, select Configuration > Instrumentation.  If you are using a 4.2.x version of the Controller, select the Application level within the tiers/nodes list and then check the checkbox.   If you are using a 4.3.x version of the Controller, select the "More" option.   In either case, remember that BT Lockdown applies at the Application Level.

Select Configuration, then Instrumentation, then More

When you enable this feature, any new URLs that don't match an existing rule will be sent to the "AllOtherTraffic" container directly, regardless of the current state of BTs and their thresholds.