Discussion Feed

  This article is intended for anyone installing the Apache Agent.  Please make sure the following 11 prerequisites are in place before installing the agent:   Insure that SELinux is not active. Issue the command sestatus at the host command prompt. If SELinux is active and in enforcing mode, you will need to disable it before continuing. Use the command  sudo setenforce 0 to temporarily disable SELinux. NOTE: If your system uses SELInux, you will need to create an appropriate policy file in order to allow the Agent and Proxy task to access the necessary log and configuration files. Determine what the Apacher mode is. Apache can be configured with a number of different modes, including Event, PreFork, or Worker mode. See here for more details on determining the mode your system is configured for: https://www.tecmint.com/check-apache-modules-enabled/  If you are running in Worker mode, make sure the ServerLimit and ThreadsPerChild directives are valid. The product of the two (when you multiple the two directive settings together) must be less than 4033. Update the httpd.conf (or the apache2.conf file) to reflect this. The last line of the httpd.conf file must include the appdynamics_agent.conf file explicitly.  DO NOT RELY on metacharacters or global matching; the file must not be located where other include statements might apply. Determine the Apache worker process userid. With the Apache server running, issue the command  ps -ef | grep httpd  (use apache2 if httpd is not the name of the server binary). You will see a "root" process id and several "worker" child processes listed in the output.  The userid associated with the children of the parent process is the Apache worker process id. This can be set explicitly with the Apache User directive.  See here for more details: https://httpd.apache.org/docs/2.4/mod/mod_unixd.html#user  Make sure that the number of available file descriptors is at least 10000 (10K) for both the Proxy and the Aapche worker process userids. Edit the  /etc/security/limits.conf Edit file to verify that this is the case.  There should be both a hard and soft setting for the nofiles entries for the appropriate userid(s). Insure that the Apache tier has not been previously registered as the incorrect type. If there is already a tier that matches the AppDynamicsTierName value, and its type is not WebServer (i.e. the icon shown isn't a green and blue globe in front of a gray server rack) then most likely the Machine Agent has incorrectly registed the tier. Stop the Machine Agent & Proxy task, delete the tier at the Controller UI, then restart the Proxy to allow it to register the tier as the appropriate type (Apache). Once the tier is registered, restart the Machine Agent. You will want to make sure that the Apache tiers are first registered before using the Machine Agent, but once the tier is registered, you can start both agents in any order. The default (OOTB) match rule for BTs will be in effect, which creates BTs based on the first two segments of the URL. Once the Agent begins reporting and there is load on the server, you will see BTs register. To avoid BT Overflow, you'll want to make sure that you create the appropriate custom match rules for the Apache agent. For more details on avoiding BT Overflow, please see the KB article here: https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-avoid-quot-BT-OverFlow-quot-within-the-Apache-Agent/ta-p/27889 Make sure that the Proxy command has been issued. Use the command  ps -ef | grep proxy | grep WEB_SERVER | grep -v grep  to make sure that there is only one Proxy active. Our best practices suggest you run the Proxy as the same userid as the Apache server worker process. At a minimum, the Proxy task should be started at host boot up and remain active through out the life of the host. There should be no need to restart the Proxy task unless the appdynamics_agent.conf file changes. It is possible to stop/start the webserver without affecting it. Please see here for more details about launching the Proxy: https://docs.appdynamics.com/display/PRO45/Install+the+Apache+Agent#InstalltheApacheAgent-LaunchtheProxy  The default (OOTB) match rule for BTs will be in effect, which creates BTs based on the first two segments of the URL. Once the Agent begins reporting, and there is load on the server, you will see BTs register.  To avoid BT Overflow, you'll want to make sure that you create the appropriate custom match rules for the Apache agent.  For more details on avoiding BT Overflow, pleas see the KB article here: https://community.appdynamics.com/t5/Knowledge-Base/How-do-I-avoid-quot-BT-OverFlow-quot-within-the-Apache-Agent/ta-p/27889  Apply load to the system. Note that there is a several minute delay between the Agent and the Controller. Make sure the time limit is set to "Last 5 minute" and that you wait at least several minutes for information to be reflected. Any changes made to the configuration or Controller UI will require some time to propogate through the system.     ... View more

This article is intended for anyone currently using the 4.3.7 (or earlier) or 4.4.0 version of the AppDynamics C++/C SDK, which uses a separate Proxy (JVM) task, and who wishes to upgrade to the 4.4.1 libagent-based (proxy-less) version of the SDK.   Contents: Why upgrade? Differences between versions New features More information   Why upgrade? There are numerous reasons to do so, the most important is that this newer version represents the current and future SDK architecture.  The next reason is resource consumption.  The older proxy-based version uses a stand-alone JVM, which has a minimum 300M footprint, all of the additional CPU consumption that a JVM requires, the use of socket files (which consume up to 8K worth of open file descriptors), not to mention the necessary coordination of user IDs and file permissions between the Agent and Proxy tasks, and any additional SELinux policy concerns that that introduces.    These reasons all justify moving to the newer SDK architecture, and we strongly encourage you to do so.   Differences between versions The differences between the old and new formats, although not 100% ABI compatible, will be quite easy to navigate.  The major differences in porting to the new version is that the configuration-structure initialization.  In the previous version, you needed to allocate the structure, then initialize each field.  In the newer version, the structure is already allocated, you simply call setter functions to fill in the fields you wish to set.    Presuming you were declaring the fields as const chars (in both versions).  For example: const char APP_NAME[] = "SampleC"; const char TIER_NAME[] = "SampleCTier1"; const char NODE_NAME[] = "SampleCNode1"; const char CONTROLLER_HOST[] = "osxlttyu"; const int CONTROLLER_PORT = 8080; const char CONTROLLER_ACCOUNT[] = "customer1"; const char CONTROLLER_ACCESS_KEY[] = "SJ5b2m7d1$354"; const int CONTROLLER_USE_SSL = 0;    In the older versions (4.2 through 4.3.7 and 4.4.0) within your own code you would declare and allocate an  appd_config struct  and assign the settings to it.   For example:  appd_config cfg; appd_config_init(&cfg); cfg.app_name = (char*)APP_NAME; cfg.tier_name = (char*)TIER_NAME; cfg.node_name = (char*)NODE_NAME; cfg.controller.host = (char*)CONTROLLER_HOST; cfg.controller.port = CONTROLLER_PORT; cfg.controller.account = (char*)CONTROLLER_ACCOUNT; cfg.controller.access_key = (char*)CONTROLLER_ACCESS_KEY; cfg.controller.use_ssl = CONTROLLER_USE_SSL; #ifdef _WIN32 cfg.appd_proxy_config.tcp_control_port = PROXY_CTRL_PORT; cfg.appd_proxy_config.tcp_request_port = PROXY_REQ_PORT; cfg.appd_proxy_config.tcp_reporting_port = PROXY_REP_PORT; #endif cfg.init_timeout_ms = 0;   In the newer SDK (4.3.8 & 4.4.1 and later versions) you would call an initialization routine (which zeros the internally stored structure and returns its address) then call setter methods for each field in that structure.    For example: appd_config* cfg = appd_config_init(); // appd_config_init() resets the configuration object and pass back an handle/pointer appd_config_set_app_name(cfg, APP_NAME); appd_config_set_tier_name(cfg, TIER_NAME); appd_config_set_node_name(cfg, NODE_NAME); appd_config_set_controller_host(cfg, CONTROLLER_HOST); appd_config_set_controller_port(cfg, CONTROLLER_PORT); appd_config_set_controller_account(cfg, CONTROLLER_ACCOUNT); appd_config_set_controller_access_key(cfg, CONTROLLER_ACCESS_KEY); appd_config_set_controller_use_ssl(cfg, CONTROLLER_USE_SSL);   New features   There are a few other newer features within the SDK to be aware of. The first is that you no longer need to start the proxy task, as this is done automatically with the call to appd_sdk_init() , which will spawn a limited number of tasks to handle collating BTs, Metrics, Snapshots, etc., and communicate them to the Controller (all within the same user ID as the calling method).    Additionally, there is no longer any need to worry about file descriptors (we no longer use sockets for the C++/libagent SDK and the NodeJS/libagent-based Agent), although this is still the case for the Apache, Python, PHP, and other dynamics languages agents.  A libagent (proxy-less) version of them are all on our roadmap, speak with your AppDynamics Account Manager for details on when they might be available.   Lastly, there are two separate examples on our GitHub repository: An example C program, at https://github.com/Appdynamics/C_SDK_Example that shows a working program which will create BTs. A parameterized SDKTest utility at https://github.com/Appdynamics/SDKTest.   This sample program, once compiled (it requires boost rev 1.66, see the Makefile for details) will allow you to verify that your environment is configured correctly.     And, as always, if you are having any problems using the C++/C libagent (proxy-less) based SDK, please feel free to submit a help request to us.   More information: Documentation: Using the C/C++ Agent SDK Documentation: C/C++ Agent SDK Reference Ask a question: Dynamic languages discussion board   ... View more

This article discusses the issue with the third-party library know as ZMQ (ZeroMQ) and outlines the steps needed to determine if the known issue with socket handling is occurring.   The AppDynamics (Apache) Agent's current design implements a communication channel between the Agent (which runs within the Apache worker threads) and a Proxy task (self-contained JVM application).  The Proxy task is responsible for batching the data and sending it to the Controller. The Agent communicates with the Proxy via socket files (by default located in the .../logs/appd-sdk directory).  For every worker thread, there are at least two sockets consumed. AppDynamics uses the ZeroMQ socket library as the underpinning mechanism for these connections.   A known problem exists within the rev of ZMQ that we use, whereby a crash can (infrequently) occur when closing the socket. This results in the Apache worker thread exiting prematurely. The in-flight HTML request is lost and the main Apache control thread spawns a new worker thread to replace the thread that died. This behavior is usually seen on very heavily loaded systems (CPU usage > 50%, RAM > 50%) and appears to be mitigated by one or more of the following: increasing the number of cores increasing the available memory reducing the webserver load restricting the number of concurrent Apache worker threads To determine if the isssue is occuring, make sure that the ulimit for the Apache worker thread id (usually "httpd" or "apache2") has its core file size set high enough to capture a core file, that the open-file-descriptors value is greater than 10K, that you have installed the gdb debugger, and are running a symbolicated version of the Agent, as provided by AppDynamics technical support. When a crash occurs, open the file using gdb and produce a stack trace, looking for the term "epoll.c". If you find any thread that has abnormally ended while inside that routine, chances are very high that the problem occurred because of the ZMQ bug.   If you believe this is the case, please submit an APM-Apache support request and include the stack trace output in the ticket, along with the ulimit settings ( /etc/security/limits ) for the Apache worker thread userid.     Under typical operating conditions, the HTTP browser request will be resent. The net effect of this issue is a small increase in Apache overhead, some reduction in webserver through-put, plus any additional disk or CPU consumption if writing core files. ... View more

Issue This article is intended for anyone who is attempting to view the Controller from a browser via an Apache Reverse Proxy and encounters an error due to encoded slashes.  The problem seems to occur when accessing the Server Tabs, which as of the 4.3 Controller, contain encoded slashes within the URL sent back to the browser. Clicking on the URL will generate an error that looks similar to the following:   The requested URL The requested URL /controller/restui/userPreferenceManager/setGeneralUserPreference/SIMMachinesListController-stateId--SIM_MODE-4.3-metricToggles/[{"selected":false,"label":"Availability (%)","type":0,"metricName":"Hardware Resources|Machine|Availability"},{"selected":false,"label":"Availability (%) Trend","type":1,"metricName":"Hardware Resources|Machine|Availability"},{"selected":true,"label":"CPU (%)","type":0,"metricName":"Hardware Resources|CPU|%Busy"},{"selected":true,"label":"CPU (%) Trend","type":1,"metricName":"Hardware Resources|CPU|%Busy"},{"selected":false,"label":"CPU Steal (%)","type":0,"metricName":"Hardware Resources|CPU|%Stolen"},{"selected":false,"label":"CPU Steal (%) Trend","type":1,"metricName":"Hardware Resources|CPU|%Stolen"},{"selected":true,"label":"Disk I/O (%)","type":0,"metricName":"Hardware Resources|Disks|Avg IO Utilization (%)"},{"selected":true,"label":"Disk I/O (%) Trend","type":1,"metricName":"Hardware Resources|Disks|Avg IO Utilization (%)"},{"selected":true,"label":"Disk Usage (%)","type":0,"metricName":"Hardware Resources|Volumes|Used (%)"},{"selected":true,"label":"Disk Usage (%) Trend","type":1,"metricName":"Hardware Resources|Volumes|Used (%)"},{"selected":false,"label":"Load - 1min","type":0,"metricName":"Hardware Resources|Load|Last 1 minute"},{"selected":false,"label":"Load - 1min Trend","type":1,"metricName":"Hardware Resources|Load|Last 1 minute"},{"selected":true,"label":"Memory (%)","type":0,"metricName":"Hardware Resources|Memory|Used %"},{"selected":true,"label":"Memory (%) Trend","type":1,"metricName":"Hardware Resources|Memory|Used %"},   Solution  To avoid this issue, add the Apache directive to the httpd.conf file, and restart the Apache server. You'll need to use version 2.2.18 or later of Apache.   AllowEncodedSlashes NoDecode   Please note that this is an Apache-specific issue and is not related to our Controller or the Apache Agent.  We are providing this information only to help assist you with getting past the issue; we do not offer Apache configuration or tuning support. Please contact your local system adminstrator for assistance.   For additional details, we recommend you review the Apache documentation at http://httpd.apache.org/docs/2.2/mod/core.html#allowencodedslashes   ... View more

Contents BT Oveflow Detecting BT Overflow Avoiding BT Overflow Resolving BT Overflow BT Lockdown   BT Overflow This article is intended for anyone who has installed the Apache Agent and has encountered the situation where one or more Nodes has stopped reporting because the BT Threshold was exceeded. This occurs when the number of BTs created within a Node exceeds the hard limit of 50.  This will result in all additional BTs created on that Node being sent to the " All Other Traffic " container, and under some circumstances can result in loads not being reported as expected.   It also possible for a new BT to be within the 50 BT threshold for the Node, but to exceed the hard limit of 200 BTs for the Application (the collection of all Tiers and Nodes belonging to it), in which case the BT will also be sent to the " All Other Traffic " container for that Node, even if it is within the 50 BT/Node threshold. The BT Threshold is described in more detail in the AppDynamics Business Transaction documentation .     This situation can also occur when first installing an Apache Agent to an already instrumented topology. See here if you already have an AppDynamics-instrumented set of application servers and are about to attempt adding an Apache Agent upstream of them, or, if having just done so, are now not seeing BTs (or load) being reported that was previously working.  Some customers run into the BT Overflow condition (also referred to as BT Explosion), when installing an Apache Agent  because, unlike other Agents, there may be no "default" Custom Match Rules set out of the box (or the default might be set to use the first two segments of the URL, which will result in fewer BTs being created), but with all segments of the URI being used, you will very quickly overflow the thresholds.   Given that there might be more than 50 unique URLs within the address space that the server (and Agent) services, it is possible for this to occur.   as every "unique" URL processed by the Apache Agent will be associated with a Business Transaction (of the same URL name, by default), so while /web/images/image1 and /web/images/image2 will both be treated as the “ /web/images ” BT, /web/store , and /web/info will each generate a unique, separate BT.   When there are only a limited number of permissible URLs within Apache's address space, this may not be an issue, but for systems where thousands of different URLs could be served, and there are no default or Custom Match Rules, this can quickly exceed the BT threshold.   Detecting BT Overflow There are several ways to determine if you've already exceeded the BT limit for a given Node.   From the Controller UI, select "Business Transactions".   Make sure to unset any previously added Filters, and that all checkboxes are empty, and especially, to uncheck the box for "Transactions with Performance Data", to insure that all BTs are correctly listed.   Set the timespan to "last 1 month", then look at the displayed count on the upper righthand corner of the screen.   It will say something along the lines of  " Showing N of M ", where N is the number of BTs that match the (default) filter, and M is the total number of available BTs for the current timeframe.   If N is greater than 50, or if M is greater than 200, you have exceeded the BT Threshold, and need to revise the custom match rules. Avoiding BT Overflow The best way to avoid a BT Overflow situation is to create appropriate Custom Match Rules , which allow one to group various URLs into a single BT, and to insure that there are at most no more than 50 possible BTs generated within an Apache Agent for a given Node.  See Avoiding Overflow With Earlier Agents if you are using a 4.2 or older Agent.   Custom match rules can be created using regular expression search patterns, with both inclusion and exclusion attributes, and tailored to meet the needs of most circumstances.   See here for more details on how to create a Custom Match Rule. Best Practice is to create a rule that matches the first few segments of the URL only, (or some generic portion of the URL that ties a logical group of URLs together) allowing all of the URLs to be represented by a single BT.   Start by selecting the Configuration ⇒ Instrumentation menus, then select the “ Transaction Detection ” tab, then select the Rules subtab.   Drill down on the “ Web Server ” entry.   This will allow you to access the Rule Editor.   Create a series of Rules that will allow no more than 50 possible BTs.  You can name each Rule separately, and assign it a priority.   The Rules will be applied in order of priority, with the highest priority first.   Turn on Transaction Monitoring, and configure the UI to Discover Transactions automatically for all HTTP web requests monitored by the Agent.   You can also create Exclude Rules with will ignore certain URLs.  Exclude Rules are executed after Include Rules, i.e. have a lower priority.   Avoiding Overflow With Earlier Agents If you are using the older 4.2.x Agent, you'll access the custom match rule mechanism by going to Configuration==>Instrumentation, then selecting "Transaction Detection", then highlight the Application level (in this example it is called "MyApache2-4.2.13.1App", then selecting the "Web Server" tab on the far right.   You should then see a menu that displays a black icon with the blue character "<>" within it.  Make sure that the Transaction Monitoring "Enabled" checkbox is checked, as well as the other two check boxes for "Discover Transactions automatically..." and "Configure Naming" Next create a new rule by clicking on the "+" icon.   Create a custom match rule (in the example I used "Default Catch All") that has a priority of zero, and matches any URL that begins with "/".   This will match every URL that comes into the server, and send them to a BT labeled "Default Catch All". Then create additional custom match rules, that have a higher priority (say 10) that will match specific patterns of URLs.  A common approach is to use some number of URL segments (usually 2) as a starting point, then refining the match rules as you become more familiar with which URLs should be grouped together. Resolving BT Overflow To reduce the number of BTs created within an Apache Node you will need to first create a series of Custom Match Rules that either group URLs into BTs and/or exclude specific URLs (see above).     Once you have created the appropriate rules, you’ll need to delete those BTs that were already created (either by default or from previous rules that were scoped too broadly) and are overflowing the BT Threshold. Starting at the Business Transactios menu, drill down into the “ All Other Traffic ” item for the Apache Agent Node; look for a black rectangular icon with white characters “ ...> ” inside it.   Make sure that there are no Filters turned on, so that all BTs are displayed.   Click on the “View Traffic Details” selection at the top right of the screen to see a list of BTs in the All Other Traffic view.  If there are any items listed, you can highlight on the individual line items and drill down to see their FlowMap, or right click to select other options.  Having a sense of which URLs are being sent to the All Other Traffic container can help determine how best to create your Custom Match Rules .     Delete the BTs that were already (erroneously) created by selecting (highlighting) one or more BT line items in the overflow list, then right clicking, and selecting "Delete".   If a BT (listed within All Other Traffic) is instead one you’d like to keep, you can promote that BT by highlighting the item and then pressing the “Register” button (large plus sign), which will automatically register the BT, providing there is room.  i.e. You haven’t exceeded the BT threshold.   BT Lockdown If you are running on an active server that can not be cycled at the moment and new URLs are continually being registered as BTs, you might also consider enabling BT Lockdown to prevent deleted BTs from automatically re-registering.   This is an application-wide parameter that prevents the creation of any "new" BTs, and instead sends them to the " All Other Traffic " container, if they aren’t already registered.   URLs that are already registered (or part of previously defined BT match rule that was registered) will continue to be reported as before, but any new, unregistered URLs will be sent to All Other Traffic .   You can get to the BT Lockdown screen from the Configuration ⇒ Instrumentation menus.  Select “Transaction Detection” and then “More” (version 4.3 Controller) or select the Application (version 4.2 Controller).   Be aware that enabling BT Lockdown will prevent the snapshots of BTs that are sent to All Other Traffic container; if you wish to see snapshots for all traffic, even those in overflow, and you have BT Lockdown enabled, then create a default “Catch All” Custom Match Rule (that matches all URLs), with a priority of 0 (lowest), so that any URLs that would have been sent to All Other Traffic are instead contained within it, and will include snapshots. ... View more

This article is intended for anyone who is about to install the Apache Agent as an upstream (entry point within your topology) web server. It may also prove useful for anyone who has already done so and is now unable to see Business Transactions being generated from Apache (usually due to the BT thresholds being exceeded).   Content Key Contepts Business Transaction Limits Setting up Health Rules Business Transaction Lockdown   Key Concepts We use the Business Transaction model, which means that we track the entire user transaction through the network topology. In other words, the path that the request took, and how long it took.   It is important to remember that a BT is a "label" and represents a request or a group of associated requests. For instance, you might have a "UserLogin" BT, that is used to track requests that are associated with the "/login" URL. You might also have a "StatusRequest" BT, that is used to track any user status request URLs (perhaps you've created a Custom Match Rule regular expression that matches all URLs that begin with "/system/status"). These are examples of two BTs, but they could represent lots of URL requests the Apache server responds to, and could constitute thousands of individual requests from hundreds of users within a given timeframe.   When displayed within a flow-map, the BT will be shown as having been created by the "originating tier" (i.e. the first AppDynamics instrumented server it encountered) and is then correlated with other instrumented servers as that request migrates through the network topology.   Business Transaction Limits There is a limit on the number of BTs that any given Node can create (50), and a limit on how many can be represented within an Application (200). After either of these limits are reached, any new BTs will be sent to the "AllOtherTraffic" container for the Node on which it occurred.   Out of the box, the Apache Agent will treat any new URL request as a new BT. This will very quickly exceed the node limit for Apache if your request stream uses lots of slightly different URL names. It can also exceed the application limit if your previously instrumented downstream servers were already at or near that threshold.   If that request was previously being captured as a BT from a downstream server, it will now originate with Apache but be sent to the "AllOtherTraffic" container. This is why when adding an Apache Agent to an existing (fully-instrumented) topology, you may sometimes stop seeing load being reported from a server downstream of it (the Apache Agent has now started reporting the URL that was previously originating at it), or not see load reported from the Apache server itself (because you exceeded the Node threshold).   What is important about that idea is that the first Agent (i.e. any server on which you have installed and enabled our APM software) will be considered the "Originating Tier," and add a Singularity Header to all subsequent (downstream) server requests. This additional header will be used by the downstream tiers to correlate the requests and display a flow-map, so if the Apache (upstream) Node has exceeded its threshold, then the previously reporting downstream tiers/nodes may stop showing that BT.   The total number of BTs in effect will be listed in the top right corner of the Business Transactions page, and given based on the time frame specified, for the Application, Tier, Node, or BT in question.  Click on the Business Transactions page for the Apache Node to see the current total number of BTs created.   This value should not exceed 50 for the Node, or 200 for the Application.   To avoid this situation, it is important to review the current BT levels. Take a look at the Business Transactions pages for the entire application, and then for each node of each tier, choose a time frame longer than 1 week. If, in the upper right-hand corner of each page (highlighted in red), you see a message like "showing N transactions of M" (where either N or M are values that exceed either the node or application, depending on what you are viewing) then you'll need to revise the match rules to limit the BTs created (keeping it within 50 for each Node, and 200 for the Application as a whole), delete any BTs that were created inadvertently, and promote any BTs from within the "AllOtherTraffic" container that don't belong there.   Setting up Health Rules In a properly running system, the AllOtherTraffic containers should remain empty.   If you see BTs being sent there, then you need to reconfigure. The best way to do this (for Apache) is to stop the web server, then stop the proxy task, then from the UI, create the appropriate Custom Match Rules such that all of the BTs created (within Apache) do not exceed 50.   Select Configuration, then Instrumentation, then Rules Within Rules, Select WebServer, then right click to edit, then Rule Configuration   You'll also want to go into the Business Transactions page, then select "AllOtherTraffic" container, and then select the "View All Traffic" link, which will display the list of BTs within that container. Either delete or promote these items as necessary. Once you've created the match rules that cover all of the URLs you want to track, you may want to create a rule that acts as a "CatchAllOther" BT. This rule should have the lowest priority, and match any URL. Any URL request that doesn't match a prior rule will go into this BT. You can then set up a health rule alert on this BT to notify you when a new URL is discovered that doesn't match any prior rule, to decide which BT is should be included in.    Once you've properly set up your rules, restart the proxy and the Apache server, and run load on your system. You should see BTs being created as configured, with Apache cited as the "originating tier" (this is best seen from the "Transaction Snapshots" page, selecting a line item then right clicking to "View Details", then selecting the "Waterfall View"); the downstream tiers should correlate to them.       In a similiar manner, you can check that the "AllOtherTraffic" containers are empty and that the "CatchAllOthers" BT doesn't contain anything you aren't interested in tracking. If everything doesn't report as expected, review your match rules for each tier, and repeat the start/stop steps described above. Remember that there is some propagation delay when changing configurations/match rules, and allow a few minutes for these changes to take effect across all nodes/tiers within the application.   Business Transaction Lockdown   If you'd rather not use a catch-all BT, try enabling the "Business Transaction Lockdown" checkbox. This feature prevents any new BTs from being created by default. Having this set prevents unexpected BTs from exceeding the thresholds, and prevents any new BTs from being created by default. To find this feature, select Configuration > Instrumentation.  If you are using a 4.2.x version of the Controller, select the Application level within the tiers/nodes list and then check the checkbox.   If you are using a 4.3.x version of the Controller, select the "More" option.   In either case, remember that BT Lockdown applies at the Application Level.   Select Configuration, then Instrumentation, then More When you enable this feature, any new URLs that don't match an existing rule will be sent to the "AllOtherTraffic" container directly, regardless of the current state of BTs and their thresholds. ... View more

The following article describes how to install and configure a single Apache Agent source directory, but run multiple concurrent Apache Agents using the read-only copy of the installed executable, while maintaining separate logging, IPC, and configuration files.   The Apache Agent runs as a load module within the Apache Server framework. The Proxy runs as a self-contained GlassFish JVM application. T he Apache Agent uses the ZeroMQ messaging system and Unix socket files to communicate between the Agent and the Proxy tasks; the Proxy communicates with the Controller via HTTP(S) messaging.   Customers with large installations and/or numerous Agents often wish to install only a single (read-only) copy of the Agent and Proxy but execute multiple instances from that source installation. This allows for ease of upgrades and guarantees that all concurrent Apache Agents will always be executing from the same release after a system reboot.   A few disclaimers: While it is possible to execute multiple Agents from a single "writable" install directory without following the guidelines described here, doing so will result in only one of the competing Agents successfully creating the sdk.log , api.log , and api_user.log files. The other files will be overwritten. By following the steps outlined here, you can maintain separate logs, IPC, and configuration files and no diagnostic or logging information will be lost. Each Agent will have its own logs and configuration files, and proxy task.  There is minimal reduction in disk usage using this approach. Each Agent will still consume disk space for logging, configuration files, IPC, etc. The read-only copy of the installed software can be mounted locally or via a Network Attached Storage system (NAS), but all log files, configuration files, and any IPC related files (i.e. the appd-sdk  directory) must be mounted locally. AppDynamics does not support NAS for non-static files. If your installation uses SELinux, follow the SELinux guidelines found within the installation guide, and consult your system (or infosec) administrator for additional assistance. One approach is to temporarily disable SELinux when installing, to avoid any complications during the initial setup. Once everything is configured, re-enable SELinux and create whatever additional policy files are required. Always discuss this with the appropriate administrators before proceeding. Before beginning, completely read through the Apache Agent installation directions specific to your download release version listed on our documents site. Each version may have slightly different instructions, it is important that you follow the steps listed for your specific release. The various directory and file locations are, by default, determined by the location of the installed software, which in this case would be a read-only directory. As we want to customize these files for each Agent executed, it is necessary to configure the start-up logic so that each Agent has its own logs & IPC directory (on a writable, locally attached filesystem), along with separate configuration files (which determine the writable file locations).   Assume for purposes of this article that you wish to create a directory /opt/appd_readonly/appdynamics-sdk-native that contains a read-only copy of the installed Agent software, and several separate Apache Agent directories that will each have their own logs, IPC, and conf directories.   To that end, you will need to create a separate directory structure for each Agent executed. We'll call them /opt/appd_stage , /opt/appd_test , /opt/appd_prod .  Within each, there will be a runSDKProxy script, along with a logs subdirectory and conf subdirectory.  Within the conf subdirectory will be a customized copy of the original appdyanmics_sdk_logs4cxx.xml file created when the original (read-only) copy of the install script was executed.   NOTE: This document assumes that for each Agent, a separate Proxy will be started. It is possible to run multiple Agents using only a single Proxy, although there is a performance degradation (on heavily loaded servers) as you increase the number of unique Application Contexts above about ten contexts. An Application Context is the unique tuple of the Application name, the Tier name, and the Node name.    Prerequisites:  Make sure that the user ID of the installed software, as well as any additional directories or files created, is owned by the same user ID/group ID of the Apache worker thread. This will eliminate a number of unrelated permissions issues, but may require root access to accomplish, especially if the Apache worker thread user ID is created as a "no-login" user.  Normally the Apache server main process will execute as root, but for security concerns, all worker threads will run with a different user ID/group ID. By default, this will be either "httpd" or "apache2", depending on your operating install and local configuration. To determine the Apache worker thread user ID/group ID, run the following Unix command on your machine: ps -ef | grep httpd | grep -v grep Note: you may need to substitute "apache2" for "httpd", depending on the operating system used.     The output should look something like this: root 5379 1   0 May28 ? 00:00:19  /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start httpd 6801 5379 0 May29 ? 00:00:00  /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start httpd 6804 5379 0 May29 ? 00:00:12  /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start httpd 6805 5379 1 May29 ? 00:36:07  /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start httpd 6853 5379 1 May29 ? 00:32:48 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start httpd 6034 5379 2 May29 ? 00:39:34 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start httpd 6918 5379 2 May29 ? 00:41:36 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start  Notice that the user ID "5379" is the parent process, running as root, and that all child processes are spawned as "httpd".   Alternatively, you can view the httpd.conf (or apache2.conf ) configuration file (or one of its subfiles, usually included from within the conf.d subdirectory), looking for the Apache directives " User" and " Group", which explicitly set the user ID and group ID of the worker thread .    Installation Follow the guidelines shown in the version-specific Apache Agent installation documentation to create the /opt/appd_readonly directory from the downloaded tarball, using the user ID/group ID of the Apache worker thread.    1. Install to a directory that you have write access to, and make it read-only later on. tar -xvf appd_tarball.gz -C /opt/appd_readonly/ Using the -C option for the tar command allows you to specify where the software will be installed.   2. Run the install script in the directory.   cd /opt/appd_readonly/appdynamics-sdk-native install.sh This creates the appdynamics_sdk_log4cxx.xml file within the conf directory. ls -l conf total 8 -rw-rw-r-- 1 user1 user1 2049 Apr 28 13:48 appdynamics_sdk_log4cxx.xml -rwxrwxrwx 1 user1 user1 2004 Apr 14 14:34 appdynamics_sdk_log4cxx.xml.template   3. Rename the appdynamics_sdk_log4cxx.xml file to appdynamics_sdk_log4cxx.xml.generic . mv appdynamics_sdk_log4cxx.xml appdynamics_sdk_log4cxx.xml.generic   4. Delete the logs directory. We will create separate ones for each executable copy. rm /opt/appd_readonly/appdynamics-sdk-native/logs   5. Change the permission on the entire install directory to read-only (0x444) recursively. chmod -R 444/opt/appd_readonly/appdynamics-sdk-native   Agent Configuration 1. Next, create a new directory for each Agent you wish to execute and name them accordingly.   In this example, we create three agents for each of the stage, test, and production environments. mkdir /opt/appd_stage /opt/appd_test /opt/appd_prod   2. Within each of these environment directories, create logs and conf subdirectories. mkdir /opt/appd_stage/logs /opt/appd_stage/conf mkdir /opt/appd_test/logs /opt/appd_test/conf mkdir /opt/appd_prod/logs /opt/appd_prod/conf   3. Within each of the logs directory, create an appd-sdk subdirectory. This is the directory used for IPC between the Agent and Proxy tasks. mkdir /opt/appd_stage/logs/appd-sdk mkdir /opt/appd_test/logs/appd-sdk mkdir /opt/appd_prod/logs/appd-sdk   4. Copy the read-only copy of the appdynamics_sdk_log4cxx.xml.generic to each of the conf directories, renaming it to appdynamics_sdk_log4cxx.xml . cp /opt/appd_readonly/appdynamics-sdk-native/conf/appdynamics_sdk_log4cxx.xml.generic /opt/appd_stage/conf/appdynamics_sdk_log4cxx.xml cp /opt/appd_readonly/appdynamics-sdk-native/conf/appdynamics_sdk_log4cxx.xml.generic /opt/appd_test/conf/appdynamics_sdk_log4cxx.xml cp /opt/appd_readonly/appdynamics-sdk-native/conf/appdynamics_sdk_log4cxx.xml.generic /opt/appd_prod/conf/appdynamics_sdk_log4cxx.xml   5. Edit each of these new files, modifying the three file entries that specify the logging file names.   Note: Replace the directory path portion of the sdk.log , api.log , and api_user.log locations specified (within the appender name blocks) to point to the new log directory specific to each (stage, test, and prod for this example) for each Agent. For example, for the stage XML file, you should have something that now looks like this: <appender name="main" class="org.apache.log4j.RollingFileAppender"> <param name="file" value="/opt/appd_stage/logs/sdk.log"/> <param name="HeaderOnlyInNewFile" value="true"/> <param name="MaxFileSize" value="20MB"/> <param name="MaxBackupIndex" value="5"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" /> <param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" /> </layout> </appender> <appender name="api" class="org.apache.log4j.RollingFileAppender"> <param name="file" value="/opt/appd_stage/logs/api.log"/> <param name="HeaderOnlyInNewFile" value="true"/> <param name="MaxFileSize" value="20MB"/> <param name="MaxBackupIndex" value="5"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" /> <param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" /> </layout> </appender> <appender name="api_user" class="org.apache.log4j.RollingFileAppender"> <param name="file" value="/opt/appd_stage/logs/api_user.log"/> <param name="HeaderOnlyInNewFile" value="true"/> <param name="MaxFileSize" value="20MB"/> <param name="MaxBackupIndex" value="5"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" /> <param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" /> </layout> </appender>   6. For each Agent, create the appdynamics_agent.conf file, making sure it is the last line included from the httpd.conf file of the associated Apache server. (See the installation documentation for more details.)   7. Modify each appdynamics_agent.conf file so that the Load File and Load Module directives refer to the read-only location of the installation directory load module and libraries. LoadFile /opt/appd_readonly/appdynamics-sdk-native/sdk_lib/lib/libappdynamics_native_sdk.so LoadModule appdynamics_module /opt/appd_readonly/appdynamics-sdk-native/WebServerAgent/Apache/libmod_appdynamics22.so   ZeroMQ Socket Files If you want to use a separate directory for the ZeroMQ socket files, modify the appdynamics_agent.conf file such that the AppDynamicsProxyCommDir directive refers to the Agent logs directory. To place all the socket files in the tmp directory, include the following directive in the appdynamic_agent.conf files for each of the Agents. This is often done to reduce the number of socket files created between host system restarts.   Example:  For the stage environment, add the following line to the appdynamics_agent.conf file: AppDynamicsProxyCommDir /tmp/appd_stage/appd_sdk/   Proxy Configuration When Apache starts up, it will include the Agent load module, which first checks for the existence of the APPD_SDK_LOG_CONFIG_PATH environment variable. This specifies the location of the appdynamics_sdk_log4cxx.xml configuration file.   If not found, it will default to the conf file located in the read-only install directory, which is no longer appropriate. If you see messages in the Apache error log during startup stating that the Agent could not open the appdynamics_sdk_log4cxx.xml configuration file, it may be because the environment variable was not exported properly.   Note: You must completely stop then start Apache for it to properly obtain any newly exported environment variables. Do not use the restart or graceful command to reboot Apache.   To set the environment variable correctly, modify the Apache start script (usually this is part of the system.d service script) to include an export of the environment APPD_SDK_LOG_CONFIG_PATH to point to the  appdynamics-sdk-log4cxx.xml file for each Agent.   Note: Do not use the SetEnv directive within Apache to set this variable; it is executed too late in the Apache start chain to be used properly.   1. Modify each of the Apache startup (or system services) scripts accordingly.   Example:    For the stage Apache instance, modify the system start script to export the location of the stage configuration file before Apache is invoked: ...(prior Apache server set-up)... export APPD_SDK_LOG_CONFIG_PATH=/opt/appd-stage/conf/appdynamics_sdk_log4cxx.xml ...(start of Apache)...    Next, we need to create the start script for the proxy tasks for each Apache Agent that will be executed.   2. Copy the read-only version of the runSDKProxy script to the high level directory related to each Agent.     It is recommended that the Proxy task always be started before the corresponding Apache server (and its Agent). The following script should be invoked before the Apache service is executed: cp /opt/appd_readonly/appdynamics-sdk-native/runSDKProxy /opt/appd_stage/ cp /opt/appd_readonly/appdynamics-sdk-native/runSDKProxy /opt/appd_test/ cp /opt/appd_readonly/appdynamics-sdk-native/runSDKProxy /opt/appd_prod/   3. Within the runSDKProxy script ( /opt/appd-state/runSDKProxy , /opt/appd-test/runSDKProxy , /opt/appd-prod/runSDKProxy ), modify the containingDir  variable within each script to point to the read-only install directory.   Example: containingDir=/opt/appd_readonly/appdynamics-sdk-native/   4. Modify the logsDirectory  variable within each of the scripts to point to the logs directory for the Agent.   E xample: Modify the logsDir setting in the stage script to contain: logsDirectory="/opt/appd_stage/logs"   5. Create a Proxy init.d script (either one for each environment, or as a single script that starts all AppDynamics Apache Agent proxies), so that it invokes the modified runSDKProxy script, starting the Proxy task using nohup and directing the output to proxy.out within the logs directory for each Agent. nohup /opt/appd_stage/runSDKProxy.sh >>/dev/null 2>/opt/appd_stage/logs/proxy.out & When the proxy start script is invoked, it will execute a self-contained GlassFish JVM application that will establish a connection with the Controller, create log files, and populate the appd-sdk directory. Look for a file named "0", and others with names like proxy-1214768854178727131.   Agent Execution 1. Start the proxy services first (it is recommended you do so as a Unix system service) making sure that each proxy creates files in the Agent-associated logs directory, and that they create the appd-sdk  directory in each of their respective logs directories. 2. Stop the Apache web server, then start it. This forces the Apache server to pick up the new environment variable. 3. Apply load to each of the Apache web servers. 4. Verify that the Agent has the appropriate license and can register with the Controller. 5. Check that the Agent is able to create the appropriate files ( sdk.log , api.log , and api_user.log ) in its Agent logs directory.   At this point, you should be able to see load registering on the Controller UI for each Agent. There should also be log files for each Agent, and their corresponding proxies, and the Agents should display on the Controller UI as active and healthy. If any of the above is otherwise, review the steps listed here, being especially careful about typos when specifying file names. ... View more