How do I configure a Multi-Tenant, Multi-Proxy AppDynamics-Instrumented Apache Environment?

[Erin]

This article is intended for anyone attempting to install more than one AppDynamics-instrumented Apache web server on the same host, where the number of Application Contexts exceeds the per-Proxy limit, and the Agents need to be distributed across multiple Proxies.

This approach minimizes the disk space requirements of the software, as well limits the RAM consumption of the proxy tasks. If tuned properly, it can provide the best possible use of the system resources while still providing full functionality, performance monitoring, and diagnostic logging.

Contents

• Example
• Basic Concepts
• Setting Up Multiple Apache Servers Using One Install Directory
• Installed Agent File Name Length & Type
• Agent Installation
• Proxy Initialization
• Configuring Agent Logging Output
• Testing The Configuration

Example

In this article we’ll explore setting up thirty AppDynamics-instrumented Apache web servers on a single host sharing three separate Proxy tasks. Each separate Proxy task will use a single Agent installation. Assume that the Apache server Nodes are each named T1WebNN (where NN represents a Node value between 01 and 30), and that they each used T1Web  as their Tier Name and Stage1 as the Application Name.

We’ll also assume that we have correctly installed thirty Apache web servers, and three separate copies of the AppDynamics Agent software into the directories /opt/appd/apache-proxyN (where N represents a value between 1 and 3 indicating the Proxy task), following the instructions provided in the AppDynamics Apache Installation guide.

 

Basic Concepts 

Agent/Proxy Communication

The AppDynamics Apache Agent (as part of the Apache worker threads) communicates with the Controller via a Proxy task, sending Business Transactions, Exit Calls, and BackEnd registration requests, and receives back from the Controller configuration updates, via the Proxy. These communications between the Agent (within each Apache worker thread) and the Proxy occur via ZeroMQ, a socket-based third party library.

For every Apache worker thread that is invoked, the Agent will connect with the Proxy, using a well known socket (filename “0”), contained (by default) in the “appd-sdk” directory within the “logs” directory of the installed Agent software (by default that installation directory is called “/opt/appdynamics-sdk-native”), although many installations will modify this installation directory name to suit their needs.

For this example, we will create three separate installations that contain three separate communications subdirectories, labeling them “/opt/appd/apache-proxyN” where N is the value 1, 2, or 3, differentiating which Proxy task is to be associated with what servers.

 

Application Contexts

Next, let’s review the notion of an Application Context. An Application Context is an AppDynamics construct used to designate a unique tuple (identifier string) within the Proxy, comprised of the Application Name, the Tier Name, and the Node Name. Each unique tuple of App/Tier/Node represents one context.

For instance, were you to install thirty Apache servers on the same host, all running within the same Application Name and under the same Tier Name but each using a unique Node Name, there would be thirty separate Application Contexts, as each Apache server would represent a separately named AppDynamics Node Name (server name).

Apache VirtualHost Support

There is a caveat to this. Apache (and version 4.2.7+ AppDynamics Agent) has support for “Virtual Hosts”, where a single Apache server can service numerous separate server names and/or ports.

For each Virtual Host specified within the Apache configuration, you’ll need to associate a separate Application Context within the VirtualHost configuration block within the Apache config file (for more details, see Map Virtual Hosts to AppDynamics Tiers).   

A typical entry for a VirtualHost block added to the first Apache server might look like this:

Listen 80
<VirtualHost *:80>
    DocumentRoot "/www/customer_com_80"
    ServerName customer.com
    ...
    AppDynamicsApplicationContext Stage1 T1Web custcom80
</VirtualHost>
<VirtualHost *:80>
    DocumentRoot "/www/customer_org_80"
    ServerName customer.org
    ...
    AppDynamicsApplicationContext Stage1 T1Web custorg80
</VirtualHost>

The above entry will create two additional Nodes (in addition to the default  “T1Web01” Application Context specified in the appdynamics_agent.conf file. These two additional Nodes will be called “custcom80” and “custorg80” and will be represented on the UI within the “Stage1” application and “T1Web” Tier.

Setting Up Multiple Apache Servers Using One Install Directory

In our example, we are going to install thirty Apache web servers using only three AppDynamics Agent install directories.

We will share the “appdynamics_agent.conf” file between multiple Apache httpd.conf files, so we recommend placing it in the <agent install dir>/conf directory, and adding a comment to note the fact that the file will be included by multiple web servers.

For instance, let’s assume that in our example, the first ten Apache web servers all include the file /opt/appd/apache-proxy1/conf/appdynamics_agent.conf file, the next ten servers include the  /opt/appd/apache-proxy2/conf/appdynamics_agent.conf file, and the last ten servers include the  /opt/appd/apache-proxy3/conf/appdynamics_agent.conf file. Remember that this include must be the last line of the httpd.conf file of each Apache server, as any modules included after the AppDynamics load module will not be properly instrumented and could result in incorrect monitoring.

Installed Agent File Name Length & Type

The Proxy and the Agent both use the installed agent directory pathname to determine the default location when creating logs, reading configuration files, and/or accessing sockets.

Because the Linux file system prohibits using any socket file with a name greater than 108 characters (a hard-coded limitation from the earliest versions of Unix), and because the appd-sdk directory is included within the install directory structure, and because the socket files are contained within subdirectories within that directory, and those entries have potentially long file & directory names, it is important that you leave enough room in the absolute pathname to account for them. Therefore the fully qualified pathname of the Agent install directory should be less than 65 characters.

Also, the appd-sdk directory (within the logs directory of the agent install directory) must be placed on locally attached disks. Do not use NFS mounted partitions.

Agent Installation

When the Agent is installed and enabled, at Apache start-up, it uses the first segments of the load module path name to determine the “install directory” location, which is where it expects to find the “conf “ directory (which contains the “appdynamics_sdk_log4cxx.xml” file) and the “logs” director (which contains the “appd-skd” communications directory).

For our example, let's assume we’ve installed the Agent software into three separate directories, each with the same substructure. Normally the Agent installs, by default, into /opt/appd-sdk-native. In this example, we will create three subdirectories under “/opt/appd” and run three separate Proxy tasks, one from each of them.

/opt/appd/apache-proxy1, /opt/appd/apache-proxy2, /opt/appd/apache-proxy3

The default files and directories of interest here are:

Directories

Within each of these three separate agent trees, we will run one Proxy task (for every ten servers that each uses one Application Context). Each tree would contain its own shared communication directory (appd-sdk) within the logs directory of the agent directory.

Following along with the example, for the first ten Apache servers, one would edit the appdynamics_agent.conf file (included from each Apache httpd.conf file) for each server, setting the LoadFile and LoadModule parameters to use the first Proxy’s install directory path:

LoadFile /opt/appd/apache-proxy1/sdk_lib/lib/libappdynamics_native_sdk.so
LoadModule appdynamics_module /opt/appd/apache-proxy1/WebServerAgent/Apache/libmod_appdynamics.so 
AppDynamicsEnabled On
AppDynamicsControllerHost mycontroller.saas.appdynamics.com
AppDynamicsControllerPort 80
AppDynamicsControllerSSL OFF
AppDynamicsAccountName customer1
AppDynamicsAccessKey zd8yjh5yuy5k 
AppDynamicsApplication Stage1
AppDynamicsTier T1Web
AppDynamicsNode T1Web01
AppDynamicsResolveBackends ON

For the second ten servers, edit the conf file to use the “/opt/appd/apache-proxy2/” pathname, and so on.

When the first ten servers start, they would use the /opt/appd/apache-proxy1/logs/appd-sdk directory for their (ZeroMQ) sockets communication directory, and the ".../appd-sdk/0" file within it as the well-known socket to connect to, to communicate with the Proxy. The second set of servers would use the /opt/appd/apache-proxy2/logs/appd-sdk directory instead. This would ensure that no more than ten application contexts were handled by any Proxy task.

Proxy Initialization 

At host startup, you will need to run the "runSDKProxy.sh" startup script to start the Proxy, from within each of the three installation directories, spawning three separate Proxy tasks. We suggest that no Proxy task should service more than about ten Application Contexts simultaneously. Exceeding this amount will result in performance degradation.

Because we’ve disabled the "AppDynamicsLaunchProxy" directive, use the following command to start the Proxy task (for the first Proxy in our example) :

nohup /opt/appd/apache-proxy1/runSDKProxy.sh 
>>/dev/null 2>/opt/appd/apache-proxy1/logs/proxy.out &

You can use the following command to show whether or not the Proxy task is running:

ps -ef | grep proxy | grep -v grep

Typical output will look something like this:

root       8416   8412  6 11:39 pts/11   00:00:00 /opt/appd/apache-proxy1/proxy/jre/bin/java -server -Xmx300m -Xms50m -classpath /opt/appd/apache-proxy1/proxy/conf/logging:/opt/appd/apache-proxy1/proxy/lib/*:/opt/appd/apache-proxy1/proxy/lib/tp/*:/opt/appd/apache-proxy1/proxy/* -Djava.library.path=/opt/appd/apache-proxy1/proxy/lib/tp -Dappdynamics.agent.logs.dir=/opt/appd/apache-proxy1/logs -Dcomm=/opt/appd/apache-proxy1/ogs/appd-sdk -DagentType=NATIVE_WEB_SERVER -Dappdynamics.agent.runtime.dir=/opt/appd/apache-proxy1/proxy -Dlog4j.ignoreTCL=true -XX:MaxPermSize=120m -XX:-UseGCOverheadLimit com.appdynamics.ee.agent.proxy.bootstrap.ProxyControlEntryPoint

For each active Proxy task, there will be one process entry. Note that the -DagentType=NATIVE_WEB_SERVER indicates that this is an Apache Agent

The Proxy task will produce several log files, most importantly will be some labeled “proxy.<datestamp>.log”, and others labeled “proxyCore.<datestamp>.log”. These files contain diagnostic information used when solving problems with the system, and are produced only by the Proxy, not the Agents.

Configuring Agent Logging Output 

The Web Server Agents also produce logging information, with file names like “api.log”, “api_user.log”, and “sdk.log”, but because this is a multi-tenant configuration, there are multiple Agents all attempting to write, by default, to the same output location using the same filename.   

If left as is, only the first Agent (i.e. the first web server to start) will produce output in the logs directory, the others will be unable to store their information. It is possible to leave the configuration as is, but should a problem arise, the information needed to diagnose it might be lost. Therefore it is strongly recommended you modify the configuration to save each web server’s log files separately.

To do so, first we will need to tell (separately) each of the Apache web server Agent where to write their logging output, contained in the file “appdynamics_sdk_log4cxx.xml”. This file is created by the install.sh script, and is normally found, by default, in the <agent_install_dir>/conf directory.

Having run the install.sh script as part of the installation, it created the file, We will use this file as the basis of the xml files used by the Agents. It should look something like this:

<?xml version="1.0" encoding="UTF-8" ?>
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">

<appender name="main" class="org.apache.log4j.RollingFileAppender">
 <param name="file" value="/opt/apache-proxy1/logs/sdk.log"/>
 <param name="HeaderOnlyInNewFile" value="true"/>
 <param name="MaxFileSize" value="20MB"/>
 <param name="MaxBackupIndex" value="5"/>
 <layout class="org.apache.log4j.PatternLayout">
  <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" />
  <param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
 </layout>
</appender>

<appender name="api" class="org.apache.log4j.RollingFileAppender">
 <param name="file" value="/opt/apache-proxy1/logs/api.log"/>
 <param name="HeaderOnlyInNewFile" value="true"/>
 <param name="MaxFileSize" value="20MB"/>
 <param name="MaxBackupIndex" value="5"/>
 <layout class="org.apache.log4j.PatternLayout">
  <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" />
  <param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
 </layout>
</appender>

<appender name="api_user" class="org.apache.log4j.RollingFileAppender">
 <param name="file" value="/opt/apache-proxy1/logs/api_user.log"/>
 <param name="HeaderOnlyInNewFile" value="true"/>
 <param name="MaxFileSize" value="20MB"/>
 <param name="MaxBackupIndex" value="5"/>
 <layout class="org.apache.log4j.PatternLayout">
  <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" />
  <param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
 </layout>
</appender>

<logger name="api" additivity="false">
    <level value="info"/>
    <appender-ref ref="api"/>
</logger>

<logger name="api_user" additivity="false">
    <level value="info"/>
    <appender-ref ref="api_user"/>
</logger>

<root>
  <priority value="info" />
  <appender-ref ref="main"/>
</root>

</log4j:configuration>

In the conf directory, create ten separate subdirectories, one for each server, in which we will place a modified copy of the appdynamics_sdk_log4cxx.xml file. Copy the appdynamics_sdk_log4cxx.xml file into each of the subdirectories, and to ensure that the original file does not get used by accident, rename it to “appdynamics_sdk_logs4cxx.xml.orig”.   

The /opt/appd/apache-proxy1/conf directory should now look like this:

Directories 

There are three lines we will want to alter in each of these ten new webserver XML files. Locate the lines that begin: <param name="file" value="/opt/appd/apache-proxy1/logs/..."/>

Within each of the files, and change the three pathname values to be:

value="/opt/appd/apache-proxy1/logs/webserver01_logs/sdk.log"
value="/opt/appd/apache-proxy1/logs/webserver01_logs/api.log"
value="/opt/appd/apache-proxy1/logs/webserver01_logs/api_user.log"

Next we’ll create a separate log directory (for each webserver) within the corresponding “/opt/appd/apache-proxy1/logs” directory. Following the above example, the logs directory should now look like this:

Directories

Within each of these subdirectories, the Apache Agent will create skd.log, api.log, and api_user.log files when the Agent is executed. Repeat this process for the second and third installations as well, following the same naming scheme.

When running, the each of the ten logs directories for a given Proxy task should contain the following files:Directories 

Lastly, we need to modify the Apache web server environment so that each server uses a separate log4cxx.xml file when configuring its logging output at startup. This is only possible by setting an environment variable that will be used as an override instead of the default location.

To set that environment variable correctly, modify the Apache start script (usually this is part of the system.d service script) to include an export of the environment APPD_SDK_LOG_CONFIG_PATH to point to the corresponding “/opt/appd/apache-proxy1/conf/webserver01/appdynamics-sdk-log4cxx.xml” file, for each of the ten Apache invocations.

Note: Do not use the SetEnv directive within Apache to set this variable; it is executed too late in the Apache start chain to be used properly.

Note: If you are using a system service to start the Apache web servers, modify each of the Apache startup (or system services) scripts accordingly. 

Example: For an Apache instance, modify the system start script to export the location of the stage configuration file before Apache is invoked:

export APPD_SDK_LOG_CONFIG_PATH=/opt/appd/apache-proxy1/conf/webserver01/appdynamics_sdk_log4cxx.xml

Make sure that each Apache server can start properly, and access has read/write access to the entire installation tree.

Testing The Configuration

Start each of the three Proxy tasks first. Each Proxy should create a few files within the logs directory, including a file called proxyCore.<datestamp>.log.

Ignoring the logging data stamps, it will contain a few lines, ending with something like the following:

INFO com.singularity.proxyControl.ProxyControlEntryPoint - Should register node at
startup:false

This indicates that the Proxy task is running and waiting for connections from the Agents.  

Next start the Apache Agents and apply load. Each Agent (web server) will begin writing to its own logging directory, using the configuration found in the appdynamics_sdk_log4cxx.xml file it was provided. You should now see files with the names api.log, api_user.log, and sdk.log appear in the various web server logging directories.

At this point, the proxy will attempt to register with the Controller, and you may see the Application Tier, and Node appear on the Controller.

Apply some moderate load, and you should begin to see that reflected on the Controller UI, although please allow several minutes for this to occur, as there is a propagation delay for the request to reach the Controller, and the configuration information to be returned back (via the Proxy) to the Agents.