The following article describes how to install and configure a single Apache Agent source directory, but run multiple concurrent Apache Agents using the read-only copy of the installed executable, while maintaining separate logging, IPC, and configuration files.
The Apache Agent runs as a load module within the Apache Server framework. The Proxy runs as a self-contained GlassFish JVM application. The Apache Agent uses the ZeroMQ messaging system and Unix socket files to communicate between the Agent and the Proxy tasks; the Proxy communicates with the Controller via HTTP(S) messaging.
Customers with large installations and/or numerous Agents often wish to install only a single (read-only) copy of the Agent and Proxy but execute multiple instances from that source installation. This allows for ease of upgrades and guarantees that all concurrent Apache Agents will always be executing from the same release after a system reboot.
A few disclaimers:
- While it is possible to execute multiple Agents from a single "writable" install directory without following the guidelines described here, doing so will result in only one of the competing Agents successfully creating the
sdk.log, api.log, and api_user.log files. The other files will be overwritten. By following the steps outlined here, you can maintain separate logs, IPC, and configuration files and no diagnostic or logging information will be lost. Each Agent will have its own logs and configuration files, and proxy task.
- There is minimal reduction in disk usage using this approach. Each Agent will still consume disk space for logging, configuration files, IPC, etc.
- The read-only copy of the installed software can be mounted locally or via a Network Attached Storage system (NAS), but all log files, configuration files, and any IPC related files (i.e. the
appd-sdk directory) must be mounted locally. AppDynamics does not support NAS for non-static files.
- If your installation uses SELinux, follow the SELinux guidelines found within the installation guide, and consult your system (or infosec) administrator for additional assistance. One approach is to temporarily disable SELinux when installing, to avoid any complications during the initial setup. Once everything is configured, re-enable SELinux and create whatever additional policy files are required. Always discuss this with the appropriate administrators before proceeding.
- Before beginning, completely read through the Apache Agent installation directions specific to your download release version listed on our documents site. Each version may have slightly different instructions, it is important that you follow the steps listed for your specific release.
The various directory and file locations are, by default, determined by the location of the installed software, which in this case would be a read-only directory. As we want to customize these files for each Agent executed, it is necessary to configure the start-up logic so that each Agent has its own logs & IPC directory (on a writable, locally attached filesystem), along with separate configuration files (which determine the writable file locations).
Assume for purposes of this article that you wish to create a directory /opt/appd_readonly/appdynamics-sdk-native that contains a read-only copy of the installed Agent software, and several separate Apache Agent directories that will each have their own logs, IPC, and conf directories.
To that end, you will need to create a separate directory structure for each Agent executed. We'll call them /opt/appd_stage, /opt/appd_test, /opt/appd_prod. Within each, there will be a runSDKProxy script, along with a logs subdirectory and conf subdirectory. Within the conf subdirectory will be a customized copy of the original appdyanmics_sdk_logs4cxx.xml file created when the original (read-only) copy of the install script was executed.
NOTE: This document assumes that for each Agent, a separate Proxy will be started. It is possible to run multiple Agents using only a single Proxy, although there is a performance degradation (on heavily loaded servers) as you increase the number of unique Application Contexts above about ten contexts. An Application Context is the unique tuple of the Application name, the Tier name, and the Node name.
Prerequisites:
- Make sure that the user ID of the installed software, as well as any additional directories or files created, is owned by the same user ID/group ID of the Apache worker thread. This will eliminate a number of unrelated permissions issues, but may require root access to accomplish, especially if the Apache worker thread user ID is created as a "no-login" user.
- Normally the Apache server main process will execute as root, but for security concerns, all worker threads will run with a different user ID/group ID. By default, this will be either "httpd" or "apache2", depending on your operating install and local configuration. To determine the Apache worker thread user ID/group ID, run the following Unix command on your machine:
ps -ef | grep httpd | grep -v grep
Note: you may need to substitute "apache2" for "httpd", depending on the operating system used.
The output should look something like this:
root 5379 1 0 May28 ? 00:00:19 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start
httpd 6801 5379 0 May29 ? 00:00:00 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start
httpd 6804 5379 0 May29 ? 00:00:12 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start
httpd 6805 5379 1 May29 ? 00:36:07 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start
httpd 6853 5379 1 May29 ? 00:32:48 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start
httpd 6034 5379 2 May29 ? 00:39:34 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start
httpd 6918 5379 2 May29 ? 00:41:36 /usr/sbin/httpd -d /u01/IBM/HTTPServer -k start
Notice that the user ID "5379" is the parent process, running as root, and that all child processes are spawned as "httpd".
Alternatively, you can view the httpd.conf (or apache2.conf) configuration file (or one of its subfiles, usually included from within the conf.d subdirectory), looking for the Apache directives "User" and "Group", which explicitly set the user ID and group ID of the worker thread.
Installation
Follow the guidelines shown in the version-specific Apache Agent installation documentation to create the /opt/appd_readonly directory from the downloaded tarball, using the user ID/group ID of the Apache worker thread.
1. Install to a directory that you have write access to, and make it read-only later on.
tar -xvf appd_tarball.gz -C /opt/appd_readonly/
Using the -C option for the tar command allows you to specify where the software will be installed.
2. Run the install script in the directory.
cd /opt/appd_readonly/appdynamics-sdk-native
install.sh
This creates the appdynamics_sdk_log4cxx.xml file within the conf directory.
ls -l conf
total 8
-rw-rw-r-- 1 user1 user1 2049 Apr 28 13:48 appdynamics_sdk_log4cxx.xml
-rwxrwxrwx 1 user1 user1 2004 Apr 14 14:34 appdynamics_sdk_log4cxx.xml.template
3. Rename the appdynamics_sdk_log4cxx.xml file to appdynamics_sdk_log4cxx.xml.generic.
mv appdynamics_sdk_log4cxx.xml appdynamics_sdk_log4cxx.xml.generic
4. Delete the logs directory. We will create separate ones for each executable copy.
rm /opt/appd_readonly/appdynamics-sdk-native/logs
5. Change the permission on the entire install directory to read-only (0x444) recursively.
chmod -R 444/opt/appd_readonly/appdynamics-sdk-native
Agent Configuration
1. Next, create a new directory for each Agent you wish to execute and name them accordingly.
In this example, we create three agents for each of the stage, test, and production environments.
mkdir /opt/appd_stage /opt/appd_test /opt/appd_prod
2. Within each of these environment directories, create logs and conf subdirectories.
mkdir /opt/appd_stage/logs /opt/appd_stage/conf
mkdir /opt/appd_test/logs /opt/appd_test/conf
mkdir /opt/appd_prod/logs /opt/appd_prod/conf
3. Within each of the logs directory, create an appd-sdk subdirectory. This is the directory used for IPC between the Agent and Proxy tasks.
mkdir /opt/appd_stage/logs/appd-sdk
mkdir /opt/appd_test/logs/appd-sdk
mkdir /opt/appd_prod/logs/appd-sdk
4. Copy the read-only copy of the appdynamics_sdk_log4cxx.xml.generic to each of the conf directories, renaming it to appdynamics_sdk_log4cxx.xml.
cp /opt/appd_readonly/appdynamics-sdk-native/conf/appdynamics_sdk_log4cxx.xml.generic /opt/appd_stage/conf/appdynamics_sdk_log4cxx.xml
cp /opt/appd_readonly/appdynamics-sdk-native/conf/appdynamics_sdk_log4cxx.xml.generic /opt/appd_test/conf/appdynamics_sdk_log4cxx.xml
cp /opt/appd_readonly/appdynamics-sdk-native/conf/appdynamics_sdk_log4cxx.xml.generic /opt/appd_prod/conf/appdynamics_sdk_log4cxx.xml
5. Edit each of these new files, modifying the three file entries that specify the logging file names.
Note: Replace the directory path portion of the sdk.log, api.log, and api_user.log locations specified (within the appender name blocks) to point to the new log directory specific to each (stage, test, and prod for this example) for each Agent. For example, for the stage XML file, you should have something that now looks like this:
<appender name="main" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/opt/appd_stage/logs/sdk.log"/>
<param name="HeaderOnlyInNewFile" value="true"/>
<param name="MaxFileSize" value="20MB"/>
<param name="MaxBackupIndex" value="5"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" />
<param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
</layout>
</appender>
<appender name="api" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/opt/appd_stage/logs/api.log"/>
<param name="HeaderOnlyInNewFile" value="true"/>
<param name="MaxFileSize" value="20MB"/>
<param name="MaxBackupIndex" value="5"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" />
<param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
</layout>
</appender>
<appender name="api_user" class="org.apache.log4j.RollingFileAppender">
<param name="file" value="/opt/appd_stage/logs/api_user.log"/>
<param name="HeaderOnlyInNewFile" value="true"/>
<param name="MaxFileSize" value="20MB"/>
<param name="MaxBackupIndex" value="5"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS z} %-5p %X{pid} [%c{2}] %m%n" />
<param name="HeaderPattern" value="AppDynamics Native SDK %X{version} %X{pid}%n" />
</layout>
</appender>
6. For each Agent, create the appdynamics_agent.conf file, making sure it is the last line included from the httpd.conf file of the associated Apache server. (See the installation documentation for more details.)
7. Modify each appdynamics_agent.conf file so that the Load File and Load Module directives refer to the read-only location of the installation directory load module and libraries.
LoadFile /opt/appd_readonly/appdynamics-sdk-native/sdk_lib/lib/libappdynamics_native_sdk.so
LoadModule appdynamics_module /opt/appd_readonly/appdynamics-sdk-native/WebServerAgent/Apache/libmod_appdynamics22.so
ZeroMQ Socket Files
If you want to use a separate directory for the ZeroMQ socket files, modify the appdynamics_agent.conf file such that the AppDynamicsProxyCommDir directive refers to the Agent logs directory. To place all the socket files in the tmp directory, include the following directive in the appdynamic_agent.conf files for each of the Agents. This is often done to reduce the number of socket files created between host system restarts.
Example: For the stage environment, add the following line to the appdynamics_agent.conf file:
AppDynamicsProxyCommDir /tmp/appd_stage/appd_sdk/
Proxy Configuration
When Apache starts up, it will include the Agent load module, which first checks for the existence of the APPD_SDK_LOG_CONFIG_PATH environment variable. This specifies the location of the appdynamics_sdk_log4cxx.xml configuration file.
If not found, it will default to the conf file located in the read-only install directory, which is no longer appropriate. If you see messages in the Apache error log during startup stating that the Agent could not open the appdynamics_sdk_log4cxx.xml configuration file, it may be because the environment variable was not exported properly.
Note: You must completely stop then start Apache for it to properly obtain any newly exported environment variables. Do not use the restart or graceful command to reboot Apache.
To set the environment variable correctly, modify the Apache start script (usually this is part of the system.d service script) to include an export of the environment APPD_SDK_LOG_CONFIG_PATH to point to the appdynamics-sdk-log4cxx.xml file for each Agent.
Note: Do not use the SetEnv directive within Apache to set this variable; it is executed too late in the Apache start chain to be used properly.
1. Modify each of the Apache startup (or system services) scripts accordingly.
Example: For the stage Apache instance, modify the system start script to export the location of the stage configuration file before Apache is invoked:
...(prior Apache server set-up)...
export APPD_SDK_LOG_CONFIG_PATH=/opt/appd-stage/conf/appdynamics_sdk_log4cxx.xml
...(start of Apache)...
Next, we need to create the start script for the proxy tasks for each Apache Agent that will be executed.
2. Copy the read-only version of the runSDKProxy script to the high level directory related to each Agent.
It is recommended that the Proxy task always be started before the corresponding Apache server (and its Agent). The following script should be invoked before the Apache service is executed:
cp /opt/appd_readonly/appdynamics-sdk-native/runSDKProxy /opt/appd_stage/
cp /opt/appd_readonly/appdynamics-sdk-native/runSDKProxy /opt/appd_test/
cp /opt/appd_readonly/appdynamics-sdk-native/runSDKProxy /opt/appd_prod/
3. Within the runSDKProxy script (/opt/appd-state/runSDKProxy, /opt/appd-test/runSDKProxy, /opt/appd-prod/runSDKProxy), modify the containingDir variable within each script to point to the read-only install directory.
Example:
containingDir=/opt/appd_readonly/appdynamics-sdk-native/
4. Modify the logsDirectory variable within each of the scripts to point to the logs directory for the Agent.
Example: Modify the logsDir setting in the stage script to contain:
logsDirectory="/opt/appd_stage/logs"
5. Create a Proxy init.d script (either one for each environment, or as a single script that starts all AppDynamics Apache Agent proxies), so that it invokes the modified runSDKProxy script, starting the Proxy task using nohup and directing the output to proxy.out within the logs directory for each Agent.
nohup /opt/appd_stage/runSDKProxy.sh >>/dev/null 2>/opt/appd_stage/logs/proxy.out &
When the proxy start script is invoked, it will execute a self-contained GlassFish JVM application that will establish a connection with the Controller, create log files, and populate the appd-sdk directory. Look for a file named "0", and others with names like proxy-1214768854178727131.
Agent Execution
1. Start the proxy services first (it is recommended you do so as a Unix system service) making sure that each proxy creates files in the Agent-associated logs directory, and that they create the appd-sdk directory in each of their respective logs directories.
2. Stop the Apache web server, then start it. This forces the Apache server to pick up the new environment variable.
3. Apply load to each of the Apache web servers.
4. Verify that the Agent has the appropriate license and can register with the Controller.
5. Check that the Agent is able to create the appropriate files (sdk.log, api.log, and api_user.log) in its Agent logs directory.
At this point, you should be able to see load registering on the Controller UI for each Agent. There should also be log files for each Agent, and their corresponding proxies, and the Agents should display on the Controller UI as active and healthy. If any of the above is otherwise, review the steps listed here, being especially careful about typos when specifying file names.
