Not a customer?
Start a free trial
Not a customer? Click the 'Start a free trial' link to begin a 30-day SaaS trial of our product and to join our community.
Existing Cisco AppDynamics customers should click the 'Sign In' button to authenticate to access the community
Leverage the Chef configuration management tool to deploy and configure AppDynamics Agents in your environment
Introduction
Many AppDynamics customers want to rapidly deploy APM agents within their existing code roll-out processes. They also struggle to keep the agent versions consistent or upgrade them in a controlled manner. If your organization currently uses Chef or is considering it, this article shows how to quickly deploy the Java APM and Machine agents.
I deploy the AppDynamics Java APM and Machine Agent to a CentOS 7 server running Apache Tomcat. This simplifies the example to adapt for more sophisticated cases, such as Wildfly or WebLogic. Feel free to adapt this article to suit your needs.
Table of Contents
- Configuration Management: It's A Thing! | Prerequisites
- Gather Ingredients | Simplified Chef Workflow
- Infrastructure Assumptions
- Cookbook Table of Contents
- Prepare Ingredients | Default Attributes, Static Files, Template Files, The Recipes
- Preheat the Oven
- Cook the Entrée
- Garnish and Serve
- And for Dessert…
1. Configuration Management: It's a thing!
Given the complexities of modern infrastructure and deployments, every organization needs a tool to manage the moving parts. When I visit customer sites, I'm surprised that so many have never heard of configuration management. They either cobble together brittle scripts requiring significant rewrites for every new deployment, or they have error-prone checklists. Neither approach is sustainable.
Chef allows you to focus on what needs to be done. How it gets accomplished is Chef's problem. Complex problems are sliced into small pieces, called recipes. One or more recipes are grouped together to form cookbooks. One or more cookbooks are stored on a Chef server for use when applying changes to a target host, called a node in Chef parlance.
Multiple cookbooks can be applied to one or more nodes. The specific attributes that need to appear in a given host's configuration can come from the recipe's default values; or, from the Chef server's environment, role, and node to dynamically substitute the relevant value at the appropriate time. You manage the Chef server to update the relevant cookbooks and key/value pairs in your environments (development, QA, production), roles (AppServer, MailServer, WebServer, etc.), and nodes (wkstn01.stevencortright.com shown in this article).
Prerequisites
The complete cookbook, AppD_Java.tar.gz, is attached to this article. Untar it and follow along!
I assume you have a Chef development environment on your laptop. If not, head over to https://learn.chef.io/#/ to get started.
You also need to install other tools such as VirtualBox, Vagrant, and Test Kitchen. See an introductory Chef book for more details.
2. Gather Ingredients
To follow the culinary metaphor of Chef, you need to inspect your kitchen to verify you have all the necessary ingredients. Here's what we need.
- A Chef development environment (probably your laptop)
- A Chef server (either self-hosted or SaaS)
- A Target host (a VM or bare metal server/workstation) running CentOS 7.x
- A File Repository (the target host does double-duty in my home lab environment)
- A Source Control server (optional but strongly recommended)
I also assume you have access to an AppDynamics Controller; but omitted it from the diagram for brevity.
Simplified Chef Workflow
The simplified Chef workflow is:
- Iterate on a cookbook/recipes
- Test cookbook changes in Test Kitchen on a VM on your laptop
- Check-in successful changes to source control repository
- Peer review
- Knife cookbook changes to Chef server
- Knife environment/role/node changes for a group of servers
- Chef run on target node(s)
In the interest of space, this article does not exhaustively follow these steps. When you customize this cookbook, remember to iterate on small changes, test, then check in your progress before undertaking more changes.
3. Infrastructure Assumptions
The AppDynamics agents (Java and Machine) are static artifacts (zip/RPM file). Generally, you should pull down the latest files from the download site and store them on a file repository in your corporate network. When you run a Chef deployment to a target environment, the agents should be pulled from the local repository.
It’s a poor approach to always pull the artifacts over the Internet, through your firewall, through your organization's network, and finally onto the target hosts in a given environment. It's much slower. The external bandwidth can be costly. If the download site is unavailable, your deployment could be delayed. It's better to pull the artifacts from the download site periodically and cache it locally. Write a cron job to fetch the latest files or torment an intern with this task.
The recipe that downloads the artifacts onto the target node assumes there is a locally accessible Web server hosting the relevant agents. This was the simplest solution for this article. Other approaches would be to deploy a Yum repository, a network file share, or set up a Sonatype Nexus solution.
In your environment, install and configure a Web server. Then copy a few Java APM and Machine Agent artifacts from https://download.appdynamics.com/download/ to the Web server directory structure. Validate that you can access the artifacts using curl or wget. Double-check iptables/local firewall rules, file permissions, and SELinux configurations. This step is a crucial prerequisite to allow the cookbook to work.
4. Cookbook Table of Contents
The example cookbook presented in this post is called appd_apm_java. Below is an abridged list of directories you should focus on.
├── attributes
│ └── default.rb
├── files
│ └── default
│ ├── restartTomcat.sh
│ └── tomcat-users.xml
├── recipes
│ ├── Configure.rb
│ ├── InstallAgents.rb
│ ├── Prereqs.rb
│ ├── StartServices.rb
│ ├── WebRetrieve.rb
│ ├── default.rb
├── templates
│ ├── appd.conf.erb
│ ├── controller-info.xml.erb
│ └── ma-controller-info.xml.erb
...
A useful Chef cookbook consists of:
- Default attributes (attributes/default.rb)
- Possibly one or more static text files (files/default/*)
- Usually, one or more template files (templates/*) that configure the deployed software
- The recipes (recipes/*) that direct Chef what to do.
The next sections describe these items in more detail.
5. Prepare Ingredients
Television chefs have their ingredients in tidy cups which they dump in the mixing bowl while talking to the camera. We'll follow the same format here by writing a set of default attributes. Have a look at the abridged attributes/default.rb file.
Default Attributes
...
default[cbname]['source_url'] = "CHANGE_ME!"
default[cbname]['java_apm_prefix'] = "AppServerAgent-"
default[cbname]['java_apm_versbld'] = "20.3.0.29587"
default[cbname]['java_apm_suffix'] = ".zip"
default[cbname]['ma_prefix'] = "appdynamics-machine-agent-"
default[cbname]['ma_version'] = "20.3.2.2520"
default[cbname]['ma_arch'] = ".x86_64"
default[cbname]['ma_suffix'] = ".rpm"
# Controller & Agent Related Defaults
default[cbname]['controller_host'] = "CHANGE_ME!"
default[cbname]['controller_port'] = "8090"
default[cbname]['controller_ssl_enabled'] = "false"
default[cbname]['simple_hostname'] = "false"
default[cbname]['application_name'] = "TomcatAdmin"
default[cbname]['tier_name'] = "Admin"
default[cbname]['account_name'] = "customer1"
default[cbname]['account_key'] = "CHANGE_ME!"
default[cbname]['sim_enabled'] = "true"
default[cbname]['machine_path'] = "Bldg 1|2nd Floor|My Laptop|"
...
The attributes with CHANGE_ME! are a strong hint you need to alter these lines; or, override them with your Chef server's environment/role/node values. All standard Chef cookbook stuff.
The only parts that merit special attention are the way the AppDynamics agent filenames are composed. In the case of Java, it is the string AppServerAgent- plus a version/build number plus the string .zip. The Machine Agent is the string appdynamics-machine-agent- plus version/build number plus an architecture signifier plus the string .rpm.
This cookbook deliberately uses a mix of zip and RPM files for the agents. There is a zip version of the Machine Agent available; but not an RPM for the Java Agent. Feel free to modify the cookbook to use what you prefer.
Static Files
The next batch of ingredients we need to look at is the static files under files/default. The tomcat-users.xml is largely boilerplate from the tomcat-7 package with only one interesting line near the bottom--
<user username="admin" password="password" roles="manager-gui,admin-gui"/>
This line allows me to open the Webapp console on (default) port 8080 and administer my Tomcat instance. The app prompts me to enter a username and password (hard-coded in the static file as admin/password). Change the contents of this file as you see fit.
The other static file is a Bash script that attempts to start the Tomcat instance using systemd. It echoes the results of that attempt to standard output. This file will be installed under the Machine Agent directory structure. It can be used with health rules and actions to trigger a Tomcat restart should the health rule trigger. I put this piece into the cookbook to demonstrate you are not limited to just basic installation and configuration. You can put in other pieces that integrate with the running system and AppDynamics.
Template Files
Lastly, we have template files for Tomcat startup and Machine Agent configuration. These files differ from the static ones in that there are variables enclosed in angle brackets (example: <%= @java_agent %>). Chef leans heavily on the Ruby language's templating capabilities to dynamically substitute the attribute's value during the Chef run at the appropriate place in a template file. This provides a lot of power to customize complex configuration files at runtime.
The template/appd.conf.erb is dropped on the target node under the /etc/tomcat/conf.d/ directory. This tells the Catalina startup script to add AppDynamics parameters to the Java startup command line.
The template/ma-controller-info.xml.erb file is dropped under the Machine Agent conf directory. This template file contains the needed parameters to point the Machine Agent to your AppDynamics Controller.
Chef drops the ".erb" suffix when it writes the transformed template files to the target nodes.
The Recipes
Notice the included recipes in recipes/default.rb are ordered as--
- Prereqs
- WebRetrieve
- InstallAgents
- Configure
- StartServices
Chef executes the included recipes in that order. If there's a critical failure in an included recipe, execution halts with an error message.
The Prereqs recipe handles package installation for Tomcat and some web applications. It also installs the unzip and psmisc packages. On a full-fledged server, these packages are likely already installed. What I found while developing this cookbook using Chef's Test Kitchen was that the CentOS image was so slim it lacked commands such as unzip, pstree, and others. I couldn't install the Java Agent because the unzip command wasn't present. Lastly, Prereqs creates the installation directory where the AppDynamics agents will be installed. Have a look at the attributes/default.rb file for the location.
WebRetrieve uses the remote_file Chef command to pull down the Java and Machine Agents using HTTP. Most of the heavy lifting in this recipe is in the filename concatenation variable declarations. Deriving the artifact file names originating from AppDynamics's download site is the trickiest piece in the whole recipe and cookbook. Once you nail down the naming convention, everything else falls into place.
InstallAgents unpacks the bits onto the target node. The Java APM Agent is only available as a zip file, so unzip is executed as a shell command. This section could be further improved to only unzip when the directory does not already exist. The lack of idempotency shouldn't be critical here. The rpm_package command, by contrast, will not run additional times if the specified Machine Agent version is already installed.
Configure drops all the static and template files where they belong on the target node. Variable substitutions are performed by Chef as needed.
There are a couple of interesting points worth noting in this recipe. First, the machine_path tries to pull the node default value, assuming you set it. If it's not set, then the fictitious value from attributes/default.rb is used. When you look at the Servers tab in the Controller and click on a given node, the value of machine_path appears near the top of the screen. This is handy for hosts racked at fixed positions in a data center. For VMs you may consider inserting meta information by updating node information on the Chef server. I included this feature in the recipe for demonstration purposes.
The other interesting point is that the appd.conf file covers a lot of ground. Some of this should probably not be included on the Java command line (account_key is the most glaring example). I chose to put it all in one Tomcat configuration file for simplicity's sake.
StartServices is very simple. It uses systemd to enable and start both Tomcat and the Machine Agent — that's all there is to it!
6. Preheat the Oven
We need to override some properties across the environments and nodes in Chef. Here is what I used in my dev environment.
$ knife environment show dev
chef_type: environment
cookbook_versions:
default_attributes:
description: Global settings for development environment
json_class: Chef::Environment
name: dev
override_attributes:
appd_apm_java:
account_key: 7f66be8e-026f-4741-9b8e-9a8eb1a1d067
application_name: TomcatAdmin-Dev
controller_host: ncm003
dir_group: tomcat
dir_user: tomcat
java_apm_versbld: 4.5.19.29609
ma_version: 4.5.16.2357
tier_name: WebUI
Notice I downgraded the java_apm_versbld to an older version than what's in the attributes/default.rb file. When Chef runs on my development workstation, the older version will be deployed.
Let's take a look at what Chef knows about my wkstn01 node.
$ knife node show wkstn01 --medium
Node Name: wkstn01
Environment: dev
FQDN: wkstn01
IP: 192.168.4.44
Run List: recipe[appd_apm_java]
Roles:
Recipes: appd_apm_java, appd_apm_java::default, appd_apm_java::Prereqs, appd_apm_java::WebRetrieve, appd_apm_java::InstallAgents, appd_apm_java::Configure, appd_apm_java::StartServices, appd_apm_java::test
Platform: centos 7.7.1908
Tags:
Attributes:
appd_apm_java:
machine_path: San Diego|Floor 6|Rm 1|
tags:
I added the machine_path attribute to the wkstn01 node to reflect where it's physically located. The recipe that configures the Machine Agent uses this attribute to show location data in the Servers view of the Controller.
Lastly, remember to use the knife command to upload the appd_apm_java cookbook to your Chef server. Validate it is on the server with the desired version number in the metadata.rb file.
7. Cook the Entrée
It's finally time to put the entrée in the oven! The abridged output shown below is from a brute force Chef run on the target node.
$ knife bootstrap wkstn01 -P CHANGE_ME! -U root -E dev -r 'recipe[appd_apm_java]' --ssh-verify-host-key 'never'
Connecting to wkstn01
Performing legacy client registration with the validation key at /Users/scortrig/Google Drive File Stream/My Drive/projects/chef/development/.chef/ncm-validator.pem...
Delete your validation key in order to use your user credentials for client registration instead.
Bootstrapping wkstn01
[wkstn01] -----> Existing Chef Infra Client installation detected
[wkstn01] Starting the first Chef Infra Client Client run...
[wkstn01] Starting Chef Infra Client, version 15.8.23
…
No surprises here. Incorporating this approach into a CI/CD pipeline provides a lot of power. For the sake of brevity, I did not incorporate a Chef role; but adding that would provide you even more control over the cookbook's attributes at runtime.
8. Garnish and Serve
After all that prep, here's the finished product. The Tiers & Nodes view in the Controller shows my workstation with the Java APM Agent at 40% up and the Machine Agent at 20% up (over the last 5-minute time window).
If you consult the Servers view, the machine_path attribute shows the San Diego location instead of the default Bldg 1 value baked into the cookbook's default attributes.
To see Business Transactions, you need to exercise the Tomcat Webapp console with the following URLs:
- localhost:8080/
- localhost:8080/manager/status/all
- localhost:8080/docs/html-manager-howto.html
- localhost:8080/manager/html/list
- localhost:8080/manager/status
You should see the Tomcat administrator console. If you're trying to use the actual hostname from a remote machine, you may have to modify the firewall rules to allow external traffic to your target host on port 8080.
9. And for Dessert…
This article is just the starting point for automating AppDynamics in your organization. Hopefully, this cookbook is streamlined enough that you can enhance it to cover use cases beyond Apache Tomcat on CentOS 7.x.
Experiment and share your experiences with the AppDynamics community below.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form