Setting up RabbitMQ and Monitoring with Splunk AppDynamics

[Abhi.Bajaj]

Comprehensive Guide to RabbitMQ Setup, Integration with Python, and Monitoring with AppDynamics

Introduction

RabbitMQ is a powerful open-source message broker that supports a variety of messaging protocols, including AMQP. It allows developers to build robust, scalable, and asynchronous messaging systems. However, to ensure optimal performance, monitoring RabbitMQ metrics is crucial. This tutorial walks you through setting up RabbitMQ, integrating it with a Python application, and monitoring its metrics using AppDynamics.

Step 1: Setting Up RabbitMQ

1.1 Install RabbitMQ via Docker

To quickly get RabbitMQ up and running, use the official RabbitMQ Docker image with the management plugin enabled.

Run the following command to start RabbitMQ:

docker run -d --hostname my-rabbit --name rabbitmq \
  -e RABBITMQ_DEFAULT_USER=guest \
  -e RABBITMQ_DEFAULT_PASS=guest \
  -p 5672:5672 -p 15672:15672 \
  rabbitmq:management

• Management Console: Accessible at http://localhost:15672.
• Default Credentials:

Username: guest

Password: guest

1.2 Verify the Setup

Once the container is running, verify the RabbitMQ server by accessing the Management Console in your browser. Alternatively, test the API endpoint:

curl -u guest:guest http://localhost:15672/api/overview

This should return RabbitMQ metrics in JSON format.

Step 2: Writing a Simple RabbitMQ Producer and Consumer in Python

2.1 Install Required Library

Install the pika library for Python, which is used to interact with RabbitMQ:

pip install pika

2.2 Create the Producer Script (send.py)

This script connects to RabbitMQ, declares a queue, and sends a message.

import pika

# Connect to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare a queue
channel.queue_declare(queue='hello')

# Publish a message
channel.basic_publish(exchange='', routing_key='hello', body='Hello RabbitMQ!')
print(" [x] Sent 'Hello RabbitMQ!'")

connection.close()

2.3 Create the Consumer Script (receive.py)

This script connects to RabbitMQ, consumes messages from the queue, and prints them.

import pika

# Connect to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()

# Declare a queue
channel.queue_declare(queue='hello')

# Define a callback to process messages
def callback(ch, method, properties, body):
    print(f" [x] Received {body}")

channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=True)

print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

2.4 Test the Application

a. Run the consumer in one terminal:

python3 receive.py

b. Send a message from another terminal:

python3 send.py

c. Observe the message output in the consumer terminal.

 [x] Sent 'Hello RabbitMQ!'
 [x] Received b'Hello RabbitMQ!'

Step 3: Monitoring RabbitMQ with AppDynamics

3.1 Configure RabbitMQ Management Plugin

Ensure that the RabbitMQ Management Plugin is enabled (default in the Docker image). It exposes an HTTP API that provides metrics.

3.2 Create a Custom Monitoring Script

Use a shell script to fetch RabbitMQ metrics and send them to the AppDynamics Machine Agent.

script.sh

#!/bin/bash

# RabbitMQ Management API credentials
USERNAME="guest"
PASSWORD="guest"
URL="http://localhost:15672/api/overview"

# Fetch metrics from RabbitMQ Management API
RESPONSE=$(curl -s -u $USERNAME:$PASSWORD $URL)

if [[ $? -ne 0 || -z "$RESPONSE" ]]; then
  echo "Error: Unable to fetch RabbitMQ metrics"
  exit 1
fi

MESSAGES=$(echo "$RESPONSE" | jq '.queue_totals.messages // 0')
MESSAGES_READY=$(echo "$RESPONSE" | jq '.queue_totals.messages_ready // 0')
DELIVER_GET=$(echo "$RESPONSE" | jq '.message_stats.deliver_get // 0')

echo "name=Custom Metrics|RabbitMQ|Total Messages, value=$MESSAGES"
echo "name=Custom Metrics|RabbitMQ|Messages Ready, value=$MESSAGES_READY"
echo "name=Custom Metrics|RabbitMQ|Deliver Get, value=$DELIVER_GET"

3.3 Integrate with AppDynamics Machine Agent

1. Place the Script: Copy the script.sh script to the Machine Agent monitors directory:

cp script.sh <MachineAgent_Dir>/monitors/RabbitMQMonitor/

2. Create monitor.xml: Create a monitor.xml file to configure the Machine Agent:

<monitor>
    <name>RabbitMQ</name>
    <type>managed</type>
    <enabled>true</enabled>
    <enable-override os-type="linux">true</enable-override>
    <description>RabbitMQ
    </description>
    <monitor-configuration>
    </monitor-configuration>
    <monitor-run-task>
        <execution-style>periodic</execution-style>
        <name>Run</name>
        <type>executable</type>
        <task-arguments>
        </task-arguments>
        <executable-task>
            <type>file</type>
            <file>script.sh</file>
        </executable-task>
    </monitor-run-task>
</monitor>

3. Restart the Machine Agent: Restart the agent to apply the changes:

cd <MachineAgent_Dir>/bin
./machine-agent &

Step 4: Viewing Metrics in AppDynamics

1. Log in to your AppDynamics Controller.
2. Navigate to Servers > Custom Metrics.
3. Look for metrics under:
   Custom Metrics|RabbitMQ

You should see metrics like:

• Total Messages
• Messages Ready
• Deliver Get