Cisco AppDynamics Community

Abhi.Bajaj · ‎04-16-2024

We’ll explore how to deploy a robust application monitoring solution using AWS ECS (Elastic Container Service) and Cisco AppDynamics. This integration allows businesses to leverage the scalability of AWS and the comprehensive monitoring capabilities of AppDynamics, ensuring applications perform optimally in a cloud environment.

What is AWS ECS?

AWS ECS (Elastic Container Service) is a highly scalable, high-performance container orchestration service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances. ECS eliminates the need to install, operate, and scale your own cluster management infrastructure, making it easier to schedule and run Docker containers on AWS.

What is AppDynamics?

AppDynamics is a powerful application performance management (APM) and IT operations analytics (ITOA) tool that helps monitor, analyze, and optimize complex software environments. It provides real-time visibility and insights into the performance of applications, enabling organizations to quickly detect, diagnose, and resolve issues to improve user experience and operational efficiency.

Application Image

The application we are deploying is packaged in a Docker image, abhimanyubajaj98/java-tomcat-wit-otel-app-buildx, which contains a Java-based web application running on Apache Tomcat. This image is enhanced with the OpenTelemetry Java agent to facilitate detailed performance monitoring and telemetry.

Configuration Overview

Our setup involves several AWS resources managed through Terraform, a popular infrastructure as code (IaC) tool, ensuring our infrastructure is reproducible and maintainable. Below is a high-level overview of our configuration:

ECS Cluster

AWS ECS Cluster: Acts as the hosting environment for our Docker containers.
Task Definitions: Specifies the Docker containers to be deployed, their CPU and memory allocations, and essential configurations such as environment variables and network mode.

IAM Roles and Policies

IAM Roles and Policies: Ensure proper permissions for ECS tasks to interact with other AWS services, such as retrieving Docker images from ECR and sending logs to CloudWatch.

Container Setup

Machine Agent Container: Hosts the AppDynamics machine agent, configured to monitor the underlying EC2 instances and collect machine metrics.
Java Application Container: Runs the main Java application with OpenTelemetry instrumentation, configured to send telemetry data to AppDynamics.
OpenTelemetry Collector Container: Aggregates and forwards telemetry data to the AppDynamics controller.

Security and Network

Network Mode: Uses host networking to ensure that containers can communicate efficiently and leverage the networking capabilities of the host EC2 instance.
Security Groups: Configured to allow appropriate inbound and outbound traffic necessary for operation and monitoring.

Detailed Steps and Configuration

ECS Cluster Creation: Define an ECS cluster using Terraform to serve as the runtime environment for the containers.
Task Definitions: Specify containers that need to be run as part of the ECS service. Include detailed settings for:

Image versions
CPU and memory requirements
Environment variables for configuration
Volume mounts for persistent or shared data

IAM Configuration: Set up IAM roles and attach policies that grant necessary permissions for ECS tasks, including logging to CloudWatch and accessing ECR for image retrieval.

Logging and Monitoring: Configure CloudWatch for logging and set up AppDynamics for advanced monitoring, linking it with OpenTelemetry for comprehensive observability. Deployment and Management: Use Terraform to manage deployment and updates to the infrastructure, ensuring consistency and reproducibility.

provider "aws" {
  region = "us-east-1" # Change to your preferred AWS region
}

resource "aws_ecs_cluster" "ecs_cluster" {
  name = "ecs_cluster_with_agents"
  tags = {
    owner                 = "Abhi Bajaj"
  }
}
resource "aws_ecs_task_definition" "container_tasks" {
  family = "container_tasks"
  network_mode = "host"
  container_definitions = jsonencode([
    {
      "name" : "machine-agent-container",
      "uid"      : "0",
      "privileged": true,
      "image" : "docker.io/appdynamics/machine-agent:root-latest",
      "cpu" : 256,
      "memory" : 512,
      "essential" : true,
      "environment" : [
        {
          "name" : "APPDYNAMICS_CONTROLLER_HOST_NAME"
          "value" : "xxx.saas.appdynamics.com"
        },
        {
          "name" : "APPDYNAMICS_CONTROLLER_PORT"
          "value" : "443"
        },
        {
          "name" : "APPDYNAMICS_AGENT_ACCOUNT_ACCESS_KEY"
          "value" : "xxx"
        },
        {
          "name" : "APPDYNAMICS_AGENT_ACCOUNT_NAME"
          "value" : "xxx"
        },
        {
          "name" : "APPDYNAMICS_AGENT_ACCOUNT_NAME"
          "value" : "xxx"
        },
        {
          "name" : "APPDYNAMICS_AGENT_UNIQUE_HOST_ID"
          "value" : "machine_agent_ecs"
        },
        {
          "name" : "APPDYNAMICS_CONTROLLER_SSL_ENABLED"
          "value" : "true"
        },
        {
          "name" : "APPDYNAMICS_SIM_ENABLED"
          "value" : "true"
        },
        {
          "name" : "APPDYNAMICS_DOCKER_ENABLED"
          "value" : "true"
        }
      ],
      "mountPoints" : [
        {
          "containerPath" : "/hostroot/proc",
          "sourceVolume" : "proc",
          "readOnly" : true
        },
        {
          "containerPath" : "/hostroot/sys",
          "sourceVolume" : "sys",
          "readOnly" : false
        },
        {
          "containerPath" : "/hostroot/etc",
          "sourceVolume" : "etc",
          "readOnly" : false
        },
        {
          "containerPath" : "/var/run/docker.sock",
          "sourceVolume" : "docker_sock",
          "readOnly" : false
        }
        // Add more mount points as needed
      ]
    },
    {
      "name" : "ecs_with_otel_java_app",
      "image" : "abhimanyubajaj98/java-tomcat-wit-otel-app-buildx",
      "cpu" : 512,
      "memory" : 1024,
      "network_mode" : "host",
      "privileged": true,
      "essential" : true,
      "environment" : [
        {
          "name" : "JAVA_TOOL_OPTIONS"
          "value" : "-Dotel.resource.attributes=service.name=ECS_otel_abhi,service.namespace=ECS_otel_abhi"
        }
      ]
    },
    {
      "name" : "OpenTelemetryCollector",
      "image" : "appdynamics/appdynamics-cloud-otel-collector",
      "privileged": true,
      "network_mode" : "host",
      "memory" : 1024,
      "cpu" : 512,
      "ports": [
            {
                "containerPort": 13133,
                "hostPort": 13133
            },
            {
                "containerPort": 4317,
                "hostPort": 4317
            },
            {
                "containerPort": 4318,
                "hostPort": 4318
            }
        ],
        "environment" : [
        {
          "name" : "APPD_OTELCOL_CLIENT_ID"
          "value" : "xxx"
        },
        {
          "name" : "APPD_OTELCOL_CLIENT_SECRET"
          "value" : "xxxx"
        },
        {
          "name" : "APPD_OTELCOL_TOKEN_URL"
          "value" : "https://xxx-pdx-p01-c4.observe.appdynamics.com/auth/4f8da76d-01a8-4df6-85cd-3a111fba946e/default/oauth2/token"
        },
        {
          "name" : "APPD_OTELCOL_ENDPOINT_URL"
          "value" : "https://xxx-pdx-p01-c4.observe.appdynamics.com/data"
        }
        ],
        "mountPoints" : [
        {
          "containerPath" : "/hostroot/etc",
          "sourceVolume" : "etc",
          "readOnly" : true
        },  
        {
          "containerPath" : "/hostroot/sys",
          "sourceVolume" : "sys",
          "readOnly" : false
        }
        ]
    }  
  ])

  volume {
    name      = "proc"
    host_path = "/proc"
  }

  volume {
    name      = "sys"
    host_path = "/sys"
  }

  volume {
    name      = "etc"
    host_path = "/etc"
  }

  volume {
    name      = "docker_sock"
    host_path = "/var/run/docker.sock"
  }
resource "aws_ecs_service" "container_services" {
  name            = "container-services"
  cluster         = aws_ecs_cluster.ecs_cluster.id
  task_definition = aws_ecs_task_definition.container_tasks.arn
  desired_count   = 1
}


##############################################################################################################


resource "aws_launch_template" "ecs_launch_template" {
  name          = "alma"
  image_id      = "ami-xxxxx" # Amazon ECS-Optimized Amazon Linux 2 (AL2) x86_64 AMI
  instance_type = "t2.medium"
  user_data = base64encode(<<EOF
#!/bin/bash
sudo su
echo ECS_CLUSTER=ecs_cluster_with_agents >> /etc/ecs/ecs.config
wget https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_amd64/amazon-ssm-agent.deb
dpkg -i amazon-ssm-agent.deb
systemctl enable amazon-ssm-agent
EOF
  )
  vpc_security_group_ids = ["sg-xxxx"]
  iam_instance_profile {
    name = aws_iam_instance_profile.dev-resources-iam-profile.name
  }

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name = "ECS_with_Agents"
      Owner = "abhibaj@cisco.com"
    }
  }
}


resource "aws_autoscaling_group" "auto_scaling_group" {
  name                      = "ecs_asg"
  availability_zones        = ["us-east-1a", "us-east-1b"]
  desired_capacity          = 1
  min_size                  = 1
  max_size                  = 10
  health_check_grace_period = 300
  health_check_type         = "EC2"

  launch_template {
    id = aws_launch_template.ecs_launch_template.id
  }
}

resource "aws_ecs_capacity_provider" "provider" {
  name = "alma"
  auto_scaling_group_provider {
    auto_scaling_group_arn = aws_autoscaling_group.auto_scaling_group.arn

    managed_scaling {
      status                    = "ENABLED"
      target_capacity           = 100
      minimum_scaling_step_size = 1
      maximum_scaling_step_size = 100
    }
  }
}

resource "aws_ecs_cluster_capacity_providers" "providers" {
  cluster_name       = aws_ecs_cluster.ecs_cluster.name
  capacity_providers = [aws_ecs_capacity_provider.provider.name]
}





#############################################



resource "aws_iam_instance_profile" "dev-resources-iam-profile" {
  name = "ec2_profile_for_services_otel"
  role = aws_iam_role.dev-resources-iam-role.name
}

resource "aws_iam_role" "dev-resources-iam-role" {
  name               = "role_for_services_ec2_otel"
  description        = "The role for the developer resources on EC2"
  assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Principal": {"Service": "ec2.amazonaws.com"},
"Action": "sts:AssumeRole"
}
}
EOF
  tags = {
    Owner = "abhibaj"
  }
}
resource "aws_iam_role_policy_attachment" "dev-resources-ssm-policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  role       = aws_iam_role.dev-resources-iam-role.name
}
resource "aws_iam_role_policy_attachment" "ecr_read_only_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.dev-resources-iam-role.name
}

resource "aws_iam_role_policy_attachment" "ecs_full_access_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonECS_FullAccess"
  role       = aws_iam_role.dev-resources-iam-role.name
}


resource "aws_iam_role_policy_attachment" "ecs_ecs_task_execution_policy_attachment" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
  role       = aws_iam_role.dev-resources-iam-role.name
}

resource "aws_iam_role_policy_attachment" "ecs-instance-role-attachment" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
  role       = aws_iam_role.dev-resources-iam-role.name
}

To deploy->
Please edit ENV Variables subsituted with “xxx”

Once done, Run

terraform init
terraform apply — auto-approve

How it will look on your UI is:

Screenshot 2024-04-16 at 12.51.27 AM.png