How to observe OpenTelemetry demo app in Splunk AppDynamics

[Saurabh.Deshpande]

Here's how to configure the OpenTelemetry Community Demo application to send telemetry data to Splunk AppDynamics. 

Since its launch in May 2019, the OpenTelemetry (OTel) Project has become one of the most popular open-source projects after Kubernetes®. For newcomers to the observability domain, OpenTelemetry™ provides a standard way to collect telemetry data (metric, logs and traces) from software applications and send it to one or more backends to analyze application performance. The backends can be open source (Jaeger ,Zipkin etc.), commercial (such as Splunk AppDynamics, Splunk Observability) or both.  

To enable faster adoption and showcase instrumentation best practices, the OTel community has built a demo application, OpenTelemetry Community Demo. In this blog, I'll show how to configure the OpenTelemetry demo to send Traces data to Splunk AppDynamics for further analysis.  

Backstory: Splunk AppDynamics and OpenTelemetry demo 

Splunk AppDynamics provides full stack observability of hybrid and on-prem applications and their impact on business performance. In addition to the proprietary ingestion format, AppDynamics also supports OpenTelemetry trace ingestion from various language agents (Java, dotnet, python, golang etc.) giving customers more options in how they want to ingest telemetry data. 

OpenTelemetry Community Demo is a simulated version of an eCommerce store selling astronomy equipment. The app consists of 14+ microservices communicating with each other via HTTP or grpc. The microservices are built using a variety of programming languages (Java, Javascript, C#, etc.) and instrumented using OpenTelemetry (auto, manual, or both). The diagram below shows the data flow and programming languages used.  

(Image credit:  OpenTelemetry Demo contributors.) 

In addition to the microservices shown here, the demo app also comes with supporting components such as OpenTelemetry Collector, Grafana, Prometheus, and Jaeger to export and visualize traces, metrics, and so on. The OpenTelemetry Collector is highly configurable. Once exporters for various backends are defined and enabled in the service pipeline, the Collector can be set up to send telemetry data to multiple backends simultaneously. The diagram below shows the OTel demo with supporting components, as well as a dotted line to Splunk AppDynamics, which we will configure in the next section. 

Sending OpenTelemetry trace data to Splunk AppDynamics 

1. Using the steps described in the Open Telemetry demo Docker deployment documentation, deploy the demo app on your local machine. Confirm it’s working by going to http://localhost:8080/ and completing an item checkout workflow. 
2. Contact your Splunk AppDynamics account representative to set up an AppDynamics account for your company. The account will have a URL format similar to https://<your-company/account-name>.saas.appdynamics.com and will be the central location where you'll see all telemetry data from your applications.  
3. Generate API Key by going to your AppDynamics url >  Otel > Get Started > Access Key.
4. Go to the Processors, Exporters, and Service Configuration sections and note down the values of below Keys. We will use them in the next section: 
   1. appdynamics.controller.host  
   2. appdynamics.controller.account 
5. As described below, update the file src/otel-collector/otelcol-config-extras.yml in your cloned repo. Alternatively, copy the GitHub gist. Make sure there are no yaml validation errors by opening this file in an IDE with yaml support (VSCode etc.). The contents of this file get merged with src/otelcollector/otelcol-config.yml at runtime to create the consolidated OpenTelemetry Collector configuration. 

     processors:
       resource:
         attributes:
         - key: appdynamics.controller.account
           action: upsert
           value: "from AppD account url > Otel > Configuration > Processor section"
         - key: appdynamics.controller.host
           action: upsert
           value: "from AppD account url > Otel > Configuration > Processor section"
         - key: appdynamics.controller.port
           action: upsert
           value: 443
         - key: service.namespace
           action: insert
           value: otel-demo-local-mac
       batch:
         timeout: 30s
         send_batch_size: 90
   
     exporters:
       otlphttp/appdynamics:
         endpoint: "from AppD account url > Otel > Configuration > Exporter section"
         headers: {"x-api-key": "from AppD account url > Otel > Configuration > API Key"}
   
     service:
       pipelines:
         traces:
           receivers: [otlp]
           processors: [transform, resource, batch]
           exporters: [otlp, spanmetrics, otlphttp/appdynamics, debug]

6. Stop the docker containers (make stop ) and then start them (make start). Wait a few minutes and confirm that you can access the OpenTelemetry demo app UI at ​​​​http://localhost:8080/. Next, log in to your Splunk AppDynamics URL. You'll then see a service flow map that shows various microservices and the interactions between them.
7. An observability platform should be able to detect an increase in error rates of the microservices it’s monitoring. Fortunately, the OpenTelemetry demo has an error injection capability via feature flags to test this functionality. Go to the feature flag UI at ​​​http://localhost:8080/feature/ and enable the productCatalogFailure feature flag. This will cause the product catalog service to return an error for a specific product ID and respond correctly to all other product IDs. Note the increase in error rate in home page. To view errors details, click on Troubleshoot > Errors > Error Transactions > Details. AppDynamics accurately captures the error reason as ”Product Catalog Feature Flag Enabled”. AppDynamics provides health rules and alerts functionality to respond quickly to such situations. 

OpenTelemetry and Splunk AppDynamics 

The OpenTelemetry Community Demo application is a valuable and safe tool for learning about OpenTelemetry and instrumentation best practices. In this blog, we showed how to configure the demo app to send telemetry data to Splunk AppDynamics. We also explored some key Splunk AppDynamics features such as FlowMap, APM metrics, and an observed increase in error rates via a fault-injection scenario.  

Interested in trying this workflow to learn more about OpenTelemetry and Splunk AppDynamics? Go over Splunk AppDynamics OpenTelemetry documentation and  Sign up for a free trial of Splunk AppDynamics. Then, clone and deploy the opentelemetry-demo repo and send its telemetry data to Splunk AppDynamics to gain valuable insights.