Table of Contents
Many of our customers use the Controller Metrics and Events API to retrieve metric data information and information on various types of activities in a monitored environment, including Controller events. This article outlines a customer's use case and includes a generalized example script.
Use Case
The customer is using the AppDynamics metrics API every minute to load the metric data from the Controller to an external system. The customer has multiple Controllers in the AppDynamics SaaS environment and hosts hundreds of applications. They load dozens of metrics from each application.
The customer is using the API for the following use cases:
- Use a third-party tool for troubleshooting in cases where they need 1-minute granularity for longer than 24 hours. For example, the customer often needs to perform root cause analysis but does not get to it until a few days after the incident occurred.
- Use third-party tools to calculate percentiles, consolidate metrics across different controllers at the app or tier level as required and create alerts.
- Use a third-party tool to retain data for several years. Since their peak periods occur just once per year, customers might need data to be retained for 3 years to provide meaningful comparisons and plan their future deployments better.
Problem
The customer uses a set of python scripts to retrieve the metrics. They consistently run into timeout errors and worry that the APIs are not scaling to their requirements.
Our team engaged with the customer and offered to review their process and scripts. Upon review, we found that the python script opens 1000+ of concurrent connections to the Controller. There are only 10 threads dedicated to processing the REST requests on the server side. As a result, the threads wait indefinitely and then timeout.
Solution
- Use a thread pool to invoke the REST APIs.
- This way the customer will have control on the concurrency. This should help to sustain 10 concurrent requests on the server side, getting the best performance.
- Decrease the frequency of the job.
- For example, run the job every 5 minutes rather than every one minute.
- We recommend restricting the number of Metric data points returned per REST call to less than 50K.
Metric REST API Example
A RESTful API is an application program interface (API) that uses HTTP requests to GET data. The following section gives some examples of wildcards usage on Metric Data REST API. (This section scrolls horizontally.) The same example is included in the attached Metric REST API Example document for easy copy and paste.
URL: /controller/rest/applications/100/metric-data?metric-path=<metric-path>&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true
Method: GET
URL Parameters:
metric-path: Business%20Transaction%20Performance%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A
time-range-type: BEFORE_NOW
duration-in-mins: 1
rollup=true
Success Response:
Response headers:
HTTP/1.1 200 OK
Server: AppDynamics
Content-Type: application/xml
Error Response:
REST endpoints can fail is many ways. From unauthorized access to wrongful parameters etc. Listed few error types:
HTTP/1.1 403 Forbidden
Server: AppDynamics
Content-Language:
Content-Type: text/html
Incorrect Parameters:
Data type ("text"|"bin"|""): text
Response code: Non HTTP response code: java.net.URISyntaxException
Response message: Non HTTP response message: Malformed esAPPDe
Sample Request Call:
GET https://hostname/controller/rest/applications/100/metric-data?metric-path=Business%20Transaction%20Performance%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A%7C%2A&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true
Cookie Data:
JSESSIONID=c45eeb9747d8aa24338bc2b973cf;
X-CSRF-TOKEN=9a3dddf1c54ebca291c46ab10522faa7bae7c477
Request Headers:
Connection:keep-alive
Accept-Language:en-US,en;q=0.5
Accept-Encoding:gzip, deflate
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: application/json, text/plain, */*
Authorization: Basic Q2FwaXRhbE9uZS1Qcm9kJTQwQ2FwaXRhbE9uZS1Qcm9kOnRlc3QxMjM=
X-CSRF-TOKEN:9a3dddf1c54ebca291c46ab10522faa7bae7c477
Sample Response Data:
<metric-datas><metric-data>
<metricId>1578650</metricId>
<metricPath>Business Transaction
Performance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|Individual
Nodes|IP-200-254-23-APPDInterface_PROD/APPDWebServices|Average Block
Time(ms)</metricPath>
<metricName>BTM|BTs|BT:23889|Component:617|Average
Block Time (ms)</metricName>
<frequency>ONE_MIN</frequency>
<metricValues>
<metric-value>
<startTimeInMillis><timeInMilliseconds></startTimeInMillis>
<value>0</value>
<min>0</min>
<max>0</max>
<current>0</current>
<sum>0</sum>
<count>0</count>
<standardDeviation>0.0</standardDeviation>
<occurrences>0</occurrences>
<useRange>true</useRange>
</metric-value>
</metricValues>
</metric-data>
</metric-datas>
Metric Calls example of wildcards at various level:
1.Six wild card (|*|*|*|*|*|*) <metricPath>BusinessTransactionPerformance|BusinessTransactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|IndividualNodes|IP-200-254-23-APPDInterface_PROD/APPDWebServices| METRIC-NAME</metricPath>
2.Five wild card (|*|*|*|*|*) <metricPath>BusinessTransactionPerformance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|IndividualNodes | METRIC-NAME</metricPath>
3.Four wild card (|*|*|*|*) <metricPath>BusinessTransactionPerformance|Business Transactions|APPDInterface_PROD/APPDWebServices|BVLInterfaceWS2.Process|METRIC-NAME</metricPath>
4.Three wild card (|*|*|*) <metricPath>Business TransactionPerformance|Business Transactions| APPDInterface_PROD/APPDWebServices|METRIC-NAME</metricPat>
5.Two wild card (|*|*) <metricPath>BusinessTransactionPerformance|Business Transactions |METRIC-NAME</metricPath>
6.One wild card (|*) <metricPath>BusinessTransactionPerformance|METRIC-NAME</metricPath>
Request to keep metrics under 50K:
https://<HostName>/controller/rest/applications/<ApplicationID>/metric-data?metric-path=Business%20Transaction%20Performance%7C%2A%7CTOMCAT-APP%7CTransactions.SA%7C%2A&time-range-type=BEFORE_NOW&duration-in-mins=1&rollup=true
If REST Call is fetching more than 50K metrics, you can replace wild characters with hard-coded values (e.g. TOMCAT-APP & Tranaction.SA), which will control metric count.
Find a test API script on Github here: Appdynamics/metric-loader
