The following article explains the considerations and challenges when using any application that is a heavy user of I/O, like AppDynamics Controllers running on Virtual Machines (VMs). It is applicable to AppDynamics version 4.3.x.
AppDynamics’ metric intake is sensitive to storage latency at scale.
Specifically:
- AppDynamics controllers receive metrics every minute from every Agent, and are marked with a minute number.
- When the controller determines that it has buffered up packets for all the data for a given minute, it puts that data into an insert buffer.
- The insert buffer contains all the metrics (plus derived metrics), from all the nodes, for the same minute.
- The metric buffer is then sent to MySQL. The controller then issues SQL queries to roll the new data up to the tier level, collapse out the node IDs, then roll up to the application level, collapsing out the tier IDs, leaving Business Transactions (BTs).
- This all happens within a minute, since there is data dependency from node to tier to application.
The result is if the 60 second (one minute) timeframe is exceeded by the execution of the path, system performance will degrade nonlinearly.
The rule of thumb is that the filesystem must be able to absorb 200MB/sec per million metrics/min measured with fio. This is difficult to achieve on VM infrastructure above a few million metrics per minute.
We have a benchmark using FIO that closely approximates our workload available on request. Contact Professional Services or Support for assistance.
Recommendation: At this time we do not recommend running large or extra large AppDynamics Controllers on virtual machines.
Scalability is a key differentiator of the AppDynamics compared to competing solutions. We continue to invest significant resources in improving metric ingestion at scale and are working on changes that will limit sensitivity to I/O latency.
Related Links:
