Alerting for active slow queries before they finish

[Brian.Gupta]

Environment: Java app running on EC2 in AWS

We'd like to get notified when a business transaction is taking a long time while it is in progress, instead of waiting for the transaction to complete. We have the AppD java agent running with "-javaagent" monitoring the application and we are seeing occasional very slow to finish transactions, taking more than a minute to complete. We have a policy for these transactions that sends a message to a slack channel and are getting 3-4 messages a day. Usually, these line up with database queries that are taking the majority of the elapsed time.  The vast majority of the time transactions complete with an average of 6ms per event.  So, we are talking orders of magnitude slower than normal.

Java: openjdk version "11" 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

Agent: JDK 8+: AppServerAgent-1.8-21.11.4.33358.zip

Both are old but have been tested with this application.