HTTP Request parameters of web application extraction

[srinirand]

Hi all,

 As part of custom appdynamics extension to monitor a web application, we would like to extract the HTTP Request parameters (all of them) and do some processing on them. We were hoping to do this as an extension, but are lost on how to inspect the HTTPRequest submitted by the user.

We thought HTTP Data Collectors do not fit our use case as we do not wish a few of the HTTP request, rather we need all of them.

Kindly suggest or point us to any sample code you may have.

[srinirand]

Ok...here we go.

We have 80 odd applications supporting a legal regulation. As part of the
regulation, we need to run some generic rules against the input submitted
by end users. We have appdynamics already monitoring all these applications
and are hence looking to extend appdynamics agent (machine or application)
to extract the parameters from request and run the rules and report to
controller if there are any issues found.

We can then schedule alert emails based on the events reported.

[Peter.Holditch]

OK, can you give an example of some inputs and a rule?

And what should happen if the rule is violated?  SImply an alert, or should the application behaviour change in case of a violation?

[srinirand]

For now we are looking to only send alerts to the audit team though in the
future would not rule out more functionality addition as a reaction.

A simple rule is that any mention of "religion" or "race" as part of the
many input entries of these 80 odd applications need to be alerted as these
are cases of possible favouritism that needs additional audit scrutiny. So
the rule screens for matches of mention of religion or race on all the
inputs coming in the Http request.

[Peter.Holditch]

So presumably, you have a regular expression (referred to as 'regular-expression' below) that you would run on a string field, and when it matches that is indicative of a mention of religion.

On that assumption, if you collected transaction analytics data from all transactions that could contain mention of religion, you could then use ADQL to identify the offending ones.  e.g.

SELECT count * FROM transactions WHERE segments.userData.inputText REGEXP('regular-expression')

You could use this query to create a metric that you could alert on if non zero using a health rule in the usual way.

This would probably be the best way to approach your problem using the AppDynamics platform.

[srinirand]

Thank you Peter for your suggestions so far.

Some of the challenges with the afore mentioned solution would be:

1. The rule I had mentioned was one simple rule and there are many others that need to be applied.

2. The approach intends to capture the field into appdynamics schema. With the scale of inputs & applications, this would quickly grow to unmanageable size and with AppDynamics on MySQL, this would start to deteriorate the performance. There are usually multiple fields per form per application that we need to run the rules through. This would work if we could report to the appdynamics controller only if there is negative hit on the rules which is usually rare (maybe less than 5% of the time).

I was hoping that there could be some means of hooking into appdynamics given that request level stats are already being pulled out (like url, User Principal etc). The last route on my mind is to extend the machine agent and start listening into the HTTP ports on all the requests coming in, then apply the rules and report back to controller (very reluctant on this approach as it complicates on SSL, HTTP port listening etc)

[Peter.Holditch]

The event data stored by analytics is not held in MySQL, so you need not worry on that front.

Based on your other comments though, I am just not convinced that trying to implement your rules within AppDynamics is the correct architectural approach to solve this specific problem

Maybe you could apply your rules in code and publish the failure count via a JMX MBean?  You could use AppD to monitor the MBean and alert on the breaches that way?

[srinirand]

The reasons for exploring the AppDynamics customization route are:

1. AppDynamics already inspects requests

2. Rolling out to the scale of so many application is easy as they all already have AppdYnamics configured.

3. Due to the impact on each of those 80 odd applications, we wanted to keep them outside of the application code.

4. If we could do it at appdynamics, we can avoid impact on so many applications and that save us regression effort and centralization of code related to this audit rather than distribution of same with each application.