Apache and Nginx have a known issue that drops headers containing underscores.
How can I fix these headers, since they're needed for correlation purposes?
Problem
Recently, Apache and Nginx began dropping headers that include underscores in their names, like “ADRUM_1”.
AppDynamics needs this type of header name for correlation purposes. So, if Apache and Nginx interpret header names with underscores as broken and thus drop them, the necessary correlation won’t take place.
Verification
Mobile app identification happens with a header with an underscore (e.g., adrum_1). For browsers, it identifies by ADRUM: isajax:true (which does not have an underscore).
For BRUM correlation, the headers look like this:
< ADRUM_0: g%3A89b16a83-9478-40e9-a625-bf101d57c6f018
< ADRUM_1: i:563
< ADRUM_2: n:customer1_cef43e07-de57-4fce-939a-54a6c5dcdf56
< ADRUM: isAjax:true
Whereas the correlation header in case of Mobile will appear as follows:
< ADRUM_0: clientRequestGUID:3de4f995-e0ba-4d59-971f-d653e1ea44cf
< ADRUM_1: globalAccountName:customer1_7cc5e33b-579e-4e57-b302-b1eb63b30a0c
< ADRUM_1: isMobile:true
Broken Correlation Headers
You can check these headers with the following command on a valid endpoint of your application:
curl -v --header"ADRUM: isAjax:true" --header "ADRUM_1: isMobile:true" <http[s]://<endpoint-of-your-application-for-which-you-do-not-see-correlation>
The ADRUM_1: isMobile:true header is used to identify that the request is coming from a mobile app, whereas ADRUM: isAjax:true means it is for browser correlation (no underscore). But due to a known issue with Apache and Nginx, this header is stripped off due to the underscore in the header name. These are known as broken headers.
Underscore Header Work-around in Apache and Nginx
Apache work-around
Apache is silently dropping off headers (including underscores). This is documented by Apache for Apache 2.4 where they describe the following workaround:
- Add the following in the
httpd.conf:
SetEnvIfNoCase ^ADRUM.1$ ^(.*)$ fix_adrum_1=$1 RequestHeader set ADRUM-1 %{fix_adrum_1}e env=fix_adrum_1
- Restart your apache server. You will see that these are no more dropped headers.
Nginx work-around
NGINX drops HTTP headers with underscores silently. To correct this issue, change the nginx.conf file to add the following:
underscores_in_headers on;
Once you complete these workarounds, headers with underscores won’t be dropped by the webserver.
See Nginx’ configuration pitfalls documentation.
Resources
Overview of new features in Apache HTTP Server 2.4
Missing (disappearing) HTTP Headers
