Knowledge Base

cancel
Showing results for 
Search instead for 
Did you mean: 

Why can't EUM authenticate the Events Service after upgrade to 4.4: statusCode=401?

Problem:

You recently upgraded your environment from 4.3/4.2 to 4.4 and started seeing the following error in the EUM server logs:

 

05 Dec 2017 08:30:50.788 -0600 Analytics Event Dispatcher Processors-19 AD.AnalyticsPublisherHelp ERROR Error publishing analytics events for [BatchId{accountName='SampleAccount-a0Q3400000CATDkEAP', key='14a124...', eumEventType='MobileSnapshot', operation='INSERT', idPath='', mergeFields='[]'}], REST exception, Error code[401], Exception [RestException(statusCode=401, code=NotAuthorized.Account, errorMessage=HTTP 401 Unauthorized, developerMessage=null, logCorrelationId=null)], Dev Message

In Events service log it says: 

[2017-12-05T08:41:26,656-06:00] [ERROR] [dw-12722 - PATCH /v3/events/SessionRecord/event?_idPath=sessionguid&_mergeFields=browserRecords%2Cuserdata%2CuserdataLong%2CuserdataBoolean%2CuserdataDouble%2CuserdataDate%2Cmetrics_internal] [c.a.a.p.a.ElasticsearchAccountManager] Found [2] account configurations for account name [sampleaccount-a0q3400000catdkeap]. Returning one of those.

Diagnosis:

Status code 401 is for unauthorized, so you verified the keys across controller, EUM and ES properties and found they are correctly set as-

 

Controller Admin > Controller Settings > appdynamics.eum.es.keyEUM Server > eum.properties > analytics.accountAccessKeyEvent Service > events-serivice-api-store.properies > ad.accountmanager.key.eum

 

However, you still see the 401 status code.

 

Then, you run below command to see the appdynamics_accounts that got registered on the events service:

curl -v http://<eventsServiceHost>:9200/appdynamics_accounts/_search?pretty=true

You may find that there are 2 Elastic indices for appdynamics_accounts (one called appdynamics_accounts and the other one appdynamics_accounts_v2). The v2 is the newer one and should be the one used. This must have gotten created during the migration.

When EUM starts up and tries to register various event types that pertain to its functionality (such as browserrecord, mobilesessionrecord etc), it makes an API call to the Events Service to make this registration.
Unfortunately, the Events Service currently uses search indices that match both appdynamics_accounts as well as appdynamics_accounts_v2. Therefore when EUM tries to register/update say the browserrecord event type, Events Service throws a warning (in its logs) and fails that step. Essentially saying that I found multiple accounts in which to do this task and I'm not sure what to do. 
Because of this failure, EUM never really registers itself with the Events Service and therefore all the Publish events are failing.

 

Solution: 

The solution to this is fairly simple. Just manually get rid of the Appdynamics_Accounts Index thru the following command that can be run on one of the ES nodes

curl -XDELETE http://localhost:9200/appdynamics_accounts

After this step, I'd wait a minute and then restart EUM Processor.

 __________________________________________Thumbs Up Icon Kudos Screen Shot 2017-12-11 at 12.22.35 PM.pngWas this helpful? Give it kudos [click on the thumbs up icon at the end of the article, bottom left corner] to let others know too!

 Do you have a question or need clarification? [Post Your Comment] Below. 

 

Version history
Revision #:
6 of 6
Last update:
‎10-12-2018 02:20 PM
Updated by:
 
Labels (1)
0 Kudos
Comments

Don't forget. You may need to enable the http property (ad.es.node.http.enabled=true) in the "events-service-api-store.properties" file under events-service/processor/conf to allow port 9200 activity.