Problem:
You recently upgraded your environment from 4.3/4.2 to 4.4 and started seeing the following error in the EUM server logs:
05 Dec 2017 08:30:50.788 -0600 Analytics Event Dispatcher Processors-19 AD.AnalyticsPublisherHelp ERROR Error publishing analytics events for [BatchId{accountName='SampleAccount-a0Q3400000CATDkEAP', key='14a124...', eumEventType='MobileSnapshot', operation='INSERT', idPath='', mergeFields='[]'}], REST exception, Error code[401], Exception [RestException(statusCode=401, code=NotAuthorized.Account, errorMessage=HTTP 401 Unauthorized, developerMessage=null, logCorrelationId=null)], Dev Message
In Events service log it says:
[2017-12-05T08:41:26,656-06:00] [ERROR] [dw-12722 - PATCH /v3/events/SessionRecord/event?_idPath=sessionguid&_mergeFields=browserRecords%2Cuserdata%2CuserdataLong%2CuserdataBoolean%2CuserdataDouble%2CuserdataDate%2Cmetrics_internal] [c.a.a.p.a.ElasticsearchAccountManager] Found [2] account configurations for account name [sampleaccount-a0q3400000catdkeap]. Returning one of those.
Diagnosis:
Status code 401 is for unauthorized, so you verified the keys across controller, EUM and ES properties and found they are correctly set as:
Controller Admin > Controller Settings > appdynamics.eum.es.key = EUM Server > eum.properties > analytics.accountAccessKey = Event Service > events-serivice-api-store.properies > ad.accountmanager.key.eum
However, you still see the 401 status code.
Then, you run below command to see the appdynamics_accounts that got registered on the events service:
curl -v http://<eventsServiceHost>:9200/appdynamics_accounts/_search?pretty=true
You may find that there are 2 Elastic indices for appdynamics_accounts (one called appdynamics_accounts and the other one appdynamics_accounts_v2). The v2 is the newer one and should be the one used. This must have gotten created during the migration.
When EUM starts up and tries to register various event types that pertain to its functionality (such as browserrecord, mobilesessionrecord, etc), it makes an API call to the Events Service to make this registration.
Unfortunately, the Events Service currently uses search indices that match both appdynamics_accounts as well as appdynamics_accounts_v2. Therefore when EUM tries to register/update say the browserrecord event type, Events Service throws a warning (in its logs) and fails that step—essentially saying "I found multiple accounts in which to do this task and I'm not sure what to do."
Because of this failure, EUM never really registers itself with the Events Service and therefore all the Publish events are failing.
Solution:
The solution to this is fairly simple. Just manually get rid of the Appdynamics_Accounts Index thru the following command that can be run on one of the ES nodes
curl -XDELETE http://localhost:9200/appdynamics_accounts
After this step, I'd wait a minute and then restart the EUM Processor.
