Problem
You recently upgraded your environment from 4.3/4.2 to 4.4 and have started seeing the following error in the EUM server logs:
05 Dec 2017 08:30:50.788 -0600 Analytics Event Dispatcher Processors-19 AD.AnalyticsPublisherHelp ERROR Error publishing analytics events for [BatchId{accountName='SampleAccount-a0Q3400000CATDkEAP', key='14a124...', eumEventType='MobileSnapshot', operation='INSERT', idPath='', mergeFields='[]'}], REST exception, Error code[401], Exception [RestException(statusCode=401, code=NotAuthorized.Account, errorMessage=HTTP 401 Unauthorized, developerMessage=null, logCorrelationId=null)], Dev Message
The Events service log says:
[2017-12-05T08:41:26,656-06:00] [ERROR] [dw-12722 - PATCH /v3/events/SessionRecord/event?_idPath=sessionguid&_mergeFields=browserRecords%2Cuserdata%2CuserdataLong%2CuserdataBoolean%2CuserdataDouble%2CuserdataDate%2Cmetrics_internal] [c.a.a.p.a.ElasticsearchAccountManager] Found [2] account configurations for account name [sampleaccount-a0q3400000catdkeap]. Returning one of those.
Diagnosis
Status code 401 is for unauthorized access, so you verify the keys across controller, EUM and ES properties and find they are correctly set as:
Controller Admin > Controller Settings >appdynamics.es.eum.key = EUM Server > eum.properties > analytics.accountAccessKey = Event Service > events-serivice-api-store.properties > ad.accountmanager.key.eum
However, you still see the 401 status code.
Then, you run the command below to see the appdynamics_accounts that were registered on the events service:
curl -v http://<eventsServiceHost>:9200/appdynamics_accounts/_search?pretty=true
You find there are 2 Elastic indices for appdynamics_accounts (one called appdynamics_accounts and the other appdynamics_accounts_v2). The v2 index is the newer one, created during the migration, and should be the one used.
When EUM starts up and tries to register various event types that pertain to its functionality (such as browserrecord, mobilesessionrecord, etc), it makes an API call to the Events Service to make this registration.
Since the Events Service currently uses search indices that match both appdynamics_accounts as well as appdynamics_accounts_v2, when EUM tries to register/update (say, the browserrecord event type), Events Service throws a warning (in its logs)—essentially saying "I found multiple accounts in which to do this task and I'm not sure what to do," and and fails that step
Because of this failure, EUM never really registers itself with the Events Service, so all the Publish events will fail.
Solution
The solution to this is to manually get rid of the unused appdynamics_accounts index thru the following command that can be run on one of the ES nodes:
curl -XDELETE http://localhost:9200/appdynamics_accounts
After this step, I'd wait a minute and then restart the EUM Processor.
