Knowledge Base

cancel
Showing results for 
Search instead for 
Did you mean: 

Why can't EUM authenticate the Events Service after upgrade to 4.4: statusCode=401?

Problem:

You recently upgraded your environment from 4.3/4.2 to 4.4 and started seeing the following error in the EUM server logs:

 

05 Dec 2017 08:30:50.788 -0600 Analytics Event Dispatcher Processors-19 AD.AnalyticsPublisherHelp ERROR Error publishing analytics events for [BatchId{accountName='SampleAccount-a0Q3400000CATDkEAP', key='14a124...', eumEventType='MobileSnapshot', operation='INSERT', idPath='', mergeFields='[]'}], REST exception, Error code[401], Exception [RestException(statusCode=401, code=NotAuthorized.Account, errorMessage=HTTP 401 Unauthorized, developerMessage=null, logCorrelationId=null)], Dev Message

In Events service log it says: 

[2017-12-05T08:41:26,656-06:00] [ERROR] [dw-12722 - PATCH /v3/events/SessionRecord/event?_idPath=sessionguid&_mergeFields=browserRecords%2Cuserdata%2CuserdataLong%2CuserdataBoolean%2CuserdataDouble%2CuserdataDate%2Cmetrics_internal] [c.a.a.p.a.ElasticsearchAccountManager] Found [2] account configurations for account name [sampleaccount-a0q3400000catdkeap]. Returning one of those.

Diagnosis:

Status code 401 is for unauthorized, so you verified the keys across controller, EUM and ES properties and found they are correctly set as:

 

Controller Admin > Controller Settings > appdynamics.eum.es.keyEUM Server > eum.properties > analytics.accountAccessKeyEvent Service > events-serivice-api-store.properies > ad.accountmanager.key.eum

 

However, you still see the 401 status code.

 

Then, you run below command to see the appdynamics_accounts that got registered on the events service:

curl -v http://<eventsServiceHost>:9200/appdynamics_accounts/_search?pretty=true

 

You may find that there are 2 Elastic indices for appdynamics_accounts (one called appdynamics_accounts and the other one appdynamics_accounts_v2). The v2 is the newer one and should be the one used. This must have gotten created during the migration.

When EUM starts up and tries to register various event types that pertain to its functionality (such as browserrecord, mobilesessionrecord, etc), it makes an API call to the Events Service to make this registration.
Unfortunately, the Events Service currently uses search indices that match both appdynamics_accounts as well as appdynamics_accounts_v2. Therefore when EUM tries to register/update say the browserrecord event type, Events Service throws a warning (in its logs) and fails that step—essentially saying "I found multiple accounts in which to do this task and I'm not sure what to do." 

 

Because of this failure, EUM never really registers itself with the Events Service and therefore all the Publish events are failing.

 

Solution: 

The solution to this is fairly simple. Just manually get rid of the Appdynamics_Accounts Index thru the following command that can be run on one of the ES nodes

curl -XDELETE http://localhost:9200/appdynamics_accounts

After this step, I'd wait a minute and then restart the EUM Processor.

Version history
Revision #:
7 of 7
Last update:
2 weeks ago
Updated by:
 
Labels (1)


Found this article helpful? Click the Thumbs Up button.
Have an additional comment? Post it below.
0 Kudos
Comments

Don't forget. You may need to enable the http property (ad.es.node.http.enabled=true) in the "events-service-api-store.properties" file under events-service/processor/conf to allow port 9200 activity. 

How to do it in windows ?

 

OLD: appdynamics_accounts

NEW: appdynamics_accounts_v2

 

How to replace OLD with NEW

 

-------------------------------OLD------------------------------------

 

C:\Users\Administrator\Downloads\curl\bin>curl -v http://localhost:9200/appdynamics_accounts/_search?pretty=true
* Trying ::1:9200...
* TCP_NODELAY set
* Trying 127.0.0.1:9200...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> GET /appdynamics_accounts/_search?pretty=true HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json; charset=UTF-8
< Content-Length: 220
<
{
"error" : {
"root_cause" : [ ],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [ ]
},
"status" : 503
}
* Connection #0 to host localhost left intact

 

------------------------------NEW-------------------------------------

 

C:\Users\Administrator\Downloads\curl\bin>curl -v http://localhost:9200/appdynamics_accounts_v2/_search?pretty=true
* Trying ::1:9200...
* TCP_NODELAY set
* Trying 127.0.0.1:9200...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> GET /appdynamics_accounts_v2/_search?pretty=true HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 5176
<
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : 1.0,
"hits" : [ {
"_index" : "appdynamics_accounts_v2",
"_type" : "appdynamics_accounts",
"_id" : "customer1_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"_score" : 1.0,
"_source" : {
"id" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"accountName" : "customer1_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"accessKey" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"expirationDate" : "2022-10-23T23:59:00.000Z",
"eumAccountName" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-a0q2h00000gqgzhuah",
"licensingConfigurations" : [ {
"eventType" : "browserrecord|sessionrecord",
"dataRetentionPeriodDays" : 90,
"maxDailyBytes" : null,
"maxDailyDocuments" : 2000000,
"expirationDate" : "2022-10-23T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
}, {
"eventType" : "mobilesnapshot|mobilecrashreport|mobilesessionrecord|mobilenonfatalissuerecord",
"dataRetentionPeriodDays" : 90,
"maxDailyBytes" : null,
"maxDailyDocuments" : 1000000,
"expirationDate" : "2022-10-23T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
}, {
"eventType" : "biz_txn_v.*|custom_.*",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : null,
"maxDailyDocuments" : 0,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
}, {
"eventType" : "dbmon-.*",
"dataRetentionPeriodDays" : 10,
"maxDailyBytes" : 9223372036854775807,
"maxDailyDocuments" : 9223372036854775807,
"expirationDate" : "2022-10-23T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
}, {
"eventType" : "log_v.*",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : 0,
"maxDailyDocuments" : null,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
} ]
}
}, {
"_index" : "appdynamics_accounts_v2",
"_type" : "appdynamics_accounts",
"_id" : "system_8e9fa888-fa36-11e9-959f-506b8d3981a8",
"_score" : 1.0,
"_source" : {
"id" : "8e9fa888-fa36-11e9-959f-506b8d3981a8",
"accountName" : "system_8e9fa888-fa36-11e9-959f-506b8d3981a8",
"accessKey" : "changeme",
"expirationDate" : "2038-01-01T23:59:00.000Z",
"eumAccountName" : "",
"licensingConfigurations" : [ {
"eventType" : "browserrecord|sessionrecord",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : null,
"maxDailyDocuments" : 0,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.590Z"
}, {
"eventType" : "mobilesnapshot|mobilecrashreport|mobilesessionrecord|mobilenonfatalissuerecord",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : null,
"maxDailyDocuments" : 0,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.590Z"
}, {
"eventType" : "biz_txn_v.*|custom_.*",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : null,
"maxDailyDocuments" : 0,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.590Z"
}, {
"eventType" : "log_v.*",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : 0,
"maxDailyDocuments" : null,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.590Z"
} ]
}
}, {
"_index" : "appdynamics_accounts_v2",
"_type" : "appdynamics_accounts",
"_id" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-a0q2h00000gqgzhuah",
"_score" : 1.0,
"_source" : {
"id" : null,
"accountName" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-a0q2h00000gqgzhuah",
"accessKey" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"expirationDate" : "2022-10-23T06:59:59.000Z",
"eumAccountName" : null,
"licensingConfigurations" : [ {
"eventType" : "iotrecord",
"dataRetentionPeriodDays" : 8,
"maxDailyBytes" : null,
"maxDailyDocuments" : null,
"expirationDate" : null,
"lastModifiedTime" : "2019-11-01T10:56:21.885Z"
} ]
}
}, {
"_index" : "appdynamics_accounts_v2",
"_type" : "appdynamics_accounts",
"_id" : "euminternal",
"_score" : 1.0,
"_source" : {
"id" : null,
"accountName" : "euminternal",
"accessKey" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"expirationDate" : "4002-01-31T17:45:17.000Z",
"eumAccountName" : null,
"licensingConfigurations" : [ ]
}
} ]
}
}
* Connection #0 to host localhost left intact

-------------------------------------------------------------------