Knowledge Base

cancel
Showing results for 
Search instead for 
Did you mean: 

Why can't EUM authenticate the Events Service after upgrade to 4.4: statusCode=401?

How do I troubleshoot Events Service status code 401?

 

Problem

You recently upgraded your environment from 4.3/4.2 to 4.4 and have started seeing the following error in the EUM server logs:

05 Dec 2017 08:30:50.788 -0600 Analytics Event Dispatcher Processors-19 AD.AnalyticsPublisherHelp ERROR Error publishing analytics events for [BatchId{accountName='SampleAccount-a0Q3400000CATDkEAP', key='14a124...', eumEventType='MobileSnapshot', operation='INSERT', idPath='', mergeFields='[]'}], REST exception, Error code[401], Exception [RestException(statusCode=401, code=NotAuthorized.Account, errorMessage=HTTP 401 Unauthorized, developerMessage=null, logCorrelationId=null)], Dev Message

 

The Events service log says: 

[2017-12-05T08:41:26,656-06:00] [ERROR] [dw-12722 - PATCH /v3/events/SessionRecord/event?_idPath=sessionguid&_mergeFields=browserRecords%2Cuserdata%2CuserdataLong%2CuserdataBoolean%2CuserdataDouble%2CuserdataDate%2Cmetrics_internal] [c.a.a.p.a.ElasticsearchAccountManager] Found [2] account configurations for account name [sampleaccount-a0q3400000catdkeap]. Returning one of those.

 

Diagnosis

Status code 401 is for unauthorized access, so you verify the keys across controller, EUM and ES properties and find they are correctly set as:

 

Controller Admin > Controller Settings >appdynamics.es.eum.key = EUM Server > eum.properties > analytics.accountAccessKeyEvent Service > events-serivice-api-store.properties > ad.accountmanager.key.eum

 

However, you still see the 401 status code.

 

Then, you run the command below to see the appdynamics_accounts that were registered on the events service:

curl -v http://<eventsServiceHost>:9200/appdynamics_accounts/_search?pretty=true

 

You find there are 2 Elastic indices for appdynamics_accounts (one called appdynamics_accounts and the other appdynamics_accounts_v2). The v2 index is the newer one, created during the migration, and should be the one used.

When EUM starts up and tries to register various event types that pertain to its functionality (such as browserrecord, mobilesessionrecord, etc), it makes an API call to the Events Service to make this registration.

 

Since the Events Service currently uses search indices that match both appdynamics_accounts as well as appdynamics_accounts_v2, when EUM tries to register/update (say, the browserrecord event type), Events Service throws a warning (in its logs)—essentially saying "I found multiple accounts in which to do this task and I'm not sure what to do," and and fails that step 

 

Because of this failure, EUM never really registers itself with the Events Service, so all the Publish events will fail.

 

Solution

The solution to this is to manually get rid of the unused appdynamics_accounts index thru the following command that can be run on one of the ES nodes:

curl -XDELETE http://localhost:9200/appdynamics_accounts

 

After this step, I'd wait a minute and then restart the EUM Processor.

Version history
Revision #:
11 of 11
Last update:
‎10-02-2020 11:20 AM
Updated by:
 
Labels (1)


Found this article helpful? Click the Thumbs Up button.
Have an additional comment? Post it below.
0 Kudos
Comments

Don't forget. You may need to enable the http property (ad.es.node.http.enabled=true) in the "events-service-api-store.properties" file under events-service/processor/conf to allow port 9200 activity. 

How to do it in windows ?

 

OLD: appdynamics_accounts

NEW: appdynamics_accounts_v2

 

How to replace OLD with NEW

 

-------------------------------OLD------------------------------------

 

C:\Users\Administrator\Downloads\curl\bin>curl -v http://localhost:9200/appdynamics_accounts/_search?pretty=true
* Trying ::1:9200...
* TCP_NODELAY set
* Trying 127.0.0.1:9200...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> GET /appdynamics_accounts/_search?pretty=true HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json; charset=UTF-8
< Content-Length: 220
<
{
"error" : {
"root_cause" : [ ],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query",
"grouped" : true,
"failed_shards" : [ ]
},
"status" : 503
}
* Connection #0 to host localhost left intact

 

------------------------------NEW-------------------------------------

 

C:\Users\Administrator\Downloads\curl\bin>curl -v http://localhost:9200/appdynamics_accounts_v2/_search?pretty=true
* Trying ::1:9200...
* TCP_NODELAY set
* Trying 127.0.0.1:9200...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> GET /appdynamics_accounts_v2/_search?pretty=true HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.66.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Content-Length: 5176
<
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"hits" : {
"total" : 4,
"max_score" : 1.0,
"hits" : [ {
"_index" : "appdynamics_accounts_v2",
"_type" : "appdynamics_accounts",
"_id" : "customer1_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"_score" : 1.0,
"_source" : {
"id" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"accountName" : "customer1_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"accessKey" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"expirationDate" : "2022-10-23T23:59:00.000Z",
"eumAccountName" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-a0q2h00000gqgzhuah",
"licensingConfigurations" : [ {
"eventType" : "browserrecord|sessionrecord",
"dataRetentionPeriodDays" : 90,
"maxDailyBytes" : null,
"maxDailyDocuments" : 2000000,
"expirationDate" : "2022-10-23T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
}, {
"eventType" : "mobilesnapshot|mobilecrashreport|mobilesessionrecord|mobilenonfatalissuerecord",
"dataRetentionPeriodDays" : 90,
"maxDailyBytes" : null,
"maxDailyDocuments" : 1000000,
"expirationDate" : "2022-10-23T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
}, {
"eventType" : "biz_txn_v.*|custom_.*",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : null,
"maxDailyDocuments" : 0,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
}, {
"eventType" : "dbmon-.*",
"dataRetentionPeriodDays" : 10,
"maxDailyBytes" : 9223372036854775807,
"maxDailyDocuments" : 9223372036854775807,
"expirationDate" : "2022-10-23T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
}, {
"eventType" : "log_v.*",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : 0,
"maxDailyDocuments" : null,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.480Z"
} ]
}
}, {
"_index" : "appdynamics_accounts_v2",
"_type" : "appdynamics_accounts",
"_id" : "system_8e9fa888-fa36-11e9-959f-506b8d3981a8",
"_score" : 1.0,
"_source" : {
"id" : "8e9fa888-fa36-11e9-959f-506b8d3981a8",
"accountName" : "system_8e9fa888-fa36-11e9-959f-506b8d3981a8",
"accessKey" : "changeme",
"expirationDate" : "2038-01-01T23:59:00.000Z",
"eumAccountName" : "",
"licensingConfigurations" : [ {
"eventType" : "browserrecord|sessionrecord",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : null,
"maxDailyDocuments" : 0,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.590Z"
}, {
"eventType" : "mobilesnapshot|mobilecrashreport|mobilesessionrecord|mobilenonfatalissuerecord",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : null,
"maxDailyDocuments" : 0,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.590Z"
}, {
"eventType" : "biz_txn_v.*|custom_.*",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : null,
"maxDailyDocuments" : 0,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.590Z"
}, {
"eventType" : "log_v.*",
"dataRetentionPeriodDays" : 0,
"maxDailyBytes" : 0,
"maxDailyDocuments" : null,
"expirationDate" : "2019-11-16T23:59:00.000Z",
"lastModifiedTime" : "2019-11-15T21:03:25.590Z"
} ]
}
}, {
"_index" : "appdynamics_accounts_v2",
"_type" : "appdynamics_accounts",
"_id" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-a0q2h00000gqgzhuah",
"_score" : 1.0,
"_source" : {
"id" : null,
"accountName" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-a0q2h00000gqgzhuah",
"accessKey" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"expirationDate" : "2022-10-23T06:59:59.000Z",
"eumAccountName" : null,
"licensingConfigurations" : [ {
"eventType" : "iotrecord",
"dataRetentionPeriodDays" : 8,
"maxDailyBytes" : null,
"maxDailyDocuments" : null,
"expirationDate" : null,
"lastModifiedTime" : "2019-11-01T10:56:21.885Z"
} ]
}
}, {
"_index" : "appdynamics_accounts_v2",
"_type" : "appdynamics_accounts",
"_id" : "euminternal",
"_score" : 1.0,
"_source" : {
"id" : null,
"accountName" : "euminternal",
"accessKey" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"expirationDate" : "4002-01-31T17:45:17.000Z",
"eumAccountName" : null,
"licensingConfigurations" : [ ]
}
} ]
}
}
* Connection #0 to host localhost left intact

-------------------------------------------------------------------