AppDynamics DB Agent v22.6.0.2803 not showing Load Metrics

[Dietrich.Meier]

Intro
The client upgraded their Oracle DB from v12.1.0.2 to v19.15.0.0.0.
The client DBAs were experiencing issues with load and caching on their DB and found queries run by the service account the AppD Agent runs, to be a culprit, so they asked for the AppD DB Agent to be upgraded.

Initial action
Upgraded the prod, on-prem Windows DB agent from v21.2.0.2285 to v22.6.0.2803(latest available at the time).

Issue
2 of the Oracle DBs stopped reporting the DB Load metrics consistently.
The DB agent logs have no Error or Warning logs related to these 2 DB Collectors that can be troubleshot. 
There are several other Oracle DBs on the same version, the same controller, and agents that are not experiencing the issue. 
All other DB metrics are reporting in. 

Further testing
Tested in pre-prod by having the pre-prod DB agent (same new version as prod) and pre-prod controller(SaaS, v22.5.0-662) monitor the 2 problematic prod Oracle DBs. The same issue is present. No Load metrics. (points to an agent version issue)

Monitored pre-prod equivalent Oracle DB (Same oracle version, different DB), but it does not experience the issue. (Issue isolated to specific DBs)

Tested by monitoring the problematic DBs with a completely different DB agent that matches the older version that was updated away from and the issue is not present. Shows again that it's related to the new agent version. 

Conclusion 
The client could not carry on with missing Load metrics for DBs in question so the agent was rolled back to the older version that did not have this issue.

AppD support has 2 theories on this and is still asking for more queries to be run against the prod DBs as they experience the issue, but this only happening to the prod DBs and we cannot recreate it in pre-prod, so it's not a quick thing to do. (Breaking the prod DB monitoring just to wait for the issue and then run queries)
Theory 1: DB agent is not able to query the DB
Theory 2: DB agent is not able to send all metrics to SaaS Controller

The screenshot below shows the Load metrics not reported consistently for a busy prod Oracle DB. 

Shows Load metrics not reporting in

I am hoping someone else might have come across this issue as well and has a possible solution, or more evidence pointing to a possible DB agent version bug. 

*Second time I created this post because my first one just went missing after I submitted it.

[Ryan.Paredez]

Hi @Dietrich.Meier,

Thanks for sharing this, I will also share it with the PM team. It looks like your first attempt at creating this post hit our spam filter. I'm going to mark it not as spam, but then archive it since you have already recreated the same post.

[Ryan.Paredez]

Hi @Dietrich.Meier,

I reached out to the PM and he suggests creating a Support ticket. 
See this TKB article for more info: 

[Dietrich.Meier]

Hi Ryan

We already had a ticket with AppD support like I mentioned in the initial post, but the problem is that it was only happening to 2 prod DBs and we could not carry on with the missing metrics any longer in prod to troubleshoot the issue for more days, weeks.
The pre-prod DBs did not exabit the same problem, making it difficult to troubleshoot. 

At the moment we are carrying on fine with the older agent version.

Regards
Dietrich 

[Ryan.Paredez]

Hi @Dietrich.Meier,

Ah right, you did mention that. My apologies. I've pinged the PM again to see if there is anything else we can do, but I think whatever work you did with Support or trying your best to work with them will lead to the best outcome. I'll report back if I hear anything.

Thanks again for sharing all that info in your initial post.