AppDynamics Dashboard shows incorrect JVM restart time

[Anand.Gulla]

Hi Everyone,

We have Ping Directory application (LDAP) running on the Linux server. We have java appAgent and machine agent installed and configured on the Linux server.  The issue is that everyday 1.05 AM my appAgent restarting and that JVM restart time is showing on the AppDynamic dashboard instead of the Ping Directory JVM status.  I have spoken to internal AppD team and raised a support ticket but not satisfied with their responses. I understand their argument that javaagent will not start standalone as it is part of Ping Directory JVM.  I'm trying to chase the mystery what is causing the javaagent restart at exactly 1.05 AM everyday. Appreciate any help in troubleshooting this issue. 

Thanks,

Anand Gulla 

[Arun.Dasetty]

Hi @Anand.Gulla ,

I see a disconnect, have you checked the support update on the root cause of the April 29 update ticket? I manage support in India and I confirm from supoort ticket#316533? Why have not the user come back in ticket? The root cause with evidence is already shown but i am listing here as you are reaching in community, I suggest you reach any application dev team, You can see there are two JVM PID (javaagent is a thread (premain program) part of app JVM PID and it will not create JVM PID). I have "NOT" shared theory but shared evidence it is the customer starting second JVM PID here.

If you see in same host "dld201013230254" at 1:05 AM two app JVM PID started which is using same -Dappdynamics.agent.nodeName= value and you can see the evidence of difference in application runtime args difference, this args are from app not javaagent. (example app arg -Dcom.unboundid.directory.server.scriptName).

Next steps: I have offered web call in tickets/316533 see if you can join we can discuss over your concerns, Also, I will leave it to others to also comment in the community to comment on my analysis.

• First JVM PID with below JVM startup args stated at 28 Apr 01:05:35 seconds at host "dld201013230254.obm.nix.srv.westpac.com.au"
   

AD Agent init] 28 Apr. 2022 01:05:35,054  INFO AgentKernel - JVM Runtime Name: 57279@dld201013230254.obm.nix.srv.westpac.com.au

[AD Agent init] 28 Apr. 2022 01:05:35,045  INFO AgentKernel - JVM Args : -Xmx256m | -Xms128m | -javaagent:/opt/appdynamics/appagent/java/javaagent.jar | -Dappdynamics.agent.runtime.dir=/app/pingdirectory/racdirectory/logs/ | -Dappdynamics.agent.tierName=PingDirectory | -Dappdynamics.force.default.ssl.certificate.validation=true | --add-opens=java.base/java.io=ALL-UNNAMED | --add-opens=java.base/java.lang=ALL-UNNAMED | --add-opens=java.base/java.lang.invoke=ALL-UNNAMED | --add-opens=java.base/java.lang.reflect=ALL-UNNAMED | --add-opens=java.base/java.security=ALL-UNNAMED | --add-opens=java.base/java.text=ALL-UNNAMED | --add-opens=java.base/java.util=ALL-UNNAMED | --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED | --add-opens=java.desktop/java.awt.font=ALL-UNNAMED | -Dcom.unboundid.directory.server.scriptName=export-ldif | 


[AD Agent init] 28 Apr. 2022 01:05:39,642  INFO ConfigurationChannel - Auto agent registration attempted: Application Name [PING-A00B2B-DEV] Component Name [PingDirectory] Node Name [JavaTier_dld201013230254.obm.nix.srv.westpac.com.au]
[AD Agent init] 28 Apr. 2022 01:05:39,642  INFO ConfigurationChannel - Registration information received Node ID[19447446] Component ID[226630] Application ID [61972]

• Second JVM PID with just 10 seconds gap:

[AD Agent init] 28 Apr. 2022 01:05:45,845  INFO AgentKernel - JVM Runtime Name: 59819@dld201013230254.obm.nix.srv.westpac.com.au

[AD Agent init] 28 Apr. 2022 01:05:45,827  INFO AgentKernel - JVM Args : -Xmx2g | -Xms2g | -XX:+UseParallelGC | -XX:NewRatio=8 | -XX:+HeapDumpOnOutOfMemoryError | -javaagent:/opt/appdynamics/appagent/java/javaagent.jar | -Dappdynamics.agent.runtime.dir=/app/pingdirectory/racdirectory/logs/ | -Dappdynamics.agent.tierName=PingDirectory | -Dappdynamics.force.default.ssl.certificate.validation=true | --add-opens=java.base/java.io=ALL-UNNAMED | --add-opens=java.base/java.lang=ALL-UNNAMED | --add-opens=java.base/java.lang.invoke=ALL-UNNAMED | --add-opens=java.base/java.lang.reflect=ALL-UNNAMED | --add-opens=java.base/java.security=ALL-UNNAMED | --add-opens=java.base/java.text=ALL-UNNAMED | --add-opens=java.base/java.util=ALL-UNNAMED | --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED | --add-opens=java.desktop/java.awt.font=ALL-UNNAMED | -Xlog:gc*:file=/app/pingdirectory/racdirectory/current/logs/jvm/gc.log:time,uptime,level,tags:filecount=3,filesize=100m | -Dcom.unboundid.directory.server.scriptName=export-ldif | 

[AD Agent init] 28 Apr. 2022 01:05:50,251  INFO ConfigurationChannel - Auto agent registration attempted: Application Name [PING-A00B2B-DEV] Component Name [PingDirectory] Node Name [JavaTier_dld201013230254.obm.nix.srv.westpac.com.au]
[AD Agent init] 28 Apr. 2022 01:05:50,251  INFO ConfigurationChannel - Registration information received Node ID[19447446] Component ID[226630] Application ID [61972]

• If you see later JVM has additional args:

-Xlog:gc*:file and XX:NewRatio

Conclusion: The agent getting same node id because we are using manual node as per recent logs shared still. It is clear from OS and application perspective with above logs and PS output user does initiate two PIDs with same agent config parameters (-javaagent and -Dappdynamics) args with 10 seconds period gap and hence and agent shows "JVM startup time" discrepancy as same node name used for two different PIDs initiated by OS which is "Invalid config" as per doc specs.

[AD Agent init] 28 Apr. 2022 01:05:45,299  INFO XMLConfigManager - Full Agent Registration Info Resolver using node name [JavaTier_dld201013230254.obm.nix.srv.westpac.com.au]
 
Optional suggestion: As the issue is reproduced and confirms to be user initiating pingdir+  two PIDs 57279 and 59819(PS output is the evidence) at same minute the only suggestion we have is to not use manual node name, if your team cannot have dedicated JVM startup args two both JVM process and use reuse node name arg approach that is below:
 
Step-1: Remove node-name value in agent controller-info.xml at path below and save xml:
/opt/appdynamics/appagent/java/conf/controller-info.xml
/opt/appdynamics/appagent/java/ver21.11.2.33305/conf/controller-info.xml

Step-2: Now in same startup file at host "dld201013230254.obm.nix.srv.westpac.com.au]" where you are passing arg -javaagent:/opt/appdynamics/appagent/java/javaagent.jar pass additionally below and that should get node name different for both the JVMs getting started with same node name earlier.
 

-Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.agent.reuse.nodeName.prefix=CloudActivator_  -Dappdynamics.cron.vm=true 

Regards,

Arun
Support Manager

[Anand.Gulla]

Hi Arun,

I thought of posting here in community to see if anyone have seen the behaviour with the java appagent specific to Ping Directory or any other LDAP components. I'm not convinced with the java appagent logs because I showed you that the Ping Directory process being running for more than a week but not sure what initiating the java appagent to restart its process.

Anyway I updated the configuration files and Ping Directory process file based on your recommendations. I will catchup with you tomorrow at 3 PM AEST.

Thanks,

Anand Gulla

[Arun.Dasetty]

Hi Anand,

Javaagent does not have a timer to start exactly at 1:05 AM, neither javaagent will start PID, it is the app PID. I have given you evidence with the difference in JVM PID args example below arg is one of the diff arg from app PID and still we are claiming it as javaagent PID. Let's discuss this on web call tomorrow. I am sure if you try to reuse the node name approach after removing nodeName in agent or -D arg, this issue proves otherwise. So I request to try suggestions (reuse node name option) before web call or preferably before next 1:05 AM period today on target host, so call itself will not be needed.

• If you see later JVM has additional args:

-Xlog:gc*:file and XX:NewRatio

Just to inform, no other customer ever have faced an issue at exactly 1:05 AM IST and also the PingDirectory app team confirmed too they do not face any issues with any other environment with javaagent, on top of that I shared JVM PID getting started from the app with the different app -D args (i am not referring to agent args here) with 10 secs gap at 1:05 am, all this proves to how PingDirectory app processes start.

We want to address your concerns, so let us continue discussion in help ticket or in call tomorrow as i requested please try suggestions it will clear all your doubts.

Regards,

Arun