AppDyanmics CUPS service on PCF - jvm startup - controller communication failure

[jayaram.sairam]

hello -

environment - Pivotal Cloud Foundry  1.10

buildpack - JAVA buildpack version 3.17 and 4.3

CUPS service - defined all parameters per documentation

I have had my application startup working and agent communicate to controller in PCF 1.6.

In another environment ( upgrade) to PCF 1.10, im noticing a agent communication to controller fail.

The issue seems to be the IP address of the appdynamics controller being defaulted to container's ip address or some standard Cloud Foundry IPs.

Im able to see the parameters i defined in the CUPS service in the agent log, the parameter values are passed as a java options.

[jayaram.sairam]

Error

[AD Thread Pool-Global1] 03 Aug 2017 21:11:53,138 INFO ConfigurationChannel - Sending Registration request with: Application Name [MyApp], Tier Name [Services], Node Name [MyNode], Host Name [349fcb06-a8d8-4f63-7dad-bf489ca94db3] Node Unique Local ID [xxxxx], Version [Server Agent v4.2.1.8 GA #11326 rb3b8138e547e0d2dd1cee3f6b3ef9ab6e14fdf33 113-4.2.1.next-build]
[AD Thread Pool-Global1] 03 Aug 2017 21:11:53,335 ERROR ConfigurationChannel - Fatal transport error while connecting to URL [/controller/instance/0/applicationConfiguration]: java.net.SocketException: Connection reset

[jayaram.sairam]

Few lines before i see IP address that belongs to CF environment itself and not the controller IP or hostname i had provided.

INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:14, Name:appdynamics.ip.addresses, Value:10.254.0.10,Name:supportsDevMode, Value:true]

[Peter.Holditch]

Can you attach all the agenty logs from the PCF instance where the agent is running?  (log in using the ssh command to see them)

From the extract you provide, it looks like a straightforward network issue, but that is concluded from very limited context.

[jayaram.sairam]

attached is the log file. For security purposes I have edited the controller ip to X.Y.n.n .

with PCF 1.10, "willitconnect" is a built in app that provides a website to check for the connectivity. From that website, I could see a good response to the Controller host (the ip I had provided in CUPS).

[jayaram.sairam]

better log file here

[jayaram.sairam]

I'm noticing the java buildpack is downloading the corresponding agents from

Downloading AppDynamics Agent 4.3.3_6 from https://packages.appdynamics.com/java/4.3.3.6/AppServerAgent-4.3.3.6.zip (found in cache)
Expanding AppDynamics Agent to .java-buildpack/app_dynamics_agent (0.4s)

Is there a github page I could see the changes for AppServerAgent 4.3.3.6 ? 

If not, please let me know how to enable debug mode for AppDynamcis agent in PCF container

[Peter.Holditch]

Your log shows that the agent cannot connect to the controller:

Thread Pool-Global1] 03 Aug 2017 21:11:53,138  INFO ConfigurationChannel - Detected node meta info: [Name:ProcessID, Value:14, Name:appdynamics.ip.addresses, Value:10.254.0.10, Name:supportsDevMode, Value:true]
 Thread Pool-Global1] 03 Aug 2017 21:11:53,138  INFO ConfigurationChannel - Sending Registration request with: Application Name [App], Tier Name [Services], Node Name
panp], Host Name [349fcb06-a8d8-4f63-7dad-bf489ca94db3] Node Unique Local ID [tpanp], Version [Server Agent v4.2.1.8 GA #11326 rb3b8138e547e0d2dd1cee3f6b3ef9ab6e14fdf33
-4.2.1.next-build]
 Thread Pool-Global1] 03 Aug 2017 21:11:53,335 ERROR ConfigurationChannel - Fatal transport error while connecting to URL [/controller/instance/0/applicationConfiguratio
 java.net.SocketException: Connection reset
 Thread Pool-Global1] 03 Aug 2017 21:11:53,335  WARN ConfigurationChannel - Could not connect to the controller/invalid response from controller, cannot get initializati
information, controller host [X.Y.21.72], port[443], exception [Fatal transport error while connecting to URL [/controller/instance/0/applicationConfiguration]]
 Thread Pool-Global1] 03 Aug 2017 21:11:53,339  WARN AgentErrorProcessor - Agent error occurred, [name,transformId]=[com.singularity.CONFIG.ConfigurationChannel - java.n
SocketException,2147483647]
 Thread Pool-Global1] 03 Aug 2017 21:11:53,339  WARN AgentErrorProcessor - 4 instance(s) remaining before error log is silenced
 Thread Pool-Global1] 03 Aug 2017 21:11:53,340 ERROR ConfigurationChannel - Exception: Connection reset
a.net.SocketException: Connection reset

Are you sure your network allows this communication?  Can you curl or telnet to the controller IP and Port when you are SSLed in to the PCF instance?

You can see the agent release notes in the documentation here.