Controller Shutdown Fails Due to Running Job - Controller Update ES URL

[ed.barberis]

Experts,

I am experiencing an issue stopping the AppDynamics Controller 4.4.2 after a successful CLI installation using 'platform-admin.sh'. During an automated install of the AppDynamics Enterprise Console, Controller, and the Events Service (version 4.4.2), the script fails with the following error:

Controller failed to stop.

Command failed due to an error: Error running job stop: Another controller operation is in progress. Only one operation can be run on platform 1. To view the running jobs, please log in to Enterprise Console web interface.

It seems the Events Service install submits a long-running job named, 'Controller Update ES URL Job', that is still running in the background when I try to gracefully shutdown the Controller.

Question: Using the CLI, is there a way to determine if there are running jobs?

Here is a script fragment:

# create an appdynamics platform.

cd /opt/appdynamics/platform/platform-admin/bin

./platform-admin.sh create-platform --name "Spring Demo" --description "This is a demo of a simple Spring Boot application." --installation-dir /opt/appdynamics/platform/product

# add local host ('platformadmin') to platform.

./platform-admin.sh add-hosts --hosts "platformadmin"

# install appdynamics controller.

./platform-admin.sh submit-job --service controller --job install --args controllerPrimaryHost="platformadmin" controllerAdminUsername="admin" controllerAdminPassword="welcome1" controllerRootUserPassword="welcome1" mysqlRootPassword="welcome1"

# install appdynamics events service.

./platform-admin.sh install-events-service --profile "dev" --hosts "platformadmin"

# install appdynamics events service.

# shutdown the appdynamics platform components.

./platform-admin.sh stop-controller-appserver --with-db

Stopping the Controller.

Controller failed to stop.

Command failed due to an error: Error running job stop: Another controller operation is in progress. Only one operation can be run on platform 1. To view the running jobs, please log in to Enterprise Console web interface.

Thanks.

[Yogesh.Chouk]

Hi,

It looks like you have started two jobs simultaneously. Run one job at a time. Can you please use the below command to check for the running jobs and let us know if that helps.

./platform-admin.sh list-jobs --service <controller or events-service>

For more information,

https://docs.appdynamics.com/display/PRO44/Enterprise+Console+Command+Line

Thanks,

Yogesh

[ed.barberis]

Yogesh,

After a successfull install of the Enterprise Console and the Controller, I invoke the Events Service install:

$ ./platform-admin.sh install-events-service --profile dev --hosts platformadmin

$ ./platform-admin.sh show-events-service-health

This takes about 30 seconds to complete and then the script exits. The "Show Events Service Health" script also completes successfully. The result is shown in the following "Jobs" dashboard screenshot:

Notice the "Controller Update ES URL Job" that was created and is currently "In Progress". Running your suggested commands only shows the available Actions, not the currently running Jobs:

apm[root]# ./platform-admin.sh list-jobs --service controller

Actions available for service:

Add Secondary Controller                 ("add_secondary", enabled)

Start Controller                         ("start", enabled)

Stop Controller                          ("stop", enabled)

Upgrade Controller                       ("upgrade", enabled)

Upgrade MySQL                            ("mysql_upgrade", enabled)

Remove Controller                        ("remove", enabled)

Retrieve Log                             ("retrieve_log", enabled)

Start MySQL                              ("mysql_start", enabled)

Stop MySQL                               ("mysql_stop", enabled)

Diagnosis                                ("diagnosis", enabled)

Actions available for "Controller" node on host "platformadmin":

None

Actions available for "MySQL" node on host "platformadmin":

None

apm[root]# ./platform-admin.sh list-jobs --service events-service

Actions available for service:

Start Events Service                     ("start", enabled)

Stop Events Service                      ("stop", enabled)

Uninstall                                ("uninstall", enabled)

Upgrade Events Service                   ("upgrade", enabled)

Restart                                  ("restart_cluster", enabled)

Add Node                                 ("add_nodes", enabled)

Retrieve Log                             ("retrieve_logs", enabled)

Scale Up Events Service                  ("install", enabled)

Actions available for "api-store" node on host "platformadmin":

Uninstall Node                           ("uninstall_node", enabled)

Restart Node                             ("restart_node", enabled)

Since this is an automated provisioning script, I would like to know how can I query the Platform to know when the "Controller Update ES URL Job" is finished so I can safely shutdown the Platform? Is there a private SQL statement I could create to give me the same information contained in the "Jobs" dashboard above? Also, what is the purpose of the ES URL job?

Regards,

-Ed

[ed.barberis · Accepted solution]

UPDATE:

I have found an acceptable solution. It seems that if I reverse the order in the installation of the platform services--installing the events service first and then followed by the controller--everything works as expected.

# install appdynamics events service (first).

./platform-admin.sh install-events-service --profile "dev" --hosts "platformadmin"

# install appdynamics controller (second).

./platform-admin.sh submit-job --service controller --job install --args controllerPrimaryHost="platformadmin" controllerAdminUsername="admin" controllerAdminPassword="welcome1" controllerRootUserPassword="welcome1" mysqlRootPassword="welcome1"

Regards,

-Ed