Symptoms
After upgrading the EUM Server from version 4.0.x or 4.1.x to 4.2+, the EUM Server will not start.
The following errors will be displayed when attempting to start the EUM Server:
08 Jun 2017 00:19:24.472 +0200 main AD.SqlAccountSettingStore ERROR
| Exception caught while getting account analytics info:
| SQL: select * from account_settings
|
| com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'eum_db.account_settings' doesn't exist
| at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
| at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
| at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
| at java.lang.reflect.Constructor.newInstance(Unknown Source)
| at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
| at com.mysql.jdbc.Util.getInstance(Util.java:387)
| at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:941)
| at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3870)
| at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3806)
| at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2470)
| at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2617)
| at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2550)
| at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1861)
| at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1962)
| at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:76)
| at com.appdynamics.eumcloud.data.sql.SqlQuery.execute(SqlQuery.java:39)
| at com.appdynamics.eumcloud.data.sql.SqlAccountSettingStore.getAccountSettings(SqlAccountSettingStore.java:64)
| at com.appdynamics.eumcloud.data.cache.AccountSettingsCache.<init>(AccountSettingsCache.java:30)
| at com.appdynamics.eumcloud.data.cache.AccountSettingsCache$$FastClassByGuice$$c7e60f52.newInstance(<generated>)
| at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
| at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
| at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
| at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
| at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
| at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
| at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
| at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
| at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
| at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
| at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
| at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
| at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
| at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
| at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
| at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
| at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
| at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
| at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
| at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
| at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
| at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
| at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
| at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
| at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
| at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
| at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
| at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
| at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
| at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
| at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
| at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
| at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
| at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
| at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
| at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
| at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
| at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
| at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
| at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
| at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
| at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
| at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
| at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
| at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
| at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
| at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
| at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
| at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
| at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
| at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:104)
| at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
| at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
| at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
| at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
| at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
| at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
| at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
| at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
| at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
| at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.startServer(EUMProcessorServerApplication.java:244)
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.run(EUMProcessorServerApplication.java:178)
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.run(EUMProcessorServerApplication.java:114)
| at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:42)
| at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:76)
| at io.dropwizard.cli.Cli.run(Cli.java:70)
| at io.dropwizard.Application.run(Application.java:72)
| at com.appdynamics.eumcloud.EUMProcessorServer.main(EUMProcessorServer.java:40)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
| at java.lang.reflect.Method.invoke(Unknown Source)
| at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
| at com.exe4j.runtime.WinLauncher$2.run(Unknown Source)
|
+---------------------------------------------------------------------------------------
08 Jun 2017 00:19:24.472 +0200 main AD.EUMFixedThreadPool INFO Created [Analytics Event Dispatcher Processors] Thread Pool with [20] Threads
08 Jun 2017 00:19:24.503 +0200 main AD.EUMFixedThreadPool INFO Created [Analytics Event Dispatcher Processors] Thread Pool with [3] Threads
08 Jun 2017 00:19:24.503 +0200 main AD.ADAppAgentModule INFO JVM not instrumented with AD Agent; reporting disabled
08 Jun 2017 00:19:24.535 +0200 main AD.ALL ERROR
| Error Starting EUM Processor Server
| java.lang.RuntimeException: Failed to verify table [mobile_infopoints]: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'eum_db.mobile_infopoints' doesn't exist
| Failed to verify table [mobile_packages]: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'eum_db.mobile_packages' doesn't exist
|
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.startServer(EUMProcessorServerApplication.java:260)
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.run(EUMProcessorServerApplication.java:178)
| at com.appdynamics.eum.processor.EUMProcessorServerApplication.run(EUMProcessorServerApplication.java:114)
| at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:42)
| at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:76)
| at io.dropwizard.cli.Cli.run(Cli.java:70)
| at io.dropwizard.Application.run(Application.java:72)
| at com.appdynamics.eumcloud.EUMProcessorServer.main(EUMProcessorServer.java:40)
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
| at java.lang.reflect.Method.invoke(Unknown Source)
| at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
| at com.exe4j.runtime.WinLauncher$2.run(Unknown Source)
|
+---------------------------------------------------------------------------------------
Diagnosis
The issue arises when the EUM installer fails to add tables to the EUM database during the upgrade process.
Related Links:
Solution
Manually add the missing tables to the EUM database using the 407to410.sql file which is located in /opt/AppDynamics/EUM/eum-processor/bin/ or [EUM_HOME]/eum-processor/bin/ after running the downloaded 4.2+ EUM installation script.
Then validate the changes using the eum-schema command.
Assumptions:
- Controller and EUM Server are hosted on a Linux OS.
- Controller home directory is
/opt/AppDynamics/Controller/.
- EUM Server home directory is
/opt/AppDynamics/EUM/eum-processor/.
- Controller IP address is 192.168.102.99.
1. In the command line, log in to EUM Server to confirm the EUM processor is not running.
cd /opt/AppDynamics/EUM/eum-processor/
bin/eum.sh stop
2. Log in to Controller host.
3. Navigate to the Controller's database directory to access the MySQL client binary.
cd /opt/AppDynamics/Controller/db/bin
4. Execute the following query to update eum_db with the missing tables.
Note: Update the Controller's installation path for the EUM Server and hostname/IP address for the eum_db.
./mysql -u root -p -P 3388 -h 192.168.102.99 -D eum_db < /opt/AppDynamics/EUM/eum-processor/bin/407to410.sql
5. On the Controller, re-run the EUM installation and select option 2. This will complete the remaining eum_db updates.
6. Once the EUM installer completes on Controller, navigate to the bin directory of the Controller's installation of the EUM Server.
cd /opt/AppDynamics/EUM/eum-processor/bin
7. Create a backup of the liquibase.properties file, then update and save the original file in order to use the current EUM user's database password.
cp liquibase.properties liquibase.properties.bak
8. Validate the changes to the EUM database using the eum-schema command.
./eum-schema updateSQL
9. The outputs after running the eum-schema command should generate similar output to the following:
-- *********************************************************************
-- Update Database Script
-- *********************************************************************
-- Change Log: liquibaseChangelog.xml
-- Ran at: 9/2/16 12:49 PM
-- Against: eum_user@localhost@jdbc:mysql://localhost:3388/eum_db?createDatabaseIfNotExist=true
-- Liquibase version: 3.2.0
-- *********************************************************************
-- Lock Database
-- Release Database Lock
10. On success, return to the EUM Server host and start the EUM Server.
