Click the Start a free trial link to start a 15-day SaaS trial of our product and join our community as a trial user. If you are an existing customer do not start a free trial.
AppDynamics customers and established members should click the sign in button to authenticate.
12-06-2021 04:48 PM - edited 12-06-2021 05:47 PM
Hi there,
Today my Controller got stuck with high CPU usage, i restarted it and now for some weird reason no one can login i can only acess the /controller/admin.jsp
Inside the server log i got a lot of entries like this:
[#|2021-12-06T22:39:37.411-0300|INFO|glassfish 4.1|javax.enterprise.system.core.security|_ThreadID=69;_ThreadName=http-listener-1(13);_TimeMillis=1638841177411;_LevelValue=800;_MessageID=NCLS-SECURITY-05046;|Audit: Authentication refused for [singularity-agent@customer1].|#]
[#|2021-12-06T22:39:37.412-0300|INFO|glassfish 4.1|javax.enterprise.system.core.security.com.sun.enterprise.security.jmac.callback|_ThreadID=69;_ThreadName=http-listener-1(13);_TimeMillis=1638841177412;_LevelValue=800;|jmac.loginfail|#]
I even tried to create a new Account but now when i log in everything is blank.. in the controller did i break something?
Is it possible to recover what i "destroyed"
Any help is appreciated.
12-07-2021 06:17 AM - edited 12-07-2021 06:22 AM
Guys the server is "comming back" i did nothing and all of sudden some parts of the configuration are showing, but i don't see nothing of the instrumentation that i did and when i log in i get this error:
Error
500 Internal Server Error
Exception Id:b13eec1b-bc08-463d-b77f-19f881a04c67
Looking inside the server.log i see many SEVERE errors, all of them are:
Unable to decrypt AccountAccess Key.
com.singularity.ee.util.security.credentialstore.exception.CryptoOperationFailureException: Failed to decrypt: Input length must be multiple of 16 when decrypting with padded cipher
Here is one of the many stacks that show the same error only different threads:
com.singularity.ee.controller.entity.AccountEntity|_ThreadID=1010;_ThreadName=config-caches-refresh-54;_TimeMillis=1638886343390;_LevelValue=1000;|Unable to decrypt AccountAccess Key.
com.singularity.ee.util.security.credentialstore.exception.CryptoOperationFailureException: Failed to decrypt: Input length must be multiple of 16 when decrypting with padded cipher
at com.singularity.ee.util.security.credentialstore.KeyStoreBasedEncryptionWrapper.decryptCipherTextString(KeyStoreBasedEncryptionWrapper.java:97)
at com.singularity.ee.util.scs.SCSDecryptionUtilityManager.decryptCipherText(SCSDecryptionUtilityManager.java:86)
at com.singularity.ee.controller.entity.AccountEntity.scsOperationsDoToDTO(AccountEntity.java:716)
at com.singularity.ee.controller.entity.AccountEntity.postLoadLifeCycleEvent(AccountEntity.java:705)
at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.hibernate.ejb.event.BeanCallback.invoke(BeanCallback.java:37)
at org.hibernate.ejb.event.EntityCallbackHandler.callback(EntityCallbackHandler.java:94)
at org.hibernate.ejb.event.EntityCallbackHandler.postLoad(EntityCallbackHandler.java:87)
at org.hibernate.ejb.event.EJB3PostLoadEventListener.onPostLoad(EJB3PostLoadEventListener.java:47)
at org.hibernate.event.def.DefaultLoadEventListener.assembleCacheEntry(DefaultLoadEventListener.java:700)
at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:605)
at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:459)
at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:227)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:147)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:1090)
at org.hibernate.impl.SessionImpl.immediateLoad(SessionImpl.java:1026)
at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:176)
at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:215)
at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:190)
at com.singularity.ee.controller.entity.AccountEntity_$$_jvst12b_108.isMarkedAsDeleted(AccountEntity_$$_jvst12b_108.java)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl.refreshCache(BackendConfigCacheServiceImpl.java:380)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59.CGLIB$refreshCache$1(<generated>)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59$$FastClassByGuice$$8ce8ba83.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
at com.appdynamics.platform.persistence.TransactionInterceptor.lambda$invoke$0(TransactionInterceptor.java:37)
at com.singularity.ee.controller.beans.model.EJBManagerBean.runWithinSupportsTransaction(EJBManagerBean.java:23)
at sun.reflect.GeneratedMethodAccessor116.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1081)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1153)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4786)
at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:656)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:824)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.appdynamics.platform.persistence.EntityManagerInterceptor.call(EntityManagerInterceptor.java:45)
at com.appdynamics.platform.persistence.GlassfishEntityManagerInterceptor.manageEntityManagerLifecycle(GlassfishEntityManagerInterceptor.java:25)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.appdynamics.controller.persistence.ControllerExceptionHandlingInterceptor.call(ControllerExceptionHandlingInterceptor.java:71)
at com.appdynamics.controller.persistence.GlassfishControllerExceptionHandlingInterceptor.handleExceptions(GlassfishControllerExceptionHandlingInterceptor.java:18)
at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:608)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:883)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:823)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:369)
at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4758)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4746)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
at com.sun.proxy.$Proxy340.runWithinSupportsTransaction(Unknown Source)
at com.appdynamics.platform.persistence.GlassfishPersistenceManager.runWithinTransactionAttributeBlock(GlassfishPersistenceManager.java:35)
at com.appdynamics.platform.persistence.TransactionInterceptor.invoke(TransactionInterceptor.java:33)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
at com.appdynamics.platform.apm.service.backend.BackendConfigCacheServiceImpl$$EnhancerByGuice$$2ddbec59.refreshCache(<generated>)
at com.singularity.ee.controller.beans.ControllerManager.lambda$getCacheRefreshForAppTasks$8(ControllerManager.java:2657)
at com.appdynamics.platform.services.threads.ExecutorRegistry$ReportMetricsCallable.call(ExecutorRegistry.java:474)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
at com.sun.crypto.provider.CipherCore.prepareInputBuffer(CipherCore.java:1005)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:848)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)
at javax.crypto.Cipher.doFinal(Cipher.java:2168)
at com.singularity.ee.util.security.credentialstore.impl.EncryptionSchemeLegacy.decrypt(EncryptionSchemeLegacy.java:68)
at com.singularity.ee.util.security.credentialstore.EncryptionCommon.decryptCipherTextString(EncryptionCommon.java:102)
at com.singularity.ee.util.security.credentialstore.KeyStoreBasedEncryptionWrapper.decryptCipherTextString(KeyStoreBasedEncryptionWrapper.java:94)
... 77 more
12-07-2021 09:53 AM - edited 12-07-2021 09:54 AM
Hi @Tadeu.Alves,
This seems like a critical issue, I would contact Support for more immediate help. If you do and support helps you, can you please share what you learned here?
Knowledge sharing is what drives this community forward.
Thanks,
Ryan, AppDynamics Community Manager
User | Count |
---|---|
6 | |
4 | |
3 | |
2 | |
2 | |
1 |
Thank you! Your submission has been received!
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form