Hello,
we using since 1 month the controller in version 4.5.0.25224 and the app agent in version 4.5.2.23891 with oracle java 1.8.0 update 141. MachineAgent is version 4.5.1.1385.
We can observe that about 10 hours after application start we receive some stacktraces and the reporting is interrupted. In this time the controller counts less calls than it is in real. After restarting the application all is fine.
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440 WARN AgentErrorProcessor - Agent error occurred, [name,transformId]=[com.singularity.tm.ServletInterceptor - java.util.ConcurrentModificationException,367]
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440 WARN AgentErrorProcessor - 4 instance(s) remaining before error log is silenced
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440 WARN AgentErrorProcessor - 499 instance(s) remaining before instrumentation point is targeted for neutralization
[https-jsse-nio-8543-exec-7] 18 Oct 2018 11:28:17,440 ERROR ServletInterceptor - Error in TEP : onMethodBeginTracked for : MethodExecutionEnvironment{
invokedObject='{ Class='org.springframework.web.servlet.DispatcherServlet', Hash code=1274906611 }',
className='org.springframework.web.servlet.FrameworkServlet',
methodName='doPost',
paramValues=[ { Class='org.apache.catalina.connector.RequestFacade', Hash code=1829916242 },
{ Class='javax.servlet.http.HttpServletResponseWrapper', Hash code=678606228 } ]}, transactionContext local copy: null, transactionContext fetched from btContext unknown
java.util.ConcurrentModificationException
at java.util.ArrayList.sort(ArrayList.java:1456) ~[?:1.8.0_141]
at java.util.Collections.sort(Collections.java:175) ~[?:1.8.0_141]
at com.singularity.ee.util.collections.f.a(f.java:175) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.common.cf.a(cf.java:93) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:424) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:308) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.common.ad.a(ad.java:252) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.http.servlet.k.a(k.java:68) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.a(i.java:1142) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.b(i.java:267) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.common.i.a(i.java:163) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.transactionmonitor.http.servlet.a.a(a.java:129) ~[appagent.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.services.bciengine.a.onMethodBegin(a.java:52) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.kernel.bootimpl.FastMethodInterceptorDelegatorImpl.safeOnMethodBeginNoReentrantCheck(FastMethodInterceptorDelegatorImpl.java:370) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.kernel.bootimpl.FastMethodInterceptorDelegatorImpl.safeOnMethodBegin(FastMethodInterceptorDelegatorImpl.java:295) ~[appagent-boot.jar:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at com.singularity.ee.agent.appagent.entrypoint.bciengine.FastMethodInterceptorDelegatorBoot.safeOnMethodBegin(FastMethodInterceptorDelegatorBoot.java:50) ~[?:Server Agent #4.5.2.23891 v4.5.2 GA compatible with 4.4.1.0 r0bab72821a20dd3f150ac291cf9a606d7b49cc93 master]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java) ~[?:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) ~[servlet-api-2.5-6.1.3.jar:6.1.3]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[?:?]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) ~[servlet-api-2.5-6.1.3.jar:6.1.3]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at de.codecentric.accessshop.interceptor.SlotResponseHeaderFilter.doFilterInternal(SlotResponseHeaderFilter.java:31) ~[?:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105) ~[?:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at de.codecentric.accessshop.interceptor.LoggingServletFilter.doFilter(LoggingServletFilter.java:97) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[?:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:89) ~[?:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77) ~[?:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) ~[?:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:107) ~[?:?]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192) ~[?:?]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165) ~[?:?]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[?:?]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108) ~[?:?]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) ~[?:?]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) ~[?:?]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) ~[?:?]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) ~[?:?]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349) ~[?:?]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784) ~[?:?]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[?:?]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) ~[?:?]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) ~[?:?]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java) [?:1.8.0_141]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[?:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,715 INFO ErrorProcessor - Sending ADDs to register [ApplicationDiagnosticData{key='org.springframework.web.client.HttpClientErrorException:-1798508104', name=org.springframework.web.client.HttpClientErrorException:, diagnosticType=STACK_TRACE, configEntities=[Type:ERROR, id:103710], summary='org.springframework.web.client.HttpClientErrorException:'}]
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724 WARN ErrorProcessor - ADD org.springframework.web.client.HttpClientErrorException:-1798508104 has been flagged for blacklisting. Registration will not be re-attempted until agent reset.
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724 WARN ErrorProcessor - ADD org.springframework.web.client.HttpClientErrorException: failed to register w/ the Controller
[AD Thread Pool-Global189] 18 Oct 2018 11:46:21,724 INFO ErrorProcessor - Stack Trace Objects not registered due to controller limit: [org.springframework.web.client.HttpClientErrorException:-1798508104]
Do you have any idea how we can fix that?
with kind regards,
Raphael
