How to resolve "thread pool task queue is full" error

[Erin]

Question:

Our logs are showing the following error message which says that the thread pool task queue is full. We restarted the Controller and the problem went away. Is this normal, and should we increase our thread pool as a result?

[#|2017-01-07T04:52:34.017-0500|SEVERE|glassfish3.1.2|grizzly|_ThreadID=34;_ThreadName=Thread-5;|doSelect exception 
java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096 
at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:493) 
at com.sun.grizzly.util.SyncThreadPool.execute(SyncThreadPool.java:191) 
at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162) 
at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:131) 
at com.sun.grizzly.NIOContext.execute(NIOContext.java:510) 
at com.sun.grizzly.NIOContext.execute(NIOContext.java:488) 
at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:374) 
at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:265)

Answer: 

As of version 4.2.x, for large profiles, our team recommends 24 CPU Cores. This machine only had 8 Cores.

To resolve this, first backup the existing domain.xml configuration file.

Then make the following changes to the domain.xml file:

From:
<thread-pool max-thread-pool-size="32" name="http-thread-pool" min-thread-pool-size="16"></thread-pool>

Change to:
<thread-pool max-thread-pool-size="64" name="http-thread-pool" min-thread-pool-size="64" max-queue-size="-1"></thread-pool>

And restart the App Server:

1. cd <ControllerDir>/bin
2. ./controller.sh stop-appserver
3. ./controller.sh start-appserver   [This will take some time, depending on the size]

[Rahul.Jangid]

HI Team,

Made the name="http-thread-pool" to 16 from 8 but after trying to stop the controller server, it fails. 

I am attaching the sbapshots of domain.xml with the change and then trying to stop the controller server. Can you please suggest what can be the issue?

domain.xml post changeFailed to stop controller

Best Regards,

Saurabh

[Mark.Early]

Since we have 16 processes on this box, I've also increased the thread pool sizes from 8 to 16.  these are the changes: 

Original: 

<thread-pool max-thread-pool-size="8" min-thread-pool-size="8" name="http-thread-pool"/>

New: 
<thread-pool max-thread-pool-size="16" min-thread-pool-size="16" name="http-thread-pool" max-queue-size="-1"/>

Testing now. 

[Claudia.Landivar]

Hi, @Mark.Early 

How did the thread pool size increase work out for you?

Claudia Landivar

Community Manager & Editor