We have been experiencing an ongoing situation where our application pool randomly fails bringing our site down until we are able to stop and start the application pool bring the site back up. This happens randomly, but usually at least once every 4 or 5 days.
We are running Coldfusion 10 with Update 18 applied and the connector rebuilt after applying update 18. The web server is IIS 7.5 running on Windows Server 2008 R2. We are using the following settings in our workers.properties file:
worker.cfusion.type=ajp13
worker.cfusion.host=localhost
worker.cfusion.port=8012
worker.cfusion.max_reuse_connections=900
worker.cfusion.connection_pool_size=900
worker.cfusion.connection_pool_timeout=60
Our isapi_redirect.dll file shows version 1.2.41.0.
When the site fails, requests appear to just hang. Using Fusion Reactor we see that requests are no longer being passed on to Coldfusion. If we attempt to stop the application pool in IIS it will sometimes shutdown properly and other times throw an error that it is unable to respond to the control request. If the application pool does stop successfully, I'm able to manually start the application pool and everything will be good until the application pool fails again. If the application pool does not respond the attempt to stop it, I will need to stop and start the website or IIS as a whole to recover the sites.
In our application pool settings we have Rapid-Fail Protection disabled. We have Recycling set to be at specific times; 22:00:00 and 06:00:00 respectively.
When the server stops responding will see entries in our httperr file for all requests that end in 2 Client_Reset followed by the application pool name. When I stop the application pool, future requests will end in 503 2 Disabled as the application pool is no longer running. As soon as I start the application pool, the server will run as expected until the issues return. It may be a day or two or up to a week before it fails again.
Coldfusion-out.log appears normal until the Client_Reset errors start in the httperr log, which at that time the outputto Coldfusion-out.log stops as well as the output to the IIS log file. Nothing appears out of the ordinary leading up to the start of the client_reset errors. We don't see any relevant entries in the coldfusion exception.log around the time that the site goes down, but we do see many of the following errors when users attempt to download pdf files through our CMS. It is my understanding that this error is ok, just a notification that the client ended the request before the entire response was completed.
"Error","ajp-bio-8012-exec-3138","01/10/16","10:41:56","MURA904069D3142238F45DE68D824E76D9 BC","coldfusion.tagext.OutputException: The cause of this output exception was that: org.apache.catalina.connector.ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error.
I'm hoping for any advice on how to troubleshoot this further so that our server is stable and does randomly crash the application pool.