I am experiencing the most bizarre thing that so far I am unable to reproduce with my own visits to the site.
After restarting Apache2 my cacti graphs show that the child processes increment consistently over the course of a day without dropping back down during off hours. This behavior eventually leaves the website inaccessible...
Looking at server-status it is filled with Ws (Sending Reply) and GET calls to my cfm applications :
Current Time: Tuesday, 22-Jul-2014 16:33:00 PDTRestart Time: Monday, 21-Jul-2014 22:51:12 PDTParent Server Generation: 0Server uptime: 17 hours 41 minutes 48 secondsTotal accesses: 194844 - Total Traffic: 3.8 GBCPU Usage: u201.55 s34.46 cu0 cs0 - .37% CPU load3.06 requests/sec - 63.2 kB/second - 20.6 kB/request73 requests currently being processed, 4 idle workers
WWWWWWWWWWWWWWWWWWWWWWWWWKWWWWWWWWWWWWWWWWWCWWWWW_WWWWWWCWWW_WWW _WKW...KWW.W_KWW....W........................................... ................................................................ ................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
| Srv | PID | Acc | M | CPU | SS | Req | Conn | Child | Slot | Client | VHost | Request |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0-0 | 15074 | 0/46/1370 | W | 7.39 | 46158 | 0 | 0.0 | 0.44 | 23.89 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=7000 HTTP/1.1 |
| 1-0 | 11563 | 0/47/468 | W | 2.75 | 58867 | 0 | 0.0 | 4.69 | 13.64 | 192.168.1.10 | www.mysite.edu | GET /catalog/index.cfm?courselist=list&dept=&searchc=PEHW%20148 |
| 2-0 | 12906 | 0/65/884 | W | 7.30 | 54536 | 0 | 0.0 | 0.80 | 14.62 | 192.168.1.10 | www.mysite.edu | GET /athletics/resources/nwaacc-athlete-of-the-week/ HTTP/1.1 |
| 3-0 | 13840 | 0/41/1085 | W | 4.01 | 51162 | 0 | 0.0 | 0.56 | 20.57 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=17001 HTTP/1.1 |
| 4-0 | 15928 | 0/20/1635 | W | 5.40 | 43715 | 0 | 0.0 | 0.06 | 41.37 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=37000 HTTP/1.1 |
| 5-0 | 18774 | 0/19/2387 | W | 0.33 | 34564 | 0 | 0.0 | 0.24 | 52.70 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=19009 HTTP/1.1 |
| 6-0 | 4321 | 0/36/6612 | W | 3.61 | 13200 | 0 | 0.0 | 0.28 | 129.74 | 192.168.1.10 | www.mysite.edu | GET /directory/index.cfm?directory=department&deptexp=28011 HTT |
| 7-0 | 13077 | 0/0/808 | W | 0.42 | 54383 | 0 | 0.0 | 0.00 | 24.81 | 192.168.1.10 | www.mysite.edu | GET /directory/index.cfm?directory=department&deptexp=6005 HTTP |
| 8-0 | 16488 | 0/118/1673 | W | 12.39 | 40692 | 0 | 0.0 | 1.30 | 35.44 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=31003 HTTP/1.1 |
| 9-0 | 10726 | 0/15/110 | W | 0.58 | 61963 | 0 | 0.0 | 0.05 | 1.83 | 192.168.1.10 | www.mysite.edu | GET /directory/index.cfm?directory=All&index=Q HTTP/1.1 |
| 10-0 | 13154 | 0/1/688 | W | 0.00 | 54165 | 0 | 0.0 | 0.00 | 16.83 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=All&firstname=Patrick&lastname=Murphy |
| 11-0 | 12590 | 0/25/516 | W | 4.45 | 55851 | 0 | 0.0 | 0.76 | 11.19 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=4000 HTTP/1.1 |
| 12-0 | 12551 | 0/13/454 | W | 1.84 | 56055 | 0 | 0.0 | 0.38 | 10.00 | 192.168.1.10 | www.mysite.edu | GET /directory/index.cfm?directory=department&deptexp=20001 HTT |
| 13-0 | 13333 | 0/23/626 | W | 3.86 | 53189 | 0 | 0.0 | 0.57 | 11.66 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=31005 HTTP/1.1 |
| 14-0 | 12410 | 0/13/387 | W | 2.70 | 56484 | 0 | 0.0 | 0.42 | 10.55 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=6003 HTTP/1.1 |
| 15-0 | 13162 | 0/70/389 | W | 10.81 | 53114 | 0 | 0.0 | 0.86 | 5.60 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=6005 HTTP/1.1 |
| 16-0 | 12309 | 0/22/275 | W | 2.23 | 56878 | 0 | 0.0 | 0.43 | 3.91 | 192.168.1.10 | www.mysite.edu | GET /directory/?directory=department&deptexp=20005 HTTP/1.1 |
| 17-0 | 13163 | 0/57/341 | W | 11.85 | 53120 | 0 | 0.0 | 1.38 | 6.49 | 192.168.1.10 | www.mysite.edu | GET /catalog/index.cfm?courselist=list&dept=&searchc=ENGR%26%20 |
I have straced a hung process to only find the following :
strace -p 6472
Process 6472 attached - interrupt to quit
read(23,
Another interesting bit of info, none of these GET requests make it into my access.log file which I find very peculiar as well.
Here are my CF Specs
Server Details
Server Product ColdFusion
Version 10,0,13,287689
Tomcat Version 7.0.23.0
Edition Enterprise
Serial Number
Operating System UNIX
OS Version 3.2.0-65-generic
Update Level /opt/coldfusion10/cfusion/lib/updates/chf10000013.jar
Adobe Driver Version 4.1 (Build 0001)
JVM Details
Java Version 1.6.0_29
Java Vendor Sun Microsystems Inc.
Here are my Apache2 Specs
Server version: Apache/2.2.22 (Ubuntu)
Server built: Apr 17 2014 21:49:25
Server's Module Magic Number: 20051115:30
Server loaded: APR 1.4.6, APR-Util 1.3.12
Compiled using: APR 1.4.6, APR-Util 1.3.12
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/etc/apache2"
-D SUEXEC_BIN="/usr/lib/apache2/suexec"
-D DEFAULT_PIDLOG="/var/run/apache2.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="mime.types"
-D SERVER_CONFIG_FILE="apache2.conf"
I am hoping this is no normal behavior for Coldfusion 10.
Many thanks in advance.