From dcook at cookware.com Fri Jan 2 10:02:23 2004 From: dcook at cookware.com (David Cook) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? Message-ID: <20040102180223.GA19387@puna.cookware.com> Aloha fellow xinetd users... We've identified a potential problem in the current version of xinetd (2.3.12) that may be a defect of the code itself, or a possible new form of DOS attack. We see the following on our console: Jan 1 17:14:31 puna xinetd[16732]: [ID 385394 daemon.error] Deactivating service ftp due to excessive incoming connections. Restarting in 15 seconds. Jan 1 17:14:46 puna xinetd[16732]: [ID 385394 daemon.error] Activating service ftp Jan 1 17:15:41 puna xinetd[16732]: [ID 254256 daemon.error] service pop3, accept: Too many open files (errno = 24) Jan 1 17:15:45 last message repeated 52347 times If you look at the times... you can see that a user hitting our FTP caused it to deactivate at 17:14:31 and then it came back up (as it should) 15 seconds later. I did a netstat and confirmed that indeed, a user from France had been bombarding our FTP ports - so that's correct. However, approximatly one minute later you can see that XINETD says that pop3 accept: has too many open files (errno = 24). Looking at our netstat we do NOT see any flood of popper hits associated with this. When we get the ACCEPT error XINETD enters a infinite loop where it pounds syslogd with that error message (note, it repeated 52,347 times). BTW, we're running Solaris 8 (5.8). Both syslogd and xinetd start running up huge CPU loads. Killing xinetd solves the problem and restarting it once load drops makes things ok. We noted this happening about 2 months ago - and at that time it appeared to happen about every 25 days or so. But this week we got it several times, including times where the server was at a lull (like JAN 1) - ALL of them were accompanied by someone hitting the FTP port and causing deactivation/reactivation. In looking at the xinetd source code - the file 'connection.c' has the following chunk (starting at line 50): if( SC_WAITS( scp ) ) { cp->co_descriptor = SVC_FD( sp ); } else { cp->co_descriptor = accept( SVC_FD( sp ), &(cp->co_remote_address.sa), &sin_len ) ; M_SET( cp->co_flags, COF_NEW_DESCRIPTOR ) ; } if ( cp->co_descriptor == -1 ) { msg( LOG_ERR, func, "service %s, accept: %m", SVC_ID( sp ) ) ; return( FAILED ) ; } Note the 'accept:' in the 'msg' - this is the only place we could find in the source where that string appears - so we're 99.999% certain this is the line issuing the error. Error number 24 is EMFILE (The per-process descriptor table is full) thus indicating that no more descriptors were available when the 'accept' call was invoked. If you happen to follow how this routine is called - it is called through a series of routines which start in the infinite loop in the main. However, what happens in this case is the return(FAILED) causes the loop to try again, in a dead heat for the most part. We have placed a TEMPORARY fix - to avoid load going high while we're not monitoring - by chainging the if detecting the error to this: if ( cp->co_descriptor == -1 ) { if (errno == 24) { msg( LOG_ERR, func, "XINETD STOPPED DUE TO NO MORE DESCRIPTORS"); exit(1); } msg( LOG_ERR, func, "service %s, accept: %m", SVC_ID( sp ) ) ; return( FAILED ) ; } You can see here that we are looking specifically for error number 24, and if we see it we issue a message to the log and EXIT - hopefully that SHOULD stop xinetd and the race condition. Since I put this minor change in last night we havn't had the occurance yet - so I have yet to prove that it works (if it does work, I'm intending on placing a SLEEP(10) at the start of the xinetd main and having that error message also invoke another version of xinetd before exiting - that should keep it up an running). Anyone else experiencing this problem and is there any known fix? Mahalo From linux_4ever at yahoo.com Fri Jan 2 10:34:56 2004 From: linux_4ever at yahoo.com (Steve G) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? In-Reply-To: <20040102180223.GA19387@puna.cookware.com> Message-ID: <20040102183456.23699.qmail@web9603.mail.yahoo.com> >We've identified a potential problem in the current version >of xinetd (2.3.12) that may be a defect of the code itself, >or a possible new form of DOS attack. Yes indeed. There are several more places where this could pop up in the code and we don't handle it well. At best, they can do a DOS attack. The correct action is to suspend the offending services when this occurs. This will close & remove the listening descriptor from the select loop and you won't see the problem any more. Ending or restarting xinetd is not good either. I will code something up and put into cvs this afternoon. Thanks for the bug report. -Steve Grubb __________________________________ Do you Yahoo!? Find out what made the Top Yahoo! Searches of 2003 http://search.yahoo.com/top2003 From linux_4ever at yahoo.com Fri Jan 2 13:14:39 2004 From: linux_4ever at yahoo.com (Steve G) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? In-Reply-To: <20040102180223.GA19387@puna.cookware.com> Message-ID: <20040102211439.54725.qmail@web9608.mail.yahoo.com> Hi, I committed a fix for this into cvs. Anyone that would like to try it out please download the latest from cvs. -Steve Grubb __________________________________ Do you Yahoo!? Find out what made the Top Yahoo! Searches of 2003 http://search.yahoo.com/top2003 From dcook at cookware.com Sun Jan 4 17:50:38 2004 From: dcook at cookware.com (David Cook) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? Message-ID: <20040105015038.GA1092@puna.cookware.com> Aloha Steve: The day you issued the CVS fix I installed the change. Today we were attacked again and the change did not work. Perplexed... I discovered why: In your changes to 'connection.c' you placed: if (errno == ENFILE) cps_service_stop(sp, "no available descriptors"); else However, the error being reported is: Jan 4 19:37:05 puna xinetd[13130]: [ID 254256 daemon.error] service pop3, accept: Too many open files (errno = 24) Glancing at the errno.h we discover: #define ENFILE 23 /* File table overflow */ #define EMFILE 24 /* Too many open files */ Thus, the block should have been EMFILE (M not N): if (errno == EMFILE) cps_service_stop(sp, "no available descriptors"); else Typo :) Do you agree? I've made the changes on my end (only to connection.c) and recompiled. We'll see if this does it. Mahalo Nui -- David Cook -- Cookware Inc. -- dcook@cookware.com TQworld and tranquility: www.TQworld.com Cookware Corporate: www.cookwareinc.com Hawaii/Asia Office: (808) 966-5049 (david cook) Mainland US Office: (317) 769-5049 (deborah sellers) Have you had tranquility today? Play now at http://www.tqworld.com From linux_4ever at yahoo.com Sun Jan 4 19:36:03 2004 From: linux_4ever at yahoo.com (Steve G) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? In-Reply-To: <20040105015038.GA1092@puna.cookware.com> Message-ID: <20040105033603.3890.qmail@web9606.mail.yahoo.com> >Typo :) Do you agree? Hmmm. The man 2 accept page says: EMFILE The per-process limit of open file descriptors has been reached. ENFILE The system maximum for file descriptors has been reached. I guess I overlooked one. It should be an 'OR' condition...I just fixed it in cvs. You can find the patch here: http://www.xinetd.org/pipermail/cvs-xinetd/2004-January/000221.html Thanks for the follow up. I, for one, would like hear feedback next time you are attacked. It's good to know the problem was handled. -Steve Grubb __________________________________ Do you Yahoo!? Find out what made the Top Yahoo! Searches of 2003 http://search.yahoo.com/top2003 From dcook at cookware.com Mon Jan 5 08:27:37 2004 From: dcook at cookware.com (David Cook) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? In-Reply-To: <20040105033603.3890.qmail@web9606.mail.yahoo.com> References: <20040105015038.GA1092@puna.cookware.com> <20040105033603.3890.qmail@web9606.mail.yahoo.com> Message-ID: <20040105162737.GA4388@puna.cookware.com> > Hmmm. The man 2 accept page says: > > EMFILE The per-process limit of open file descriptors has been > reached. > > ENFILE The system maximum for file descriptors has been reached. > > I guess I overlooked one. It should be an 'OR' condition...I just > fixed it in cvs. You can find the patch here: I agree - though in my original report (and all the instances we've seen) the only error actually being reported (generated) is EMFILE - but of course, both ENFILE and EMFILE should be trapped in there. I've updated both builtin.c and connection.c as per your changes and will let you know if/when we see the new message appear. Thanks for your speedy turnaround!! From jbminn at jbminn.com Tue Jan 6 06:03:31 2004 From: jbminn at jbminn.com (John Minnihan) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] Xinetd and file descriptor limits? Message-ID: <1543.66.182.81.202.1073397811.squirrel@leadville.jbminn.com> I am experiencing an error condition involving use of xinetd and apparent file descriptor limits. To summarize, when I start xinetd and parse numerous service files (as found in /etc/xinetd.d/), I receive the following error in the syslog: ------ Jan 4 14:43:03 estes xinetd[1852]: Service sserver-4121 disabled because of lack of file descriptors Jan 4 14:43:03 estes xinetd[1852]: xinetd Version 2.3.12 started with libwrap loadavg options compiled in. Jan 4 14:43:03 estes xinetd[1852]: Started working: 1016 available services ------ I am staring numerous sservers (an ssl'ified version of pserver, Corey Minyard's version) - in fact, over a thousand of these alone. Corresponding lines do exist in /etc/services (one per sserver port). The number in the service name relates to the port it listens on. I begin assigning sserver ports at 3025, and climb from there. Ports up to 4020 listen properly. I have a minimal set of services running in the privileged port space, among them ssh, httpd, httpsd, mysql, postgres and portsentry. System particulars are: Kernel .......... 2.4.18-14 (Redhat 8 base distro) xinetd........... 2.3.12 compiled with libwrap & loadavg /proc/sys/fs/file-max ............. various settings above 1024 ulimit -n ........ various settings well above 1024 (once at 200000) Web server ........ Apache 1.3.9 I have also placed in the xinted startup script two 'ulimit' statements that are intended to increase the fd limit for the user that starts xinetd. The statements are at the top of the script before any other action takes place. Those statements are: ulimit -Hn 65535 ulimit -Sn 8192 I have experienced this error on two separate servers, each running originally identical configurations, and then tweaking on one them versions of xinetd, the kernel and apache such that the error presents in the very latest of all three: xinetd (originally 2.3.10) .... 2.3.12 kernel (originally 2.4.20-8) ...2.6.0 apache (originally 1.3.9) ......2.0.40 I am running all these commands as root. My apache user is non-root. I have also run xinetd with '-d', but haven't been able to capture the output. It seems that the '-d' overrides atemmpts to redirect to a log. How is the output of '-d' captured for review? At the time the error presents, a 'lsof | wc -l' shows 5653 lines of output. I believe this maps directly to the total of both open ports and files, right? That is well below the value returned from 'ulimit -n', as well as below the value in /proc/sys/fs/file-max. I suspect I am hitting the system or user default of 1024 file descriptors, but I am curious why that number (1024) doesn't quite match the number of services successfully started, 1016. Are these in fact related? Is the error message believed to be informative? I am stumped as to what else to review. Is there another setting that controls this & I've simply missed it? Thanks for any assistance. -- John Minnihan Founder & Chief Architect https://www.freepository.com Free software development tools since 1999 From linux_4ever at yahoo.com Tue Jan 6 08:03:04 2004 From: linux_4ever at yahoo.com (Steve G) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] Xinetd and file descriptor limits? In-Reply-To: <1543.66.182.81.202.1073397811.squirrel@leadville.jbminn.com> Message-ID: <20040106160304.76606.qmail@web9601.mail.yahoo.com> >I am experiencing an error condition involving use of xinetd >and apparent file descriptor limits. It sounds like it. >I have also placed in the xinted startup script two 'ulimit' >statements that are intended to increase the fd limit for the >user that starts xinetd. I also think there is another limit settable by sysctl. If you get past this, I think xinetd only requests 1024 descriptors. You would need to tweek it to set the rlimit higher for the per process limit. >How is the output of '-d' captured for review? I startup x and look at it though a terminal window after setting the number of lines for it to buffer. >I suspect I am hitting the system or user default of 1024 file >descriptors, but I am curious why that number (1024) doesn't >quite match the number of services successfully started, 1016. xinetd uses a couple for internal messaging, stdin,stdout,stderr are all set to /dev/null and then glibc also leaks descriptors in pmap_set, and then error logging. >I am stumped as to what else to review. Is there another setting >that controls this & I've simply missed it? Take a look in init.c. Look for setfd_limit(). You may want to tweek that function to request more descriptors. -Steve Grubb __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus From jbminn at jbminn.com Tue Jan 6 14:26:18 2004 From: jbminn at jbminn.com (John Minnihan) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] Xinetd and file descriptor limits? In-Reply-To: <20040106160304.76606.qmail@web9601.mail.yahoo.com> References: <1543.66.182.81.202.1073397811.squirrel@leadville.jbminn.com> <20040106160304.76606.qmail@web9601.mail.yahoo.com> Message-ID: <18337.207.14.148.1.1073427978.squirrel@leadville.jbminn.com> Thanks for the detailed reply. See inline comments/questions below. Steve G said: > I also think there is another limit settable by sysctl. If you > get past this, I think xinetd only requests 1024 descriptors. You > would need to tweek it to set the rlimit higher for the per > process limit. Can you provide any guidance on tweaking the rlimit? I can grok new things pretty well, but a starting point is always handy. >>How is the output of '-d' captured for review? > > I startup x and look at it though a terminal window after setting > the number of lines for it to buffer. Ah, ok. > >>I suspect I am hitting the system or user default of 1024 file >>descriptors, but I am curious why that number (1024) doesn't >>quite match the number of services successfully started, 1016. > > xinetd uses a couple for internal messaging, stdin,stdout,stderr > are all set to /dev/null and then glibc also leaks descriptors in > pmap_set, and then error logging. So that's a reasonable number. Good. > > Take a look in init.c. Look for setfd_limit(). You may want to > tweek that function to request more descriptors. I took a look at init.c and saw the function, but must admit I haven't any idea how to tweak it. I don't see where it gets it value(s) - from one of the includes? -- John Minnihan Founder & Chief Architect https://www.freepository.com Free software development tools since 1999 From ma+xi at dt.e-technik.uni-dortmund.de Tue Jan 6 15:15:56 2004 From: ma+xi at dt.e-technik.uni-dortmund.de (Matthias Andree) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? In-Reply-To: <20040105033603.3890.qmail@web9606.mail.yahoo.com> (Steve G.'s message of "Sun, 4 Jan 2004 19:36:03 -0800 (PST)") References: <20040105033603.3890.qmail@web9606.mail.yahoo.com> Message-ID: Steve G writes: >>Typo :) Do you agree? > > Hmmm. The man 2 accept page says: > > EMFILE The per-process limit of open file descriptors has been > reached. > > ENFILE The system maximum for file descriptors has been reached. > > I guess I overlooked one. It should be an 'OR' condition...I just > fixed it in cvs. You can find the patch here: > > http://www.xinetd.org/pipermail/cvs-xinetd/2004-January/000221.html > > Thanks for the follow up. I, for one, would like hear feedback > next time you are attacked. It's good to know the problem was > handled. Wait a second. The service is disabled for 15 seconds when the connection rate is too high? That's the same non-working scheme found in inetd, disable any service that connects more than N times per minute for 10 minutes. It doesn't handle high or excessive load. Disabling the service for some time will bring people back in the time when the service is up, so it disables itself again... no good. If so, what advantage is xinetd over tcpsvd or tcpserver, apart from IPv6? -- Matthias Andree Encrypt your mail: my GnuPG key ID is 0x052E7D95 From linux_4ever at yahoo.com Tue Jan 6 19:01:26 2004 From: linux_4ever at yahoo.com (Steve G) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? In-Reply-To: Message-ID: <20040107030126.63220.qmail@web9604.mail.yahoo.com> >The service is disabled for 15 seconds when the connection >rate is too high? Nooo. This bug is more like the tcp/wait problem. It causes xinetd to go into a tight loop sending syslog messages. The only way to break the loop is to close the descriptor and remove it from the select mask. The bug he described was that the per process fd limit was exhausted. This means accept fails, which causes it to go back to select, and the listening descriptor is still readable, accept fails again, etc. This is easy to do since linux has about a 2 minute turn around on used descriptors due to so_linger. Its been my experience that its rare to hit the per process limit in most situations, but it is there. This is a different problem than cps. -Steve Grubb __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus From jbminn at jbminn.com Tue Jan 6 19:21:30 2004 From: jbminn at jbminn.com (John Minnihan) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] Xinetd and file descriptor limits? In-Reply-To: <20040106160304.76606.qmail@web9601.mail.yahoo.com> References: <1543.66.182.81.202.1073397811.squirrel@leadville.jbminn.com> <20040106160304.76606.qmail@web9601.mail.yahoo.com> Message-ID: <4575.66.182.81.202.1073445690.squirrel@leadville.jbminn.com> I poked around a bit and came up with a solution. Let me know if you think this can or will cause undesirable side affects- I added a #define FD_SETSIZE 8192 to the top of init.c and recompiled. All services now start error-free. Steve G said: > > Take a look in init.c. Look for setfd_limit(). You may want to > tweek that function to request more descriptors. > > -Steve Grubb > -- John Minnihan Founder & Chief Architect https://www.freepository.com Free software development tools since 1999 From linux_4ever at yahoo.com Tue Jan 6 19:31:20 2004 From: linux_4ever at yahoo.com (Steve G) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] Xinetd and file descriptor limits? In-Reply-To: <18337.207.14.148.1.1073427978.squirrel@leadville.jbminn.com> Message-ID: <20040107033120.30844.qmail@web9608.mail.yahoo.com> >Can you provide any guidance on tweaking the rlimit? Only change the code inside the RLIMIT_NOFILE ifdef. In that section of code, set the values of rl.rlim_cur & rl.rlim_max to what you want before maxfd is initialized and call setrlimit. -Steve Grubb __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus From linux_4ever at yahoo.com Tue Jan 6 19:45:07 2004 From: linux_4ever at yahoo.com (Steve G) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] Xinetd and file descriptor limits? In-Reply-To: <4575.66.182.81.202.1073445690.squirrel@leadville.jbminn.com> Message-ID: <20040107034507.17610.qmail@web9605.mail.yahoo.com> >I added a #define FD_SETSIZE 8192 to the top of init.c and >recompiled. All services now start error-free. Try reloading xinetd by sending it a SIGHUP. If that passes I think you can use what you have. -Steve Grubb __________________________________ Do you Yahoo!? Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes http://hotjobs.sweepstakes.yahoo.com/signingbonus From jbminn at jbminn.com Tue Jan 6 20:21:42 2004 From: jbminn at jbminn.com (John Minnihan) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] Xinetd and file descriptor limits? In-Reply-To: <20040107034507.17610.qmail@web9605.mail.yahoo.com> References: <4575.66.182.81.202.1073445690.squirrel@leadville.jbminn.com> <20040107034507.17610.qmail@web9605.mail.yahoo.com> Message-ID: <4588.66.182.81.202.1073449302.squirrel@leadville.jbminn.com> Behaves as expected. Thanks. Steve G said: >>I added a #define FD_SETSIZE 8192 to the top of init.c and >>recompiled. All services now start error-free. > > Try reloading xinetd by sending it a SIGHUP. If that passes I > think you can use what you have. > > -Steve Grubb > > __________________________________ > Do you Yahoo!? > Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes > http://hotjobs.sweepstakes.yahoo.com/signingbonus > _______________________________________________ > xinetd mailing list > xinetd@xinetd.org > http://www.xinetd.org/mailman/listinfo/xinetd > -- John Minnihan Founder & Chief Architect https://www.freepository.com Free software development tools since 1999 From ma+xi at dt.e-technik.uni-dortmund.de Wed Jan 7 12:43:23 2004 From: ma+xi at dt.e-technik.uni-dortmund.de (Matthias Andree) Date: Thu Oct 6 10:34:54 2005 Subject: [xinetd] xinetd bug or new DoS attack? In-Reply-To: <20040107030126.63220.qmail@web9604.mail.yahoo.com> (Steve G.'s message of "Tue, 6 Jan 2004 19:01:26 -0800 (PST)") References: <20040107030126.63220.qmail@web9604.mail.yahoo.com> Message-ID: Steve G writes: > Its been my experience that its rare to hit the per process limit > in most situations, but it is there. This is a different problem > than cps. Ah, ok, thanks for clearing this up. That might BTW be a reason to pre-fork a master server per service when the load can be high (which is what tcpsvd and tcpserver to by design - they handle only one service, so you get to run one daemon per service). -- Matthias Andree Encrypt your mail: my GnuPG key ID is 0x052E7D95