[SOLVED] How to delete large number of files in clientmqueue?

We have a VPS server running Magento flawlessly for nearly a year. The filesystem has been completely filled with mails queued in clientmqueue, and MySQL can't add more data causing the website to halt. Sendmail was running fine, but we've had a difficult time deleting files residing in /var/spool/clientmqueue folder. The following commands failed:

# cd /var/spool/clientmqueue
# rm -rf *
-bash: /bin/rm: Argument list too long
# foreach i in {a..z}; do rm -f ${i}*; done
-bash: /bin/rm: Argument list too long
# find . -type f exec rm -f {} \;
find: cannot fork: Cannot allocate memory

There are just too many files in the clientmqueue folder, and Linux couldn't handle the number of files for deletion. Our rescue was xargs command. But, before we do this we wanted to make sure the queue isn't building up as we delete the files.

# service sendmail stop
# cd /var/spool
# mv clientmqueue clientmqueue.x
# cd clientmqueue.x
# find . -type f |xargs rm -f

It took more than a day to complete the execution of the above command, but it was deleting files slowly but surely freeing up space on the hard disk.

Removing those files is half of the problem, and identifying the process of creating those files and preventing this from happening in the future is another problem. So, how do we find the process which is writing those files in clientmqueue. The lsof command will help us identify the process.

# lsof +D /var/spool/clientmqueue
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
sendmail 14378 smmsp  cwd    DIR  202,0     4096 163992 clientmqueue

The clientmqueue is the mail queue for handling locally generated outbound emails. If you have sendmail running, and see this problem then it indicates you may have a compromised website spamming the world using your server. Your compromised website (or server) is generating a lot more mail than the sendmail can handle causing clientmqueue to fill up. You will have to view the df* and qf* files residing in the /var/spool/clientmqueue to see where these messages come from and resolve the root cause of the problem.