Ok here is the scoop...
We have been having an issue on our server, where our server load goes from 0.40 to 25.7 (or higher) when this happens the site is VERY slow in fact its unavailable to many. The only way to get everything online again and back to speed is by restarting service on the server (of course when I do this the users get Database errors while the services are restarting)
This could happen once a day, sometimes it did not happen for days or weeks. The server load spike would come out of no where. Yesterday it happened 12 times.
Last week it happened a lot as well, so we contacted both vBulletin (our forum software) and Cpanel (the folks who write software to tie all the parts of our server together nicely) for help.
The Cpanel folks had us run some tests when this problem was happening, and after running these tests they said its a vBulletin issue, sorry we can't help you.
I contacted vBulletin who told me its a cpanel issue and to contact them. I then sent them my log files which seems to show that its a vbulletin problem, and then their tune changed.
They told me to turn off everything that did not come with vBulletin (stuff like the member map, arcade, spell check, webmail, zoints, the chat room, etc...)
I wrote them back and told them that I could not just shut everything off, so I was told to turn them off one by one and see which one seems to solve the issue. And they looked through the logs and said it appeared to be a program that had some mail call functions that was causing the error.
So earlier in the week I shut off the WebMail interface (I myself was SURE this was it) but the next morning the server load shot up and crashed the site..)
So then LER did some updates to our server last night and loaded up the latest OS kernel with the hopes that would fix the problem.
Nope it happened here today twice. That was not the fix.
So now I am going back to the vBulletin advice and am turning off features one by one.
Today the arcade was turned off.
Since the arcade was turned off we have not had the issue. However with that said not enough time has gone by to determine if it is the arcade causing the issue. It could take a few days to tell that, since this problem did not happen every day.
If the arcade IS the issue, hope is not lost. I would then uninstall it (and all the games) and would then reinstall it (cleanly) this would fix a number of small bugs we have seen in the arcade. This arcade was installed over the top of another arcade software in which it replaced it. So a clean install would do us good. It would take awhile to install all 5000 games again but no biggie.
If its not the arcade then we turn it back on and try another feature and see if that correct the issue...
I do appologize for this, as I take great pride in giving you the best and fastest site possible. Over the past few weeks it has been like a full time just just sitting here watching the server load monitor and restarting services as needed to keep things running smoothly. But its something that we should not have to do.
We will get it fixed and thank you for your patience.
