Recent Site Instability Issues - August 2011

[Bob Hubbard]

Some of you may have experienced some access issues the last few days. Database errors, pages not loading, things timing out.

Well, after some digging we figured out part of the problem. We're seeing a large spike in traffic, resulting in us maxing out the server we're on. Right now, a major server upgrade isn't financially possible. With the down state of the economy, there isn't another $200 a month in the operating budget to cover the cost increase of a new box. Right now, we're doing some intensive performance tuning. Hopefully you're already seeing some faster page loads and quicker access. I'm going to be evaluating this over the next few days and determining what other options we have. We may add additional memory to the server, which will hopefully boost things a bit as well. If we do, we'll post a notice at least a day or 2 in advance. We're also going to be performing some additional software updates in the near future so we may have some down time then as well. I'll announce that in advance.

I apologize for the inconvenience, and want everyone to know we're working on it.

[Big Don]

Oh...
I thought it was on my end. I was blaming my ISP.

[Carol]

*whew*

I noticed it too and kept forgetting to drop you a line. I'm glad its "just" an overloaded server and not database-go-boom.

[Bob Hubbard]

We had over 5,000 visitors today. Ran us out of free memory (2GB) a few times which caused huge load on the drives (SCSI) and processors (AMD Athlon(tm) 64 X2 Dual Core Processor 5000+ 2.6GHZ).

When your system (server or desktop) runs out of real memory, it'll usually use your hard drive as 'virtual memory'. This is slower and causes your processor to work harder. Solution is to add more RAM and fine tune things.

All else fails, we feed the hamster some crack and run.

[The Last Legionary]

Seems a bit quicker. I'll go harass a few folks, see how quickly it bans me.



I kid. I kid.

[Bob Hubbard]

So far, no new error messages, and page loads feel faster than on KT or FMAT.

Hows it looking on everyone elses end?

[MaxiMe]

So far, no new error messages, and page loads feel faster than on KT or FMAT.

Hows it looking on everyone elses end?

Seems better on my end today.
RAM Gotta love quick and fairly inexpensive fixes
Tuning the page file on the server might help a little (if you haven't already done that).

[Bob Hubbard]

we installed memcache and tuned the php config a bit. adding more ram is pending.

[Carol]

Seems a bit quicker. I'll go harass a few folks, see how quickly it bans me.



I kid. I kid.

Wasn't it your turn to buy the hampster chow?

[MaxiMe]

Wasn't it your turn to buy the hampster chow?

Faint echoes of a rim shot. badda bing

[RandomPhantom700]

Bob:

I do not know if this is a related issue, or if it's a problem with my browser as opposed to the site's server.

However, I do not have automatic login set on my account, erego I type in my information each visit. I noticed this morning, upon initially bringing the site up, the text in my login/password would remain gray when typing (it is normally black while being typed), and more importantly, my password would not be blocked as I was typing it. Upon hitting submit, the page reloaded without logging in, and things would work normally upon the second attempt, with password properly blocked by bullets.

I've seen this twice this morning, and like I said, it could very well be a browser glitch on my end or it could be a server error. In either event, I wanted to let it be known in case anyone else has observed similar.

[Bob Hubbard]

Possibly a CSS issue, something not loading right. Happens to me sometimes on all my vB sites. Supposedly a known bug, to be fixed 'in a later release'.