Site Crash 8-25-2009 - Data Lost.

[Bob Hubbard]

My attempt tonight to import another site into MT in an attempt to preserve the other site and expand MT failed with almost catastrophic results. Despite taking the precaution of having last nights backup, in addition to 2 more current backups, all were somehow unrestorable, As a result, the most recent restorable database backup is from August 18th. All data in between is apparently lost. All posts, threads, pms, rep, game scores, and new registrations.

I deeply apologize for this and assure everyone that I am taking extra precautions to ensure that this does not happen again.

 

[Carol]

Bob,

In my line of work I have seen first-hand occasions where tested procedures, tested migrations, and proper backups all took place. The targeted system was healthy and running without errors. And somehow the migration had catastrophic results. Things like this don't usually happen. But when they do, the results are disastrous.

Its a horrible feeling when you know you've done everything right, yet the application won't start up, the minutes of downtime start becoming hours, and you have to wake up every possible resource to pull them in to resuscitating anything that can be saved. And the undercurrent to it all is that you don't have hope. You know you are looking at a lost cause. You know the fault killed off everything usable. The only possibility is finding the fault in a system that showed no signs of trouble, and fixing it. Or (proverbially) scraping out the corruption entirely and seeing if you have enough application structure left to still run something....anything. Its not your fault that it happens, and the customer likely understands that its not you're fault, but you're still the one at the switch that has to put humpty dumpty back together again after he seems smashed to smithereens.

I'm sorry the site merge didn't work, but you deserve a huge round of applause for all you did to bring MT back, and to keep our years of contributions from being lost, permanently. That was a huge effort and I am so glad you found a way to make it work.

Thank you SO much! :asian:

 

[Bob Hubbard]

I've been in a "oh ****" state for 3 hrs. The admin at the data center is as stumped as I am that none of the backups would import. I'm going to get some sleep and spend tomorrow going through the last weeks emails to try and repair some of the damage.

 

[Makalakumu]

Good luck, Bob. I've got faith in you!

 

[Bruno@MT]

I know fully well the sinking feeling of despair that you speak of.
Have a night's sleep. Things will look better tomorrow. And if they don't, well, at least you are less tired

 

[Danjo]

Wow. I have to piss off everyone all over again!

 

[Sukerkin]

So that's what happened! I was just looking for an appropriate place to note that a week or so of the sites history seemed to have gone missing; this included 'Rep' and 'Thanks' so I figured that something major had gone awry with the site merge.

Deeply sorry to hear of the catastrophy and the data loss is a particular wrench given that I'd just started to become more active again here (those pleasant 'nods' from my fellow members will be missed too but at least they still exist in my memory).

 

[Decker]

Murphy strikes again.

All the best in fixing the problem. I feel your pain.

 

[morph4me]

I was wondering what happened. Thanks for keeping everyone in the loop, Bob. Good Luck.

 

[Brian R. VanCise]

I just went through a data crash at my work. It was brutal so I understand completely.

 

[arnisador]

A lost week isn't so bad on the scale of a site that's now been here for, what, 8 years? Good luck with the remaining details.

 

[celtic_crippler]

 

[Joab]

My attempt tonight to import another site into MT in an attempt to preserve the other site and expand MT failed with almost catastrophic results. Despite taking the precaution of having last nights backup, in addition to 2 more current backups, all were somehow unrestorable, As a result, the most recent restorable database backup is from August 18th. All data in between is apparently lost. All posts, threads, pms, rep, game scores, and new registrations.

I deeply apologize for this and assure everyone that I am taking extra precautions to ensure that this does not happen again.

Such things happen, nobody is perfect, we'll get by without all of that.

 

[AceHBK]

Thanks for letting us know. Well I guess it could have been worse so I am glad you were able to save what you could.


I just thought no one liked what I had to say..

 

[mwd0818]

Wow. I have to piss off everyone all over again!

Because the last week you've been doing something different than the last five???

Don't worry Danjo, I think we have enough of your posts to keep certain people pissed off for a while . . .

In the meantime, I'm going to go back and reread the Kata versus Sparring threads . . .


Bob - not a problem - we are all thankful for your time and effort, and sometimes things happen. We will expect you to be demoted from Senior Great Supreme Forum Grandmaster down to just Grand Pooba for a week or so though . . .

 

[Jade Tigress]

What they all said Bob. :asian: We'll live.

 

[Xue Sheng]

WHAT!!!!

Everything I wrote is gone :tantrum:

Ahhh just kidding, as you well know in the wonderful world of IT things can and will go wrong....castropically... it is just all part of the game.

It does answer my question as to what the heck happened

Besides

:lisafault:

But Bob, you do of course realize that since I am an IT person I now am required to harass you mercilessly about this….even though in all honesty it can be chalked up to…stuff happens :EG:

 

[zepedawingchun]

Just think, we get to do the last 7 days all over again, and this time make corrections and do it better. Very seldom do we get to go back and start all over again.

 

[Bob Hubbard]

I'm trying to recover the last week, but it's not likely unfortunately.

 

[FearlessFreep]

You mean you lost my witty retort in the US Politics forum? You all would've been amazed. It was devastating and debilitating; a truly once in a life time combination of wise insight and cunning sarcasm with unassailable logic and uncanny accuracy.

*sigh* well everyone can just bow and submit to my presence anyway. Trust me, you would've known I deserve it had the post not been losy