Site Crash 8-25-2009 - Data Lost.

Bob Hubbard

Retired
MT Mentor
Founding Member
Lifetime Supporting Member
MTS Alumni
Joined
Aug 4, 2001
Messages
47,245
Reaction score
772
Location
Land of the Free
My attempt tonight to import another site into MT in an attempt to preserve the other site and expand MT failed with almost catastrophic results. Despite taking the precaution of having last nights backup, in addition to 2 more current backups, all were somehow unrestorable, As a result, the most recent restorable database backup is from August 18th. All data in between is apparently lost. All posts, threads, pms, rep, game scores, and new registrations.

I deeply apologize for this and assure everyone that I am taking extra precautions to ensure that this does not happen again.

:(
 

Carol

Crazy like a...
MT Mentor
Lifetime Supporting Member
MTS Alumni
Joined
Jan 16, 2006
Messages
20,311
Reaction score
541
Location
NH
Bob,

In my line of work I have seen first-hand occasions where tested procedures, tested migrations, and proper backups all took place. The targeted system was healthy and running without errors. And somehow the migration had catastrophic results. Things like this don't usually happen. But when they do, the results are disastrous.

Its a horrible feeling when you know you've done everything right, yet the application won't start up, the minutes of downtime start becoming hours, and you have to wake up every possible resource to pull them in to resuscitating anything that can be saved. And the undercurrent to it all is that you don't have hope. You know you are looking at a lost cause. You know the fault killed off everything usable. The only possibility is finding the fault in a system that showed no signs of trouble, and fixing it. Or (proverbially) scraping out the corruption entirely and seeing if you have enough application structure left to still run something....anything. Its not your fault that it happens, and the customer likely understands that its not you're fault, but you're still the one at the switch that has to put humpty dumpty back together again after he seems smashed to smithereens.

I'm sorry the site merge didn't work, but you deserve a huge round of applause for all you did to bring MT back, and to keep our years of contributions from being lost, permanently. That was a huge effort and I am so glad you found a way to make it work.

Thank you SO much! :asian:
 
OP
Bob Hubbard

Bob Hubbard

Retired
MT Mentor
Founding Member
Lifetime Supporting Member
MTS Alumni
Joined
Aug 4, 2001
Messages
47,245
Reaction score
772
Location
Land of the Free
I've been in a "oh ****" state for 3 hrs. The admin at the data center is as stumped as I am that none of the backups would import. I'm going to get some sleep and spend tomorrow going through the last weeks emails to try and repair some of the damage. :(
 

Bruno@MT

Senior Master
Joined
Feb 24, 2009
Messages
3,399
Reaction score
74
I know fully well the sinking feeling of despair that you speak of.
Have a night's sleep. Things will look better tomorrow. And if they don't, well, at least you are less tired :)
 

Sukerkin

Have the courage to speak softly
MT Mentor
Lifetime Supporting Member
MTS Alumni
Joined
Sep 15, 2006
Messages
15,325
Reaction score
493
Location
Staffordshire, England
So that's what happened! I was just looking for an appropriate place to note that a week or so of the sites history seemed to have gone missing; this included 'Rep' and 'Thanks' so I figured that something major had gone awry with the site merge.

Deeply sorry to hear of the catastrophy and the data loss is a particular wrench given that I'd just started to become more active again here (those pleasant 'nods' from my fellow members will be missed too but at least they still exist in my memory).
 
Last edited:

Decker

Green Belt
Joined
Jun 26, 2007
Messages
109
Reaction score
4
Murphy strikes again.

All the best in fixing the problem. I feel your pain.
 

morph4me

Goin' with the flow
MT Mentor
MTS Alumni
Joined
Sep 5, 2006
Messages
6,779
Reaction score
124
Location
Ossining , NY
I was wondering what happened. Thanks for keeping everyone in the loop, Bob. Good Luck.
 

arnisador

Sr. Grandmaster
MTS Alumni
Joined
Aug 28, 2001
Messages
44,573
Reaction score
456
Location
Terre Haute, IN
A lost week isn't so bad on the scale of a site that's now been here for, what, 8 years? Good luck with the remaining details.
 

Joab

2nd Black Belt
Joined
Dec 8, 2008
Messages
763
Reaction score
9
My attempt tonight to import another site into MT in an attempt to preserve the other site and expand MT failed with almost catastrophic results. Despite taking the precaution of having last nights backup, in addition to 2 more current backups, all were somehow unrestorable, As a result, the most recent restorable database backup is from August 18th. All data in between is apparently lost. All posts, threads, pms, rep, game scores, and new registrations.

I deeply apologize for this and assure everyone that I am taking extra precautions to ensure that this does not happen again.

:(

Such things happen, nobody is perfect, we'll get by without all of that.
 

AceHBK

Master Black Belt
Joined
Jan 29, 2006
Messages
1,325
Reaction score
14
Location
Arizona
Thanks for letting us know. Well I guess it could have been worse so I am glad you were able to save what you could.


I just thought no one liked what I had to say.. :)
 

mwd0818

Green Belt
Joined
Jul 23, 2009
Messages
174
Reaction score
9
Location
Louisville, KY
Wow. I have to piss off everyone all over again! ;)

Because the last week you've been doing something different than the last five???

Don't worry Danjo, I think we have enough of your posts to keep certain people pissed off for a while . . . :D

In the meantime, I'm going to go back and reread the Kata versus Sparring threads . . .


Bob - not a problem - we are all thankful for your time and effort, and sometimes things happen. We will expect you to be demoted from Senior Great Supreme Forum Grandmaster down to just Grand Pooba for a week or so though . . .
 

Xue Sheng

All weight is underside
Joined
Jan 8, 2006
Messages
34,500
Reaction score
9,764
Location
North American Tectonic Plate
:eek: WHAT!!!!

Everything I wrote is gone :tantrum:

Ahhh just kidding, as you well know in the wonderful world of IT things can and will go wrong....castropically... it is just all part of the game.

It does answer my question as to what the heck happened

Besides

:lisafault:

But Bob, you do of course realize that since I am an IT person I now am required to harass you mercilessly about this….even though in all honesty it can be chalked up to…stuff happens :EG:
 

zepedawingchun

Black Belt
Joined
Jun 15, 2009
Messages
582
Reaction score
17
Location
Moore, SC
Just think, we get to do the last 7 days all over again, and this time make corrections and do it better. Very seldom do we get to go back and start all over again.
 
OP
Bob Hubbard

Bob Hubbard

Retired
MT Mentor
Founding Member
Lifetime Supporting Member
MTS Alumni
Joined
Aug 4, 2001
Messages
47,245
Reaction score
772
Location
Land of the Free
I'm trying to recover the last week, but it's not likely unfortunately.
 

FearlessFreep

Senior Master
Joined
Dec 20, 2004
Messages
3,088
Reaction score
98
Location
Phoenix, Arizona
You mean you lost my witty retort in the US Politics forum? You all would've been amazed. It was devastating and debilitating; a truly once in a life time combination of wise insight and cunning sarcasm with unassailable logic and uncanny accuracy.

*sigh* well everyone can just bow and submit to my presence anyway. Trust me, you would've known I deserve it had the post not been losy
 

Latest Discussions

Top