View Full Version : Major Outage Hits MartialTalk
Bob Hubbard 12-14-2002, 10:36 PM On Wednesday night, at about 2am, we were hit with an extremely nasty surprise.
We were blown off the map...not just us, but our entire server was utterly and completely trashed. While the cause is under investigation, the results are evident. We suffered a complete and total data loss. All information on the server was irrepairibly lost.
I've been working non stop since then to bring over 150 web sites back up, many without current backups.
Myself, my staff, and many of my associates have been up with minimal time for sleep since then. I've had about 8 hours sleep myself since early thursday morning. To say I need a nap is an understatement.
So, whats the situation with MartialTalk?
Heres the straight shot.
We lost about a weeks worth of posts and sign ups.
There are several glitches that are still being worked out.
A few of our modifications are working halfway, and causing some display issues.
You will all notice the modified address...we are now at martialtalk.com/forum. This is for 2 reasons...1 is it let me work on bringing things back up without interuption. The other is that in the future we will be greatly expanding the scope and mission of MartialTalk.
I want to thank all of our members and supporters for their patience during this time. I also want to extend my heartfelt gratitude to all those who offered their help, or well wishes while we worked on things, and to the support staff at our datacenter who went above and beyond to help us out.
We are not through...we will rise from the ashes and become better than we were.
Peace.
arnisador 12-14-2002, 10:52 PM Congratulations and thanks to Bob Hubbard and the others who worked so hard on resurrecting MartialTalk. Thanks to a backup strategy, hard work, and luck, only about a week's worth of posts have been lost.
tunetigress 12-14-2002, 11:52 PM Awesome job you guys for getting things working working so well again! I don't know how you do it, but I'm sure I speak for us all in saying how much your efforts are appreciated in making posting possible again. :asian: :asian: :asian:
Elfan 12-14-2002, 11:52 PM Thanks guys!
Good luck with getting the rest of the stuf back together.
Bob Hubbard 12-15-2002, 01:39 AM Short update, then im getting some sleep...
I've already gotten notification of a number of hiccups, and ran into a few myself. We litterally crowbared this place back up, and it looks like a few patches are leaking.
To this end, I'm stripping out all of the mods I've put in and taking us back to 'factory spec' for a little while. Once we've stabilized the core, I'll start putting them back in. Gonna be a little bit 'vanilla' for a week or 2, but we'll come out of this leaner and meaner. :)
Thanks again!
:asian:
JDenz 12-15-2002, 02:20 AM thanks for your work guys
Rich Parsons 12-15-2002, 02:44 AM Get some Sleep Bob
and then start bright and early next week pulling off the next miracle. I cannot have you getting soft and all.
BTW I hear Santa is looking for some help in the scheduling department.
Serious thanks for all the great work your team has done.
Rich
:asian:
Buffy 12-15-2002, 03:15 AM Damn!
Not lo2
You can't win them all ;)
Hollywood1340 12-15-2002, 03:47 AM In the words of one of our former advesarys:
"I fear all we have done is awakened a sleeping giant....."
Cheers and good work Mr. Hubbard!
MT LIVES!! REMEMBER THE ALAMO!!!
It's been a rough week. Good work and get some sleep. Thanks for all the work to bring the site back up.
Mike Clarke 12-15-2002, 06:23 AM Ah, I'm reminded of the first series of Star Trek.
The Enterprise alone in space
BANG!
"What was that?
"Clingons of the starboard bow Capt."
"Scotty, Give me all you've got."
"The warp core's off line Capt, it's the dilithium crystals, there all gone."
"Jim."
"Steady bones, we've been in tighter situations than this."
"But Jim."
"Loosing your nerve now will serve no purpose doctor, it's not logical."
"You cold blooded pointy eared Vulcan, what would you know about emotions?"
And so the Enterprise worked it's way through yet another disaster, safe for one more week.
You guys at Martial Talk have the same spirit, and thank god you do. I hope you get everything up and running as before [if not better] soon.
Thanks for all your hard work and efforts above and beyond everyone. Take a bow.
Mike.
yilisifu 12-15-2002, 06:26 AM You have done a wonderful job! I enjoy this forum very much. Thank you for being so dedicated and for making the sacrifices that you have made to get through this difficult time.
TkdWarrior 12-15-2002, 08:42 AM cool job kaith..
i was really missing MT ...
-TkdWarrior-
Kenpomachine 12-15-2002, 08:53 AM Yeah, I missed you too. Glad your back again and keep up the good work :)
:asian:
RCastillo 12-15-2002, 12:20 PM Originally posted by Hollywood1340
In the words of one of our former advesarys:
"I fear all we have done is awakened a sleeping giant....."
Cheers and good work Mr. Hubbard!
MT LIVES!! REMEMBER THE ALAMO!!!
Oh great, John Wayne is making a comecack!:eek: :rofl:
tshadowchaser 12-15-2002, 12:31 PM Let me add my words of A job well done. Hope you get some sleep and rest. Thanks for getting the site back up as fast as you did.
Shadow:asian:
Bob Hubbard 12-15-2002, 12:34 PM Another short update:
I've striped us down to 'factory specs', and things look to be stabilized. Please let me know if you encounter any problems.
We lost all posts, and member signups that were made between Dec 7th and Dec 12th. If you signed up during that time, please resign up.
I realize it'll take a while to get things back on track, but want to say again, thank you to everyone for their patience, understanding and well wishes. It is greatly apreciated.
I've got a couple more client sites to bring back up, and then can focus on putting all the neat tweaks back in here. :)
:asian:
Originally posted by Kaith Rustaz
I've got a couple more client sites to bring back up, and then can focus on putting all the neat tweaks back in here. :)
Well thanks for putting us up high on the list of sites to just
bring back up! The tweaks can wait, no one's going anywhere.
And don't over do it! "If you haven't got your health, you haven't
got anything" - Count Rogan, the 6 fingered man (Princess Bride)
JDenz 12-15-2002, 02:27 PM Thanks any word on who caused the outage?
The 14th Style 12-15-2002, 02:33 PM Great job! Thanks for all the hard work!
The 14th Style
Bob Hubbard 12-15-2002, 02:37 PM The cause is still under investigation...
Short version is that it took out 2 hard drives and the backups, wiping out all of our data. We're leaning towards a controler failure at the moment.
JDenz 12-15-2002, 03:22 PM cool keep us updated
Michael Billings 12-15-2002, 07:27 PM Kudos to all who helped bring you back up.
-Michael
white belt 12-15-2002, 08:10 PM Thanks for your hard work. Any chance that you had sabatoge fom outside? Some people have a hard time with truth.
white belt
KenpoGirl 12-16-2002, 12:06 AM Well I agree it was a very tough week. I was going through major withdrawal. Sadly I must have checked the site a dozen times a day to see if it was back on line.
I am Extremely impressed that you got us back chatting so soon. But as I said on ICQ Bob, don't worry about us. Worry about your paying customers, and worry about you, don't kill yourself. We'll wait for you to get us back up and running full power again.
But that being said, I am very delighted to be back on here chatting again. :D
Dot
Edit: PHEW!!!! I just checked after posting this message. I'm still a Black Belt. I was worried there for a minute thought I might have lost too many posts. :D
Bob Hubbard 12-16-2002, 12:36 AM I managed to get some sleep last night. :)
Got almost all my clients back up and running, have about 5 left currently that need extensive rebuilding, and then I'm catching my breath for a day, n digging back in.
No clue on what actually happened. We suspect drive failure, however the investigations still ongoing. Will keep folks posted once we know something definate.
:asian:
JDenz 12-16-2002, 01:07 AM Safe to say you won't be dropping in the near future?
Cliarlaoch 12-16-2002, 03:16 AM Good job getting the place back up and running, and sorry to hear it happened in the first place, Kaith. Please express my support to the other admins. You guys rock. All the best on getting things back up and running.
Until then, I'll patiently wait and watch as we come back to life, and get some sleep, guys! :)
Clairlaoch
JDenz 12-16-2002, 05:15 AM It would be diffrent if you slept regularly
rachel 12-16-2002, 06:48 PM Thank you for the hard work, Bob! get some much needed sleep.
cali_tkdbruin 12-17-2002, 01:28 AM Excellent job guys! Great to have you back on-line. :asian:
RyuShiKan 12-18-2002, 08:12 AM bummer :erg:
JDenz 12-18-2002, 12:47 PM OKAY WAKE UP. lol just kidding
Pappy Geo 12-18-2002, 03:09 PM Bob,
Thanks for all your effort and professionalism, you have built the most userfriendly forum that I have surfed. If you were in the Northwest I would have you as webmaster!
Geo Hoover
The Opal Dragon 12-19-2002, 01:36 PM My link saved under favorites was broken but I tried getting here under Google search for the first time today and it worked. It was a big surprise to find out what happened! Thanks to everyone who has been working SO hard and doing such a wonderful job to get this site back up and running. I've REALLY missed it this last week and I'm happy to know it's working again!
Robyn :asian:
Wes Idol 12-19-2002, 03:39 PM Kaith Rustaz,
I wish you well, regarding your newest challenge. When electronics work, they feel like magic...when they don't, they feel like a curse. If there is anything I can do, please don't hesitate to ask.
Sincerely,
Wes Idol, HI
UKS
http://www.uks-kenpo.com
Just wanted to chime in: Good work in getting the site up and running!!!
/yari
JDenz 12-21-2002, 12:12 AM Any one that wants to help send 1000 dollars to Kaith Rustaz he will send you back a button and a thank you card lol.
RCastillo 12-21-2002, 12:16 AM Originally posted by JDenz
Any one that wants to help send 1000 dollars to Kaith Rustaz he will send you back a button and a thank you card lol.
I think organized crime has just made it's way onto Martial Talk:eek:
JDenz 12-21-2002, 12:20 AM lol no good politics
RCastillo 12-21-2002, 12:22 AM Originally posted by JDenz
lol no good politics
Which reminds me, there are still some members of the so called "Families" still up there in the Buffalo area! I know about you guys!:eek:
JDenz 12-21-2002, 12:28 AM lol a few and just cause one is about 2 minutes away from my house doesn't mean I am a gangster lol.
RCastillo 12-21-2002, 12:38 AM Originally posted by JDenz
lol a few and just cause one is about 2 minutes away from my house doesn't mean I am a gangster lol.
Ya know, the more I read, the hair on the back of my neck is raising up. Here's why:
1. Where your name is, it says, "White belt for life." Like Tony Soprano said, Once you get in this family, there ain't no getting out." The White belt for life handle makes me feel uneasy!
2. You're from upper NY state, and that's scary enough already! (No offense)
3. You seem to have alot of postings here on MT. I get the feeling you're "moving" in here, to set up camp, PERMANENTLY!
Look, I'm just a poor, underpaid school teacher from Texas, a servant of the state, as it were. look upon me kindly, pleaaaaaaaaase!:wah:
All in good fun!:D
JDenz 12-21-2002, 12:46 AM 1) lol
2) Ya I have lived here all my life to. Not only that I think our weather makes us tough. Walking through 4 feet of snow in 20 below wind chill makes you tough. Then in the summer the 90 degree 90 percent hummidity lol killer.
3) I am moving in you all better watch out.
4) My win in MMA came agianst a Tracy Karate guy. lol
5) We drove to Texas in 18 hours to the Airligton area so don't get any funny ideas ;=)
RCastillo 12-21-2002, 12:49 AM Originally posted by JDenz
1) lol
2) Ya I have lived here all my life to. Not only that I think our weather makes us tough. Walking through 4 feet of snow in 20 below wind chill makes you tough. Then in the summer the 90 degree 90 percent hummidity lol killer.
3) I am moving in you all better watch out.
4) My win in MMA came agianst a Tracy Karate guy. lol
5) We drove to Texas in 18 hours to the Airligton area so don't get any funny ideas ;=)
Well, as the saying goes, "The check's (protection money) in the mail!"
I want to live!:wah:
JDenz 12-21-2002, 12:54 AM lol good =-).
Bob Hubbard 12-21-2002, 01:01 AM You guys are a riot.....
so...
wheres my cut? :D
Rich Parsons 12-21-2002, 01:02 AM Originally posted by Kaith Rustaz
You guys are a riot.....
so...
wheres my cut? :D
Kaith,
I never knew you were one of the enforcers or cleanup guys or insert special nema here, guys for one of the families?
Where's My Cut? THat is at least two meanings I know of.
BTW I like the new main page look.
Rich
:D
JDenz 12-21-2002, 01:12 AM i am trying to collect it from him Kaith. I mean we need to PROTECT him from viruses and whatnot
Originally posted by RCastillo
I think organized crime has just made it's way onto Martial Talk:eek:
Are we going to have That discussion again?
RCastillo 12-21-2002, 11:04 AM Originally posted by Kaith Rustaz
You guys are a riot.....
so...
wheres my cut? :D
Oh crud, I was right. :eek:
RCastillo 12-21-2002, 11:06 AM Originally posted by Seig
Are we going to have That discussion again?
oh ,oh, the Hit Man has just arrived!:eek:
JDenz 12-21-2002, 10:57 PM Man your not having a good weeken huh?
RCastillo 12-22-2002, 01:01 AM Originally posted by JDenz
Man your not having a good weeken huh?
Around here, it's very rare!:(
JDenz 12-22-2002, 01:40 AM Ya I know
RCastillo 12-22-2002, 02:01 AM Originally posted by JDenz
Ya I know
And now, I have to contend with the "Bunch from Buffalo", NY !:eek:
JDenz 12-22-2002, 02:09 AM lol
Before the outage, you had links to "mark all forums read" both
at the top, and bottom of the main forum page. I liked that. Any
possibility of bringing it back? Also the list of who's been on
throughout the day.
arnisador 12-31-2002, 01:29 AM I also miss the Mark All Forums Read, and the previous timeout length on posts being marked read.
Bob Hubbard 12-31-2002, 02:15 AM There is a major update to the software due out very soon. Most of the old mods aren't compatable with it. Things are starting to calm down now, so I can hopefully take a day and evaluate the mods and put some back in. I miss most of those features myself.
Dragon-Slayer 04-29-2003, 07:22 AM Looks like you got things back together pretty good
JDenz 04-30-2003, 02:08 PM lol Till the next time
Gary5000 04-29-2004, 11:16 PM Yes, "Till the next time"
THX
|
|