View Full Version : Major Outage Hits MartialTalk


Bob Hubbard
12-14-2002, 10:36 PM
On Wednesday night, at about 2am, we were hit with an extremely nasty surprise.

We were blown off the map...not just us, but our entire server was utterly and completely trashed. While the cause is under investigation, the results are evident. We suffered a complete and total data loss. All information on the server was irrepairibly lost.

I've been working non stop since then to bring over 150 web sites back up, many without current backups.

Myself, my staff, and many of my associates have been up with minimal time for sleep since then. I've had about 8 hours sleep myself since early thursday morning. To say I need a nap is an understatement.

So, whats the situation with MartialTalk?

Heres the straight shot.
We lost about a weeks worth of posts and sign ups.
There are several glitches that are still being worked out.
A few of our modifications are working halfway, and causing some display issues.

You will all notice the modified address...we are now at martialtalk.com/forum. This is for 2 reasons...1 is it let me work on bringing things back up without interuption. The other is that in the future we will be greatly expanding the scope and mission of MartialTalk.

I want to thank all of our members and supporters for their patience during this time. I also want to extend my heartfelt gratitude to all those who offered their help, or well wishes while we worked on things, and to the support staff at our datacenter who went above and beyond to help us out.

We are not through...we will rise from the ashes and become better than we were.

Peace.

arnisador
12-14-2002, 10:52 PM
Congratulations and thanks to Bob Hubbard and the others who worked so hard on resurrecting MartialTalk. Thanks to a backup strategy, hard work, and luck, only about a week's worth of posts have been lost.

tunetigress
12-14-2002, 11:52 PM
Awesome job you guys for getting things working working so well again! I don't know how you do it, but I'm sure I speak for us all in saying how much your efforts are appreciated in making posting possible again. :asian: :asian: :asian:

Elfan
12-14-2002, 11:52 PM
Thanks guys!

Good luck with getting the rest of the stuf back together.

Bob Hubbard
12-15-2002, 01:39 AM
Short update, then im getting some sleep...

I've already gotten notification of a number of hiccups, and ran into a few myself. We litterally crowbared this place back up, and it looks like a few patches are leaking.

To this end, I'm stripping out all of the mods I've put in and taking us back to 'factory spec' for a little while. Once we've stabilized the core, I'll start putting them back in. Gonna be a little bit 'vanilla' for a week or 2, but we'll come out of this leaner and meaner. :)

Thanks again!
:asian:

JDenz
12-15-2002, 02:20 AM
thanks for your work guys

ace
12-15-2002, 02:41 AM
:eek:

Rich Parsons
12-15-2002, 02:44 AM
Get some Sleep Bob
and then start bright and early next week pulling off the next miracle. I cannot have you getting soft and all.

BTW I hear Santa is looking for some help in the scheduling department.

Serious thanks for all the great work your team has done.

Rich
:asian:

Buffy
12-15-2002, 03:15 AM
Damn!

Not lo2

You can't win them all ;)

Hollywood1340
12-15-2002, 03:47 AM
In the words of one of our former advesarys:
"I fear all we have done is awakened a sleeping giant....."
Cheers and good work Mr. Hubbard!
MT LIVES!! REMEMBER THE ALAMO!!!

bart
12-15-2002, 04:32 AM
It's been a rough week. Good work and get some sleep. Thanks for all the work to bring the site back up.

Mike Clarke
12-15-2002, 06:23 AM
Ah, I'm reminded of the first series of Star Trek.
The Enterprise alone in space
BANG!
"What was that?
"Clingons of the starboard bow Capt."
"Scotty, Give me all you've got."
"The warp core's off line Capt, it's the dilithium crystals, there all gone."
"Jim."
"Steady bones, we've been in tighter situations than this."
"But Jim."
"Loosing your nerve now will serve no purpose doctor, it's not logical."
"You cold blooded pointy eared Vulcan, what would you know about emotions?"

And so the Enterprise worked it's way through yet another disaster, safe for one more week.

You guys at Martial Talk have the same spirit, and thank god you do. I hope you get everything up and running as before [if not better] soon.
Thanks for all your hard work and efforts above and beyond everyone. Take a bow.

Mike.

yilisifu
12-15-2002, 06:26 AM
You have done a wonderful job! I enjoy this forum very much. Thank you for being so dedicated and for making the sacrifices that you have made to get through this difficult time.

TkdWarrior
12-15-2002, 08:42 AM
cool job kaith..
i was really missing MT ...
-TkdWarrior-

Kenpomachine
12-15-2002, 08:53 AM
Yeah, I missed you too. Glad your back again and keep up the good work :)
:asian:

RCastillo
12-15-2002, 12:20 PM
Originally posted by Hollywood1340
In the words of one of our former advesarys:
"I fear all we have done is awakened a sleeping giant....."
Cheers and good work Mr. Hubbard!
MT LIVES!! REMEMBER THE ALAMO!!!

Oh great, John Wayne is making a comecack!:eek: :rofl:

tshadowchaser
12-15-2002, 12:31 PM
Let me add my words of A job well done. Hope you get some sleep and rest. Thanks for getting the site back up as fast as you did.
Shadow:asian:

Bob Hubbard
12-15-2002, 12:34 PM
Another short update:
I've striped us down to 'factory specs', and things look to be stabilized. Please let me know if you encounter any problems.

We lost all posts, and member signups that were made between Dec 7th and Dec 12th. If you signed up during that time, please resign up.

I realize it'll take a while to get things back on track, but want to say again, thank you to everyone for their patience, understanding and well wishes. It is greatly apreciated.

I've got a couple more client sites to bring back up, and then can focus on putting all the neat tweaks back in here. :)


:asian:

Kirk
12-15-2002, 01:06 PM
Originally posted by Kaith Rustaz

I've got a couple more client sites to bring back up, and then can focus on putting all the neat tweaks back in here. :)

Well thanks for putting us up high on the list of sites to just
bring back up! The tweaks can wait, no one's going anywhere.

And don't over do it! "If you haven't got your health, you haven't
got anything" - Count Rogan, the 6 fingered man (Princess Bride)

JDenz
12-15-2002, 02:27 PM
Thanks any word on who caused the outage?

The 14th Style
12-15-2002, 02:33 PM
Great job! Thanks for all the hard work!
The 14th Style

Bob Hubbard
12-15-2002, 02:37 PM
The cause is still under investigation...

Short version is that it took out 2 hard drives and the backups, wiping out all of our data. We're leaning towards a controler failure at the moment.

JDenz
12-15-2002, 03:22 PM
cool keep us updated

Michael Billings
12-15-2002, 07:27 PM
Kudos to all who helped bring you back up.
-Michael

white belt
12-15-2002, 08:10 PM
Thanks for your hard work. Any chance that you had sabatoge fom outside? Some people have a hard time with truth.

white belt

KenpoGirl
12-16-2002, 12:06 AM
Well I agree it was a very tough week. I was going through major withdrawal. Sadly I must have checked the site a dozen times a day to see if it was back on line.

I am Extremely impressed that you got us back chatting so soon. But as I said on ICQ Bob, don't worry about us. Worry about your paying customers, and worry about you, don't kill yourself. We'll wait for you to get us back up and running full power again.

But that being said, I am very delighted to be back on here chatting again. :D

Dot

Edit: PHEW!!!! I just checked after posting this message. I'm still a Black Belt. I was worried there for a minute thought I might have lost too many posts. :D

Bob Hubbard
12-16-2002, 12:36 AM
I managed to get some sleep last night. :)
Got almost all my clients back up and running, have about 5 left currently that need extensive rebuilding, and then I'm catching my breath for a day, n digging back in.

No clue on what actually happened. We suspect drive failure, however the investigations still ongoing. Will keep folks posted once we know something definate.

:asian:

JDenz
12-16-2002, 01:07 AM
Safe to say you won't be dropping in the near future?

Cliarlaoch
12-16-2002, 03:16 AM
Good job getting the place back up and running, and sorry to hear it happened in the first place, Kaith. Please express my support to the other admins. You guys rock. All the best on getting things back up and running.

Until then, I'll patiently wait and watch as we come back to life, and get some sleep, guys! :)

Clairlaoch

JDenz
12-16-2002, 05:15 AM
It would be diffrent if you slept regularly

rachel
12-16-2002, 06:48 PM
Thank you for the hard work, Bob! get some much needed sleep.

cali_tkdbruin
12-17-2002, 01:28 AM
Excellent job guys! Great to have you back on-line. :asian:

RyuShiKan
12-18-2002, 08:12 AM
bummer :erg:

JDenz
12-18-2002, 12:47 PM
OKAY WAKE UP. lol just kidding

Pappy Geo
12-18-2002, 03:09 PM
Bob,

Thanks for all your effort and professionalism, you have built the most userfriendly forum that I have surfed. If you were in the Northwest I would have you as webmaster!

Geo Hoover

The Opal Dragon
12-19-2002, 01:36 PM
My link saved under favorites was broken but I tried getting here under Google search for the first time today and it worked. It was a big surprise to find out what happened! Thanks to everyone who has been working SO hard and doing such a wonderful job to get this site back up and running. I've REALLY missed it this last week and I'm happy to know it's working again!

Robyn :asian:

Wes Idol
12-19-2002, 03:39 PM
Kaith Rustaz,

I wish you well, regarding your newest challenge. When electronics work, they feel like magic...when they don't, they feel like a curse. If there is anything I can do, please don't hesitate to ask.

Sincerely,

Wes Idol, HI
UKS
http://www.uks-kenpo.com

Yari
12-20-2002, 05:51 AM
Just wanted to chime in: Good work in getting the site up and running!!!


/yari

JDenz
12-21-2002, 12:12 AM
Any one that wants to help send 1000 dollars to Kaith Rustaz he will send you back a button and a thank you card lol.

RCastillo
12-21-2002, 12:16 AM
Originally posted by JDenz
Any one that wants to help send 1000 dollars to Kaith Rustaz he will send you back a button and a thank you card lol.

I think organized crime has just made it's way onto Martial Talk:eek:

JDenz
12-21-2002, 12:20 AM
lol no good politics

RCastillo
12-21-2002, 12:22 AM
Originally posted by JDenz
lol no good politics

Which reminds me, there are still some members of the so called "Families" still up there in the Buffalo area! I know about you guys!:eek:

JDenz
12-21-2002, 12:28 AM
lol a few and just cause one is about 2 minutes away from my house doesn't mean I am a gangster lol.

RCastillo
12-21-2002, 12:38 AM
Originally posted by JDenz
lol a few and just cause one is about 2 minutes away from my house doesn't mean I am a gangster lol.

Ya know, the more I read, the hair on the back of my neck is raising up. Here's why:

1. Where your name is, it says, "White belt for life." Like Tony Soprano said, Once you get in this family, there ain't no getting out." The White belt for life handle makes me feel uneasy!

2. You're from upper NY state, and that's scary enough already! (No offense)

3. You seem to have alot of postings here on MT. I get the feeling you're "moving" in here, to set up camp, PERMANENTLY!

Look, I'm just a poor, underpaid school teacher from Texas, a servant of the state, as it were. look upon me kindly, pleaaaaaaaaase!:wah:

All in good fun!:D

JDenz
12-21-2002, 12:46 AM
1) lol

2) Ya I have lived here all my life to. Not only that I think our weather makes us tough. Walking through 4 feet of snow in 20 below wind chill makes you tough. Then in the summer the 90 degree 90 percent hummidity lol killer.

3) I am moving in you all better watch out.

4) My win in MMA came agianst a Tracy Karate guy. lol

5) We drove to Texas in 18 hours to the Airligton area so don't get any funny ideas ;=)

RCastillo
12-21-2002, 12:49 AM
Originally posted by JDenz
1) lol

2) Ya I have lived here all my life to. Not only that I think our weather makes us tough. Walking through 4 feet of snow in 20 below wind chill makes you tough. Then in the summer the 90 degree 90 percent hummidity lol killer.

3) I am moving in you all better watch out.

4) My win in MMA came agianst a Tracy Karate guy. lol

5) We drove to Texas in 18 hours to the Airligton area so don't get any funny ideas ;=)

Well, as the saying goes, "The check's (protection money) in the mail!"

I want to live!:wah:

JDenz
12-21-2002, 12:54 AM
lol good =-).

Bob Hubbard
12-21-2002, 01:01 AM
You guys are a riot.....

so...

wheres my cut? :D

Rich Parsons
12-21-2002, 01:02 AM
Originally posted by Kaith Rustaz
You guys are a riot.....

so...

wheres my cut? :D


Kaith,

I never knew you were one of the enforcers or cleanup guys or insert special nema here, guys for one of the families?

Where's My Cut? THat is at least two meanings I know of.


BTW I like the new main page look.

Rich
:D

JDenz
12-21-2002, 01:12 AM
i am trying to collect it from him Kaith. I mean we need to PROTECT him from viruses and whatnot

Seig
12-21-2002, 06:52 AM
Originally posted by RCastillo
I think organized crime has just made it's way onto Martial Talk:eek:
Are we going to have That discussion again?

RCastillo
12-21-2002, 11:04 AM
Originally posted by Kaith Rustaz
You guys are a riot.....

so...

wheres my cut? :D

Oh crud, I was right. :eek:

RCastillo
12-21-2002, 11:06 AM
Originally posted by Seig
Are we going to have That discussion again?

oh ,oh, the Hit Man has just arrived!:eek:

JDenz
12-21-2002, 10:57 PM
Man your not having a good weeken huh?

RCastillo
12-22-2002, 01:01 AM
Originally posted by JDenz
Man your not having a good weeken huh?

Around here, it's very rare!:(

JDenz
12-22-2002, 01:40 AM
Ya I know

RCastillo
12-22-2002, 02:01 AM
Originally posted by JDenz
Ya I know

And now, I have to contend with the "Bunch from Buffalo", NY !:eek:

JDenz
12-22-2002, 02:09 AM
lol

Kirk
12-31-2002, 01:25 AM
Before the outage, you had links to "mark all forums read" both
at the top, and bottom of the main forum page. I liked that. Any
possibility of bringing it back? Also the list of who's been on
throughout the day.

arnisador
12-31-2002, 01:29 AM
I also miss the Mark All Forums Read, and the previous timeout length on posts being marked read.

Bob Hubbard
12-31-2002, 02:15 AM
There is a major update to the software due out very soon. Most of the old mods aren't compatable with it. Things are starting to calm down now, so I can hopefully take a day and evaluate the mods and put some back in. I miss most of those features myself.

Dragon-Slayer
04-29-2003, 07:22 AM
Looks like you got things back together pretty good

JDenz
04-30-2003, 02:08 PM
lol Till the next time

Gary5000
04-29-2004, 11:16 PM
Yes, "Till the next time"
THX