# Major Outage Hits MartialTalk



## Bob Hubbard (Dec 14, 2002)

On Wednesday night, at about 2am, we were hit with an extremely nasty surprise.  

We were blown off the map...not just us, but our entire server was utterly and completely trashed.  While the cause is under investigation, the results are evident.  We suffered a complete and total data loss.  All information on the server was irrepairibly lost.

I've been working non stop since then to bring over 150 web sites back up, many without current backups.

Myself, my staff, and many of my associates have been up with minimal time for sleep since then.  I've had about 8 hours sleep myself since early thursday morning.  To say I need a nap is an understatement.

So, whats the situation with MartialTalk?

Heres the straight shot.  
We lost about a weeks worth of posts and sign ups.
There are several glitches that are still being worked out.
A few of our modifications are working halfway, and causing some display issues.

You will all notice the modified address...we are now at martialtalk.com/forum.  This is for 2 reasons...1 is it let me work on bringing things back up without interuption.  The other is that in the future we will be greatly expanding the scope and mission of MartialTalk.  

I want to thank all of our members and supporters for their patience during this time.  I also want to extend my heartfelt gratitude to all those who offered their help, or well wishes while we worked on things, and to the support staff at our datacenter who went above and beyond to help us out.

We are not through...we will rise from the ashes and become better than we were.

Peace.


----------



## arnisador (Dec 14, 2002)

Congratulations and thanks to Bob Hubbard and the others who worked so hard on resurrecting MartialTalk. Thanks to a backup strategy, hard work, and luck, only about a week's worth of posts have been lost.


----------



## tunetigress (Dec 14, 2002)

Awesome job you guys for getting things  working working so well again!  I don't know how you do it, but I'm sure I speak for us all in saying how much your efforts are appreciated in making posting possible again.  :asian: :asian: :asian:


----------



## Elfan (Dec 14, 2002)

Thanks guys!

Good luck with getting the rest of the stuf back together.


----------



## Bob Hubbard (Dec 14, 2002)

Short update, then im getting some sleep...

I've already gotten notification of a number of hiccups, and ran into a few myself.  We litterally crowbared this place back up, and it looks like a few patches are leaking.

To this end, I'm stripping out all of the mods I've put in and taking us back to 'factory spec' for a little while.  Once we've stabilized the core, I'll start putting them back in.  Gonna be a little bit 'vanilla' for a week or 2, but we'll come out of this leaner and meaner. 

Thanks again!
:asian:


----------



## JDenz (Dec 15, 2002)

thanks for your work guys


----------



## ace (Dec 15, 2002)




----------



## Rich Parsons (Dec 15, 2002)

Get some Sleep Bob 
and then start bright and early next week pulling off the next miracle. I cannot have you getting soft and all.

BTW I hear Santa is looking for some help in the scheduling department.

Serious thanks for all the great work your team has done.

Rich
:asian:


----------



## Buffy (Dec 15, 2002)

Damn!

Not lo2

You can't win them all


----------



## Hollywood1340 (Dec 15, 2002)

In the words of one of our former advesarys:
"I fear all we have done is awakened a sleeping giant....."
Cheers and good work Mr. Hubbard!
MT LIVES!! REMEMBER THE ALAMO!!!


----------



## bart (Dec 15, 2002)

It's been a rough week. Good work and get some sleep. Thanks for all the work to bring the site back up.


----------



## Mike Clarke (Dec 15, 2002)

Ah, I'm reminded of the first series of Star Trek.
The Enterprise alone in space
BANG!
"What was that?
"Clingons of the starboard bow Capt."
"Scotty, Give me all you've got."
"The warp core's off line Capt, it's the dilithium crystals, there all gone."
"Jim."
"Steady bones, we've been in tighter situations than this."
"But Jim."
"Loosing your nerve now will serve no purpose doctor, it's not logical."
"You cold blooded pointy eared Vulcan, what would you know about emotions?"

And so the Enterprise worked it's way through yet another disaster, safe for one more week.

You guys at Martial Talk have the same spirit, and thank god you do. I hope you get everything up and running as before [if not better] soon.
Thanks for all your hard work and efforts above and beyond everyone.  Take a bow.

Mike.


----------



## yilisifu (Dec 15, 2002)

You have done a wonderful job!  I enjoy this forum very much.  Thank you for being so dedicated and for making the sacrifices that you have made to get through this difficult time.


----------



## TkdWarrior (Dec 15, 2002)

cool job kaith..
i was really missing MT ...
-TkdWarrior-


----------



## Kenpomachine (Dec 15, 2002)

Yeah, I missed you too. Glad your back again and keep up the good work 
:asian:


----------



## RCastillo (Dec 15, 2002)

> _Originally posted by Hollywood1340 _
> *In the words of one of our former advesarys:
> "I fear all we have done is awakened a sleeping giant....."
> Cheers and good work Mr. Hubbard!
> MT LIVES!! REMEMBER THE ALAMO!!! *



Oh great, John Wayne is making a comecack! :rofl:


----------



## tshadowchaser (Dec 15, 2002)

Let me add my words of A job well done.  Hope you get some sleep and rest.   Thanks for getting the site back up as fast as you did.
Shadow:asian:


----------



## Bob Hubbard (Dec 15, 2002)

Another short update:
I've striped us down to 'factory specs', and things look to be stabilized.  Please let me know if you encounter any problems.

We lost all posts, and member signups that were made between Dec 7th and Dec 12th.  If you signed up during that time, please resign up.  

I realize it'll take a while to get things back on track, but want to say again, thank you to everyone for their patience, understanding and well wishes.  It is greatly apreciated.

I've got a couple more client sites to bring back up, and then can focus on putting all the neat tweaks back in here. 


:asian:


----------



## Kirk (Dec 15, 2002)

> _Originally posted by Kaith Rustaz _
> *
> I've got a couple more client sites to bring back up, and then can focus on putting all the neat tweaks back in here. *



Well thanks for putting us up high on the list of sites to just
bring back up!  The tweaks can wait, no one's going anywhere.

And don't over do it!  "If you haven't got your health, you haven't
got anything" - Count Rogan, the 6 fingered man (Princess Bride)


----------



## JDenz (Dec 15, 2002)

Thanks any word on who caused the outage?


----------



## The 14th Style (Dec 15, 2002)

Great job! Thanks for all the hard work!
The 14th Style


----------



## Bob Hubbard (Dec 15, 2002)

The cause is still under investigation...

Short version is that it took out 2 hard drives and the backups, wiping out all of our data.  We're leaning towards a controler failure at the moment.


----------



## JDenz (Dec 15, 2002)

cool keep us updated


----------



## Michael Billings (Dec 15, 2002)

Kudos to all who helped bring you back up.  
-Michael


----------



## white belt (Dec 15, 2002)

Thanks for your hard work.  Any chance that you had sabatoge fom outside?  Some people have a hard time with truth.

white belt


----------



## KenpoGirl (Dec 15, 2002)

Well I agree it was a very tough week.  I was going through major withdrawal.  Sadly I must have checked the site a dozen times a day to see if it was back on line.

I am Extremely impressed that you got us back chatting so soon.  But as I said on ICQ Bob, don't worry about us.  Worry about your paying customers, and worry about you, don't kill yourself.  We'll wait for you to get us back up and running full power again.  

But that being said, I am very delighted to be back on here chatting again.  

Dot

Edit:   PHEW!!!!   I just checked after posting this message.  I'm still a Black Belt.  I was worried there for a minute thought I might have lost too many posts.


----------



## Bob Hubbard (Dec 15, 2002)

I managed to get some sleep last night. 
Got almost all my clients back up and running, have about 5 left currently that need extensive rebuilding, and then I'm catching my breath for a day, n digging back in.

No clue on what actually happened.  We suspect drive failure, however the investigations still ongoing.  Will keep folks posted once we know something definate.

:asian:


----------



## JDenz (Dec 15, 2002)

Safe to say you won't be dropping in the near future?


----------



## Cliarlaoch (Dec 16, 2002)

Good job getting the place back up and running, and sorry to hear it happened in the first place, Kaith. Please express my support to the other admins. You guys rock. All the best on getting things back up and running.

Until then, I'll patiently wait and watch as we come back to life, and get some sleep, guys! 

Clairlaoch


----------



## JDenz (Dec 16, 2002)

It would be diffrent if you slept regularly


----------



## rachel (Dec 16, 2002)

Thank you for the hard work, Bob! get some much needed sleep.


----------



## cali_tkdbruin (Dec 16, 2002)

Excellent job guys! Great to have you back on-line. :asian:


----------



## RyuShiKan (Dec 18, 2002)

bummer :erg:


----------



## JDenz (Dec 18, 2002)

OKAY WAKE UP.  lol just kidding


----------



## Pappy Geo (Dec 18, 2002)

Bob,

Thanks for all your effort and professionalism, you have built the most userfriendly forum that I have surfed. If you were in the Northwest I would have you as webmaster!

Geo Hoover


----------



## Aikikitty (Dec 19, 2002)

My link saved under favorites was broken but I tried getting here under Google search for the first time today and it worked.  It was a big surprise to find out what happened!    Thanks to everyone who has been working SO hard and doing such a wonderful job to get this site back up and running.  I've REALLY missed it this last week and I'm happy to know it's working again!

Robyn :asian:


----------



## Wes Idol (Dec 19, 2002)

Kaith Rustaz, 

I wish you well, regarding your newest challenge.  When electronics work, they feel like magic...when they don't, they feel like a curse.  If there is anything I can do, please don't hesitate to ask.

Sincerely,

Wes Idol, HI
UKS
http://www.uks-kenpo.com


----------



## Yari (Dec 20, 2002)

Just wanted to chime in: Good work in getting the site up and running!!!


/yari


----------



## JDenz (Dec 20, 2002)

Any one that wants to help send 1000 dollars to Kaith Rustaz he will send you back a button and a thank you card lol.


----------



## RCastillo (Dec 20, 2002)

> _Originally posted by JDenz _
> *Any one that wants to help send 1000 dollars to Kaith Rustaz he will send you back a button and a thank you card lol. *



I think organized crime has just made it's way onto Martial Talk


----------



## JDenz (Dec 20, 2002)

lol no good politics


----------



## RCastillo (Dec 20, 2002)

> _Originally posted by JDenz _
> *lol no good politics *



Which reminds me, there are still some members of the so called "Families" still up there in the Buffalo area! I know about you guys!


----------



## JDenz (Dec 20, 2002)

lol a few and just cause one is about 2 minutes away from my house doesn't mean I am a gangster lol.


----------



## RCastillo (Dec 20, 2002)

> _Originally posted by JDenz _
> *lol a few and just cause one is about 2 minutes away from my house doesn't mean I am a gangster lol. *



Ya know, the more I read, the hair on the back of my neck is raising up. Here's why:

1. Where your name is, it says, "White belt for life." Like Tony Soprano said, Once you get in this family, there ain't no getting out." The White belt for life handle makes me feel uneasy!

2. You're from upper NY state, and that's scary enough already! (No offense) 

3. You seem to have alot of postings here on MT. I get the feeling you're "moving" in here, to set up camp, PERMANENTLY!

Look, I'm just a poor, underpaid school teacher from Texas, a servant of the state, as it were. look upon me kindly, pleaaaaaaaaase!:wah: 

All in good fun!


----------



## JDenz (Dec 20, 2002)

1) lol 

2) Ya I have lived here all my life to.  Not only that I think our weather makes us tough.  Walking through 4 feet of snow in 20 below wind chill makes you tough.  Then in the summer the 90 degree 90 percent hummidity lol killer.

3) I am moving in you all better watch out.

4) My win in MMA came agianst a Tracy Karate guy. lol 

5) We drove to Texas in 18 hours to the Airligton area so don't get any funny ideas ;=)


----------



## RCastillo (Dec 20, 2002)

> _Originally posted by JDenz _
> *1) lol
> 
> 2) Ya I have lived here all my life to.  Not only that I think our weather makes us tough.  Walking through 4 feet of snow in 20 below wind chill makes you tough.  Then in the summer the 90 degree 90 percent hummidity lol killer.
> ...



Well, as the saying goes, "The check's (protection money) in the mail!"

I want to live!:wah:


----------



## JDenz (Dec 20, 2002)

lol good =-).


----------



## Bob Hubbard (Dec 20, 2002)

You guys are a riot.....

so...

wheres my cut?


----------



## Rich Parsons (Dec 20, 2002)

> _Originally posted by Kaith Rustaz _
> *You guys are a riot.....
> 
> so...
> ...




Kaith,

I never knew you were one of the enforcers or cleanup guys or insert special nema here, guys for one of the families?

Where's My Cut?  THat is at least two meanings I know of.


BTW I like the new main page look.

Rich


----------



## JDenz (Dec 20, 2002)

i am trying to collect it from him Kaith.  I mean we need to PROTECT him from viruses and whatnot


----------



## Seig (Dec 21, 2002)

> _Originally posted by RCastillo _
> *I think organized crime has just made it's way onto Martial Talk *


Are we going to have _*That*_ discussion again?


----------



## RCastillo (Dec 21, 2002)

> _Originally posted by Kaith Rustaz _
> *You guys are a riot.....
> 
> so...
> ...



Oh crud, I was right.


----------



## RCastillo (Dec 21, 2002)

> _Originally posted by Seig _
> *Are we going to have That discussion again? *



oh ,oh, the Hit Man has just arrived!


----------



## JDenz (Dec 21, 2002)

Man your not having a good weeken huh?


----------



## RCastillo (Dec 21, 2002)

> _Originally posted by JDenz _
> *Man your not having a good weeken huh? *



Around here, it's very rare!


----------



## JDenz (Dec 21, 2002)

Ya I know


----------



## RCastillo (Dec 22, 2002)

> _Originally posted by JDenz _
> *Ya I know *



And now, I have to contend with the "Bunch from Buffalo", NY !


----------



## JDenz (Dec 22, 2002)

lol


----------



## Kirk (Dec 30, 2002)

Before the outage, you had links to "mark all forums read" both
at the top, and bottom of the main forum page.  I liked that.  Any
possibility of bringing it back?  Also the list of who's been on 
throughout the day.


----------



## arnisador (Dec 30, 2002)

I also miss the Mark All Forums Read, and the previous timeout length on posts being marked read.


----------



## Bob Hubbard (Dec 31, 2002)

There is a major update to the software due out very soon.  Most of the old mods aren't compatable with it.    Things are starting to calm down now, so I can hopefully take a day and evaluate the mods and put some back in.  I miss most of those features myself.


----------



## Dragon-Slayer (Apr 29, 2003)

Looks like you got things back together pretty good


----------



## JDenz (Apr 30, 2003)

lol Till the next time


----------



## Gary5000 (Apr 29, 2004)

Yes, "Till the next time" 
THX


----------

