# Site Crash 8-25-2009 - Data Lost.



## Bob Hubbard

My attempt tonight to import another site into MT in an attempt to preserve the other site and expand MT failed with almost catastrophic results. Despite taking the precaution of having last nights backup, in addition to 2 more current backups, all were somehow unrestorable,  As a result, the most recent restorable database backup is from August 18th.  All data in between is apparently lost. All posts, threads, pms, rep, game scores, and new registrations.

I deeply apologize for this and assure everyone that I am taking extra precautions to ensure that this does not happen again.


----------



## Carol

Bob, 

In my line of work I have seen first-hand occasions where tested procedures, tested migrations, and proper backups all took place.  The targeted system was healthy and running without errors.  And somehow the migration had catastrophic results. Things like this don't usually happen.  But when they do, the results are disastrous. 

Its a horrible feeling when you know you've done everything right, yet the application won't start up, the minutes of downtime start becoming hours, and you have to wake up every possible resource to pull them in to resuscitating anything that can be saved.  And the undercurrent to it all is that you don't have hope.  You know you are looking at a lost cause.  You know the fault killed off everything usable.  The only possibility is finding the fault in a system that showed no signs of trouble, and fixing it.  Or (proverbially) scraping out the corruption entirely and seeing if you have enough application structure left to still run something....anything.    Its not your fault that it happens, and the customer likely understands that its not you're fault, but you're still the one at the switch that has to put humpty dumpty back together again after he seems smashed to smithereens.

I'm sorry the site merge didn't work, but you deserve a huge round of applause for all you did to bring MT back, and to keep our years of contributions from being lost, permanently.  That was a huge effort and I am so glad you found a way to make it work.  

Thank you SO much! :asian:


----------



## Bob Hubbard

I've been in a "oh ****" state for 3 hrs. The admin at the data center is as stumped as I am that none of the backups would import.  I'm going to get some sleep and spend tomorrow going through the last weeks emails to try and repair some of the damage.


----------



## Makalakumu

Good luck, Bob.  I've got faith in you!


----------



## Bruno@MT

I know fully well the sinking feeling of despair that you speak of.
Have a night's sleep. Things will look better tomorrow. And if they don't, well, at least you are less tired


----------



## Danjo

Wow. I have to piss off everyone all over again!


----------



## Sukerkin

So that's what happened! I was just looking for an appropriate place to note that a week or so of the sites history seemed to have gone missing; this included 'Rep' and 'Thanks' so I figured that something major had gone awry with the site merge.

Deeply sorry to hear of the catastrophy and the data loss is a particular wrench given that I'd just started to become more active again here (those pleasant 'nods' from my fellow members will be missed too but at least they still exist in *my* memory).


----------



## Decker

Murphy strikes again.

All the best in fixing the problem. I feel your pain.


----------



## morph4me

I was wondering what happened. Thanks for keeping everyone in the loop, Bob. Good Luck.


----------



## Brian R. VanCise

I just went through a data crash at my work.  It was brutal so I understand completely.


----------



## arnisador

A lost week isn't so bad on the scale of a site that's now been here for, what, 8 years? Good luck with the remaining details.


----------



## celtic_crippler




----------



## Joab

Bob Hubbard said:


> My attempt tonight to import another site into MT in an attempt to preserve the other site and expand MT failed with almost catastrophic results. Despite taking the precaution of having last nights backup, in addition to 2 more current backups, all were somehow unrestorable, As a result, the most recent restorable database backup is from August 18th. All data in between is apparently lost. All posts, threads, pms, rep, game scores, and new registrations.
> 
> I deeply apologize for this and assure everyone that I am taking extra precautions to ensure that this does not happen again.


 
Such things happen, nobody is perfect, we'll get by without all of that.


----------



## AceHBK

Thanks for letting us know.  Well I guess it could have been worse so I am glad you were able to save what you could.


I just thought no one liked what I had to say..


----------



## mwd0818

Danjo said:


> Wow. I have to piss off everyone all over again!



Because the last week you've been doing something different than the last five???

Don't worry Danjo, I think we have enough of your posts to keep certain people pissed off for a while . . . 

In the meantime, I'm going to go back and reread the Kata versus Sparring threads . . . 


Bob - not a problem - we are all thankful for your time and effort, and sometimes things happen.  We will expect you to be demoted from Senior Great Supreme Forum Grandmaster down to just Grand Pooba for a week or so though . . .


----------



## Jade Tigress

What they all said Bob. :asian: We'll live.


----------



## Xue Sheng

WHAT!!!! 

Everything I wrote is gone :tantrum:

Ahhh just kidding, as you well know in the wonderful world of IT things can and will go wrong....castropically... it is just all part of the game.

It does answer my question as to what the heck happened

Besides 

:lisafault:

But Bob, you do of course realize that since I am an IT person I now am required to harass you mercilessly about this&#8230;.even though in all honesty it can be chalked up to&#8230;stuff happens :EG:


----------



## zepedawingchun

Just think, we get to do the last 7 days all over again, and this time make corrections and do it better.  Very seldom do we get to go back and start all over again.


----------



## Bob Hubbard

I'm trying to recover the last week, but it's not likely unfortunately.


----------



## FearlessFreep

You mean you lost my witty retort in the US Politics forum?  You all would've been amazed.  It was devastating and debilitating; a truly once in a life time combination of wise insight and cunning sarcasm with unassailable logic and uncanny accuracy.

*sigh*  well everyone can just bow and submit to my presence anyway.  Trust me, you would've known I deserve it had the post not been losy


----------



## Bob Hubbard

We're looking at the loss of about 3,000 posts and 40 new registrations. 10-15 of those were spam bots.  I'm trying to re-enter the legit registrations today.


----------



## Rich Parsons

Bob,

No Good Deed Goes Unpunished. Trying to save data and have regular backups and be a good host and try to give people a product they can use and be happy with. 

Shame on you for having a care for customer service. 


I really understand the lost of data and backups not working. That happened to me this summer as well. If I had not been diligent of having certain data stored on multiple formats I may have been in a serious spot. Instead I only lost a couple of months which I could reconstruct. (* Note: a couple of months might be a normal day for this site *)


So, relax and continue the good work and we will all sing your and your staffs' praise in the end.


----------



## Xue Sheng

Network back up is a VERY good thing but from time to time one must expect

:redeme:


----------



## Bruno@MT

FearlessFreep said:


> You mean you lost my witty retort in the US Politics forum?  You all would've been amazed.  It was devastating and debilitating; a truly once in a life time combination of wise insight and cunning sarcasm with unassailable logic and uncanny accuracy.
> 
> *sigh*  well everyone can just bow and submit to my presence anyway.  Trust me, you would've known I deserve it had the post not been losy



Reminds me of a dilbert comic, where he is giving a presentation to some customers.

_Our product isn't finished.
If it was, you'd see something here; here; and sometimes here.
And then you'd say 'I gotta get me some of that'.
Any questions?_


----------



## Bruno@MT

I just checked, and there's 6 rep points I won't see again in a hurry.
then again, there are worse things than losing a bit of online data.  

Don't sweat it Bob. Compared to other things, this doesn't even register as 'annoying' in the grand scheme of things. I appreciate everything you and the site staff do every day to make this site work.


----------



## Bob Hubbard

I want to thank everyone for their understanding and support while I recover from what could have been very ugly.

I'm out of rum again and days like today are not ones where you want to be out of rum.


----------



## ATC

Sorry to hear that Bob. Being in IT myself (Sys Admin to Director) I feel your pain. I have been down you road a couple of times myself. I for one can actually empathize.


----------



## terryl965

So much for all my wonderful post, well do I have to do twice as much to get where I was or three times as much so I can stay where Ia m at? These are the questions that keep me up at night.


----------



## celtic_crippler

Bob Hubbard said:


> We're looking at the loss of about 3,000 posts and 40 new registrations. 10-15 of those were spam bots. I'm trying to re-enter the legit registrations today.


 
Bless yer heart. I feel for ya! 



Bob Hubbard said:


> I want to thank everyone for their understanding and support while I recover from what could have been very ugly.
> 
> I'm out of rum again and days like today are not ones where you want to be out of rum.


 
Now I _really_ feel for ya!!


----------



## Xue Sheng

Bob Hubbard said:


> I want to thank everyone for their understanding and support while I recover from what could have been very ugly.
> 
> I'm out of rum again and days like today are not ones where you want to be out of rum.


 
 OUT OF RUM!!!!

The HECK with the website man...GO GET MORE RUM!!!!


----------



## KELLYG

BOB,

Thanks for letting us know what the deal was.   I thought that I had been talking to my self there for a while.


----------



## Hawke

Thanks for the heads up.


----------



## arnisador

Let's all try to help Kaith and the site recover by posting a week's worth of stuff in the LPT tonight!


----------



## AceHBK

Great, we lose all those posts but somehow Clark Kent manages to make it to another day....  

Thanks for the hard work Bob.


----------



## Bob Hubbard

arnisador said:


> Let's all try to help Kaith and the site recover by posting a week's worth of stuff in the LPT tonight!


I have reason to believe that part of the import problem was the miles long LPT's from the other site. I still can't delete one of them from there.


----------



## blindsage

Thanks for all your hard work Bob.


----------



## arnisador

Bob Hubbard said:


> I have reason to believe that part of the import problem was the miles long LPT's from the other site. I still can't delete one of them from there.



Eh, I think I was part of that problem... :xtrmshock


----------



## Bob Hubbard

no, different one, in an unviewable archive area there.


----------



## arnisador

If it's unviewable, how do you know it's there?


----------



## Bob Hubbard

Because I'm a Gawd.    I'm now deleting it, 400 posts at a time (software limit it seems).  only 25,000 or so more to go.........

Considering MTs 2 old ones are even larger.....ugh.


----------



## LoneRider

So what does this mean? Will we have our old posts from the pre-crash back?


----------



## Bob Hubbard

no.  All data from 8/18 through 12am 8/25 is lost.


----------



## Flea

Boy, I'm glad I found this thread.  I noticed that my Systema Breathing poem was gone and I was slapping my forehead.  What, the mods removed it?  Who would complain about _poetry?!?_   :uhyeah:

That said Bob, I want to echo everyone's appreciation for all you do for MY, especially over the past few days.


----------



## Darksoul

Bob,

-I've been a member since '04. I know what keeps me coming back here to MT. Its all the possibilities for interesting conversation and learning. What was lost really is just a drop in the bucket. I say don't worry about it. Process what happened and move on. Live and learn. Its like the cup of knowledge being bumped and some of it spilling out. Now there's room for more.

Thankyou,


Andrew


----------



## reedk

wow was everything lost for good then? i was looking for topics to read to better my knowledge of the sport. i'll check back again once new threads are started then. 

thanks

mma fanatic

http://www.hawaiiufc.com


----------

