Aug 08 Outage

[Darkgear]

Info for those who wanna know what happened aug 27-sep5.

Aug 28th the forums started giving us these errors

: Table './flash30_FF1/smf1_sessions' was created with a different version of MySQL and cannot be read
/home/flash30/public_html/Sources/Load.php
2096

I figure its some kind of corruption so I check out the DBs and there are half a dozen tables locked and unreadable. I check the host network status page and they say all is OK.  So I start replacing files and rebuilding DBs and the corruption gets worse and worse.  I finally submit a trouble ticket to support on the 28th.  I ask them to restore a previous day backup from the 27th.  I get this reply.

It would be better for us to do this after we finish the current repairs. I'm not sure if you were alerted of this:

A flawed configuration of the cPanel on our shared servers has caused a malfunction in the Apache and PHP setups on them. We are currently working on this and it should be fixed soon. Please be patient during this time and all the problems should be fixed upon our completion.

After we finish this, we can safely restore your site.

So they did have a server configuration issue.   I tell them OK finish the repair then restore. They finish repairs to the cpanel and php config on the 30th and I'm nervous because they only keep backups for 3 days then they expire.  On the 1st they say the last backup they have is from aug 28th.  I tell them its OK to restore that one.  2 days later no restore and I get a phone call.  Tech says their backup system died and the manufacturer is overnighting them a part to fix it. Good news is that the 28th backup did not expire since the whole tape library went down.  Bad news is another day to wait.  Today(5th) another call to say the backup was fixed but the backup of our forum was incomplete.  Oh well.  So I just rebuilt the forum from scratch on another hosting site and painfully had to fix all the sql errors one by one manually.  Heh still only took a few hours to completely rebuild.  I guess I should have done that earlier.

Anyway, we're back. sorry for the downtime.

Best regards,
Randy

[Darkgear]

Forgot to say that in upgrading our forum we lost some features and gained some old ones(that were updated).  I also just changed the nameservers to the URL so it will be about 24 hours before all the routers catch up and access to this site is stable again.

The google member map is back
Arcade was updated

Still working on:
wiki
Lighthouse banner
profile comments
chess

Shop probably won't be back(no more clams)

Please let me know if you see anything amiss.

Randy

[Bullzeyebill]

Thanks Randy for your efforts.

Bill

[Lips]

Thanks for your hard work Randy   



.

[arewethereyetdad]

Randy, you da man!  Thanks a million for all the time and effort.  You saved the mothership!  Glad to be home again.     

[griff]

WELCOME BACK   mr Kotter

[Rebel]

All my clams are gone, I'm ruined.   

Good to see the site back up and running though.

[arewethereyetdad]

As mentioned earlier, super job, Randy.  Hope you can get the lighthouse banner back. 

[starfiretoo]

Thanks and Goodjob!

[Hodsta]

Cheers to you - thanks - this URL is nothing without you

[BraddahBill]

Randy you are da MAN!   




Bill

[regentag]

You're an archangel, Darkgear! 

[aikiman44]

Top notch work.
Good to be back.

[PoliceScannerMan]

tight work Randy, as usual.  Your behind the scenes work keep it going, we appreciate it!!

[nolan]

  Glad to see the forum back up! Great Job!