SERVER OUTAGE / FORUMS UPGRADE: Read this first before you POST QUESTIONS

Spark

Spark

HPIC - Hatas gonna Hate
Staff member
Administrator
Super Mod
Moderator
Joined
Oct 2, 1998
Messages
15,223
Ok, to recap what happened:

2300HRS 20MAR04 - I called our server techs to backup our database prior to my upgrading the forums software. I proceeded when the backup was complete. Unfortunately, our server MySQL backend wasn't running the necessary current version, causing an upgrade failure. We attempted to restore the backup, and found out that the backup (quite frankly) didn't.

0200HRS 21MAR04 - After 3 hours of trying to fix the database, we decide to restore from an older backup (when the new RAM was installed).

2000HRS 21MAR04 - Forums come back up. Over the next 2 days, I rebuild styles, reactivate members, get search engine rebuilt, etc etc etc. Start long road to recovery.

0348HRS 24MAR04 - Forums database server suffers a massive failure, wiping out the entire RAID array, and all on server backups.

0900HRS 24MAR04 - We discover the server went down and start recovery. All available data is transferred to other boxes in the server farm for later analysis and recovery. Replacement server ordered.

1200HRS 25MAR04 - Replacement server arrives and is taken to the datacenter. We begin recovery process - operating system installed, RAID Array built, etc.

1800HRS 25MAR04 - Recovered data is discovered to be entirely corrupted. We start back again with our 20FEB backup and begin multi-hour upgrade & rebuild process

2400HRS 25MAR04 - Success. Forums come online. Forums database is backed up again on separate server so that if there is another failure, we don't have to waste another 6 hours on upgrading.

Our top priority now is restoring access to the site, which should have happened by the time you read this. At this time, SEARCHES will be disabled until we can fully rebuild the search index. That is a background process and should take no more than 48 hours.

Our next step will be to restore all paying members to their appropriate status levels, which will take until 1800HRS tomorrow (26MAR04) to complete. This will be followed by restoring custom titles.

Following this, work will begin on customization of viewing styles to get the forums looking more like "normal". Work will begin on new forums that will be added to the site. Password protected forums and higher membership levels will be enacted. Additional features will come online.

This has been an extremely trying time for us. We appreciate your patience through the outage. At this time, here is the current status of the forums:

Forums are online, however, all data created after 20FEB04 has been lost and is irretrievable - posts, memberships, attachments, etc.
The original corrupted database which we were hoping to recover was wiped out when the RAID Array failed on the database server.

All members who joined after 20FEB04 will have to rejoin AGAIN. As previously stated, we will be reinstating appropriate memberships shortly.

I had a very long talk with our tech support about our backup options and we now have a much better recovery plan in effect. Our new database server is also better setup for disaster recovery.

You should also notice a massive speed increase - we are operating on dual 3.0 ghz processors with 4gB of RAM. Storage is a RAID-5 array on multiple 30gB SCSI drives (10,000) RPM. Since we have roughly double the RAM of the size of our database, things should be smoking.

At this time, I've pretty much beat. The biggest hurdles have been cleared though, and we are well on our way to getting back to normal. Thank you for your patience and support througout this ordeal.
 
SEARCHING IS CURRENTLY DISABLED as I rebuild the search index. This should take about 12-48 hours.
 
Dang, sounds like a rough week for the Forums. Glad you were able to get things going again. We sure do apreciate what you do.:D
 
Sounds like you've had a rough time of it, Spark. We all appreciate your efforts to keep BF running.

By the way, what happened to W&C?


Edited to add: never mind the question about W&C. I wasn't logged in, and I guess I couldn't see it for that reason. It shows up OK now.
 
Nathan S said:
Sounds like you've had a rough time of it, Spark. We all appreciate your efforts to keep BF running.

By the way, what happened to W&C?
Click to expand...

Nathan,

Look for posts more than 2 months old.

Spark,

Great job man. Just think of this as practice for having children, nothing but time and money consuming :D
 
Jeez.... I think you've had Murphy sitting on your shoulder for a couple of days, Hopefully he's gone to annoy someone else now.... :D

Thanks a lot for all your hard work, I'm sure everyone appreciates it, I sure do, I was starting to have withdrawl symptoms and getting grumpy at work cause I couldn't get my BF fix :D

Thanx Spark!!

Ferreter
 
Spark,

Thanks for all your time & sweat. Faulty backups are a pain in the ...

The new forums are great. New features (quick reply is cool!), fixed bugs... We'll only appreciate them better, now ;)

Thanks, and keep up the good work.

David
 
You should also notice a massive speed increase - we are operating on dual 3.0 ghz processors with 4gB of RAM. Storage is a RAID-5 array on multiple 30gB SCSI drives (10,000) RPM. Since we have roughly double the RAM of the size of our database, things should be smoking.
Click to expand...

:D As Cartman would say, sweet. :cool: Thanks for getting us back online so quick Spark, I can only imagine how frustrating the last few days have been for you.
 
Thanks Spark for all of your hard work.
I first logged on to BFC this morning with Netscape, but for some strange reason there was no way to reply to any of the threads. There was a 'Reply' button but there was no screen to enter a message in. I'm on IE now and it seems to be woking fine. I guess it's going to take a while for all of the bugs to get ironed out.
I just wanted to let you know that your work is appreciated.
 
Spark,
Thanks for all the hard work. I appreciate it very much and look forward to renewing my Gold Membership within the next couple of weeks (I'm a state employee and we get paid once a month so I got to wait until end of March pay check is in). Get some sleep. ;) ;)
 
I haven't read all the preceding, just want to thank you Spark for your superhuman efforts getting our life going again! What an ordeal you had. I truly feel for you and wish you all the best. Sounds like you have a good solid plan in place now. Thanks for your tireless efforts. I hope you're getting some well-earned R&R while we're catching up.

Dave
 
Great job Spark!! I USED to do Database and LAN administration. And, people wonder why I got out of computers,and went back into Research!?! :D Thanks again, and good luck with the rest of the work.

Ken
 
You must log in or register to reply here.
Back
Top