BFC Server situation, outages, lags, and you.

Spark

Spark

HPIC - Hatas gonna Hate
Staff member
Administrator
Super Mod
Moderator
Joined
Oct 2, 1998
Messages
15,223
Here's the deal folks.

As you are aware, we moved from our dedicated servers to a new VMWare style setup late last year. This move was supposed to provide more reliable hosting, better support, more failover and redundancy, and all sorts of wonderful technical babble that doesn't mean anything to you guys. I was promised repeatedly that this would be a great thing, despite my reluctance.

Since the move, we've had outages daily, if not several times per day.

Initially, I was told this was because of the size of the database; because of our huge number of posts, the posts db was locking regularly, the search index was throwing problems, etc etc etc etc etc. The support guys tweaked, and fiddled, and fixed, and the search works... but the problems didn't stop.

It now turns out that this was a hardware issue all along. Apparently, despite the audit that was performed prior to our picking this hosting solution, our purchasing equipment based on the recommendations of this hosting company, and despite the reassurances that everything would be great with lots and lots of excess processing availability and such... the disk throughput is our bottleneck. At least, that's what I've been told this time. For some reason, the I/O goes into the toilet and the hosting company can't figure out why.

Needless to say, this is inexcusable. This problem should have been resolved 4 months ago. I'm not happy that you guys are dealing with problems, and I'm pissed off and embarrassed by this entire thing.

Supposedly (again) there is a plan: we are being moved to our own dedicated box instead of the virtualized machine. If this fails, my plan be is to yank the site, and move us back to dedicated servers again. If that doesn't work, who knows.

I apologize for the difficulties you guys are experiencing. I'm beyond frustrated. I've articulated how pissed off I am at a few people today, and hopefully those responsible get their heads out of their fourth point of contact.

If not, go to plan b.

Thanks for your continued support.
 
Spark, you are awesome!! Thank you for you time and all your efforts.
 
Do not be harsh on yourself... you are doing a fantastic job! We do not claim perfection, we STRIVE towards perfection.

Bill Handy
 
Spark, I know it is frustrating with what you are trying to accomplish, but I think I speak for volumes of folks that know you are doing all you can. Sometimes computer geeks just do not get it. It is not your fault, If I can send in some collectibles for you to raffle off and help with money, just say the word. A lot of folks will step up and help.

Thanks for all you are doing,
Jim
 
Spark said:
I apologize for the difficulties you guys are experiencing. I'm beyond frustrated. I've articulated how pissed off I am at a few people today, and hopefully those responsible get their heads out of their fourth point of contact.
Click to expand...

As a fellow webmaster, I feel your pain.

A couple of years ago, one of my sites was going down repeatedly. Newest version of MySQL, php, Apache, Red Hat, etc, and etc, and etc.

When it wasn't down, it was slow as Christmas. Trying to run phpbb was like pulling teeth, only slower, and more painful.

Finally determined that bad memory was the culprit. Yanked the memory, installed new chips, and all was groovy. But, for the month prior, it was one "guess fix" after another. It can, indeed, be very frustrating.

Hang in there, remember to breathe, and don't let it be a bigger issue than it really is.
 
Don't worry about it, you can only do what you can do. Most people understand and you'll get it sorted eventually. Just keep kicking arses until they fix the problem.

Thanks for all your efforts, lots of people appreciate it.
 
James, thanks for the offer, but right now I feel bad about asking for money when I can't provide a reliable service. When it's time for raffles and such, I'll ask.
 
Kevin thanks for all you do! Its a minor thing anyway, and even as a paying member I think I get so much more out of it than I pay for--I'm sure most other people feel the same way--and so we are willing to put up with any small glitches here and there...

What do I have to help you with to get a pirate next to my name?!?!

Best of luck!
 
At work we've been moving more and more of our production systems to Amazon's EC2. After about 6 months on EC2 we've had about the same level of reliability (5 9s? no ... 4? no ... 3? hell, I'm happy to get 99.0% uptime) as with Rackspace, who we've been on for about 6 years.

But the cost is at least 3 times cheaper. If you're interested, I'd be happy to relate more experiences and info privately, and provide free support for getting set up if you decide that's how you want to go.


Cheers and good luck!

- Robert
 
Spark said:
. I'm beyond frustrated. I've articulated how pissed off I am at a few people today, and hopefully those responsible get their heads out of their fourth point of contact.
Click to expand...

Hopefully you used phrases such as, "donkey punch", "robot canon", and "light on fire and kick in the nuts". :p

donkey_punch.png
 
Well you got my support and I'm glad you are working to improve things for us. You and this community will continue to have my support! Keep up the great work Spark!

ken
 
sirahren said:
Kevin thanks for all you do! Its a minor thing anyway, and even as a paying member I think I get so much more out of it than I pay for--I'm sure most other people feel the same way--and so we are willing to put up with any small glitches here and there...
Click to expand...
What he said...:thumbup:
 
Kevin, thanks for the update. It has been frustrating, but it's good to know you're on top of things.
 
Has it been frustrating? Sure. A darn PITA? Occasionally... But this is still the best damn Forum I've ever been privileged to be a member of, and I just want to THANK YOU for your continued efforts to keep this big 'ship' running smoothly. Glitches happen, and your update here to keep us in the loop is totally appreciated. I don't know about anyone else, but smooth sailing or no, I plan to stick around and continue to support this Forum for as long as you'll have me. Hang tough, ya pirate, and give 'em bloody hell! :D

Thank you, Spark!! :thumbup::thumbup:
 
Spark said:
Here's the deal folks.

As you are aware, we moved from our dedicated servers to a new VMWare style setup late last year.
Click to expand...

Virtual Machines. Seems I recall some cautionary articles on the perils of virtual machine technology. Your hosting service is probably hosting nearly impossible to detect rootkit experiments from all the usual suspects. Yeah would probably slow down I/O quite a bit. I'd find another vendor.
 
You must log in or register to reply here.
Back
Top