BFC Server situation, outages, lags, and you.

Discussion in 'Tech Support, General Q&A, Site Feedback & More' started by Spark, Apr 3, 2009.

  1. Spark

    Spark HPIC - Hatas gonna Hate Staff Member Administrator Super Mod Moderator

    Oct 2, 1998
    Here's the deal folks.

    As you are aware, we moved from our dedicated servers to a new VMWare style setup late last year. This move was supposed to provide more reliable hosting, better support, more failover and redundancy, and all sorts of wonderful technical babble that doesn't mean anything to you guys. I was promised repeatedly that this would be a great thing, despite my reluctance.

    Since the move, we've had outages daily, if not several times per day.

    Initially, I was told this was because of the size of the database; because of our huge number of posts, the posts db was locking regularly, the search index was throwing problems, etc etc etc etc etc. The support guys tweaked, and fiddled, and fixed, and the search works... but the problems didn't stop.

    It now turns out that this was a hardware issue all along. Apparently, despite the audit that was performed prior to our picking this hosting solution, our purchasing equipment based on the recommendations of this hosting company, and despite the reassurances that everything would be great with lots and lots of excess processing availability and such... the disk throughput is our bottleneck. At least, that's what I've been told this time. For some reason, the I/O goes into the toilet and the hosting company can't figure out why.

    Needless to say, this is inexcusable. This problem should have been resolved 4 months ago. I'm not happy that you guys are dealing with problems, and I'm pissed off and embarrassed by this entire thing.

    Supposedly (again) there is a plan: we are being moved to our own dedicated box instead of the virtualized machine. If this fails, my plan be is to yank the site, and move us back to dedicated servers again. If that doesn't work, who knows.

    I apologize for the difficulties you guys are experiencing. I'm beyond frustrated. I've articulated how pissed off I am at a few people today, and hopefully those responsible get their heads out of their fourth point of contact.

    If not, go to plan b.

    Thanks for your continued support.
     
  2. Bronco

    Bronco Moderator Moderator

    Feb 25, 2000
    You da man, Spark.
     
  3. T.K.C.

    T.K.C.

    May 7, 2005
    Spark, you are awesome!! Thank you for you time and all your efforts.
     
  4. wmhandy

    wmhandy

    29
    May 5, 2007
    Do not be harsh on yourself... you are doing a fantastic job! We do not claim perfection, we STRIVE towards perfection.

    Bill Handy
     
  5. JatMat

    JatMat

    Aug 26, 2006
    Spark, I know it is frustrating with what you are trying to accomplish, but I think I speak for volumes of folks that know you are doing all you can. Sometimes computer geeks just do not get it. It is not your fault, If I can send in some collectibles for you to raffle off and help with money, just say the word. A lot of folks will step up and help.

    Thanks for all you are doing,
    Jim
     
  6. AllenMcB

    AllenMcB

    75
    Mar 19, 2008
    As a fellow webmaster, I feel your pain.

    A couple of years ago, one of my sites was going down repeatedly. Newest version of MySQL, php, Apache, Red Hat, etc, and etc, and etc.

    When it wasn't down, it was slow as Christmas. Trying to run phpbb was like pulling teeth, only slower, and more painful.

    Finally determined that bad memory was the culprit. Yanked the memory, installed new chips, and all was groovy. But, for the month prior, it was one "guess fix" after another. It can, indeed, be very frustrating.

    Hang in there, remember to breathe, and don't let it be a bigger issue than it really is.
     
  7. gajinoz

    gajinoz

    Sep 2, 2003
    Don't worry about it, you can only do what you can do. Most people understand and you'll get it sorted eventually. Just keep kicking arses until they fix the problem.

    Thanks for all your efforts, lots of people appreciate it.
     
  8. Spark

    Spark HPIC - Hatas gonna Hate Staff Member Administrator Super Mod Moderator

    Oct 2, 1998
    James, thanks for the offer, but right now I feel bad about asking for money when I can't provide a reliable service. When it's time for raffles and such, I'll ask.
     
  9. avidknifeusersf

    avidknifeusersf

    958
    Mar 12, 2007
    Kevin thanks for all you do! Its a minor thing anyway, and even as a paying member I think I get so much more out of it than I pay for--I'm sure most other people feel the same way--and so we are willing to put up with any small glitches here and there...

    What do I have to help you with to get a pirate next to my name?!?!

    Best of luck!
     
  10. robertp

    robertp

    9
    Jan 2, 2009
    At work we've been moving more and more of our production systems to Amazon's EC2. After about 6 months on EC2 we've had about the same level of reliability (5 9s? no ... 4? no ... 3? hell, I'm happy to get 99.0% uptime) as with Rackspace, who we've been on for about 6 years.

    But the cost is at least 3 times cheaper. If you're interested, I'd be happy to relate more experiences and info privately, and provide free support for getting set up if you decide that's how you want to go.


    Cheers and good luck!

    - Robert
     
  11. zenheretic

    zenheretic Platinum Member Platinum Member

    Oct 29, 2005
    Hopefully you used phrases such as, "donkey punch", "robot canon", and "light on fire and kick in the nuts". :p

    [​IMG]
     
  12. babakanoosh45

    babakanoosh45

    Oct 6, 2006
    Thanks for the update. Still my favorite site!!!:thumbup:


    :D
     
  13. T. Erdelyi

    T. Erdelyi Gold Member Gold Member

    Feb 3, 2001
    Thanks for the update Kevin, I thought the problem was on my side.
     
  14. kgd

    kgd

    Feb 28, 2007
    Well you got my support and I'm glad you are working to improve things for us. You and this community will continue to have my support! Keep up the great work Spark!

    ken
     
  15. Doug C

    Doug C

    Mar 15, 2005
    What he said...:thumbup:
     
  16. Samael

    Samael

    Sep 30, 2007
    Kevin, thanks for the update. It has been frustrating, but it's good to know you're on top of things.
     
  17. jill jackson

    jill jackson

    Sep 5, 2006
    I'm sure they will be Walking the Plank if they don't get it done
    [​IMG]
     
  18. James_Terrio

    James_Terrio

    Oct 6, 2008
    I for one will continue to use and support this forum. :thumbup:
     
  19. Jaxx

    Jaxx Moderator Moderator

    Jan 18, 2006
    Has it been frustrating? Sure. A darn PITA? Occasionally... But this is still the best damn Forum I've ever been privileged to be a member of, and I just want to THANK YOU for your continued efforts to keep this big 'ship' running smoothly. Glitches happen, and your update here to keep us in the loop is totally appreciated. I don't know about anyone else, but smooth sailing or no, I plan to stick around and continue to support this Forum for as long as you'll have me. Hang tough, ya pirate, and give 'em bloody hell! :D

    Thank you, Spark!! :thumbup::thumbup:
     
  20. shunsui

    shunsui Gold Member Gold Member

    Aug 12, 2008
    Virtual Machines. Seems I recall some cautionary articles on the perils of virtual machine technology. Your hosting service is probably hosting nearly impossible to detect rootkit experiments from all the usual suspects. Yeah would probably slow down I/O quite a bit. I'd find another vendor.
     

Share This Page