Our move, our location, our traffic, and how it affects you

[Spark]

Ok guys, now that the uglyness of server configuration is out of the way....

What happened: our outages this and last week
As you know, we've been scheduled to move to this new building for some time. Unfortunately, there has been a severe lack of communication coming from our ISP, which resulted in our not being contacted when the rest of the Co-Location servers were moved last week. The result was the initial outage of our system.

Since we were off the network, we figured, "Why not upgrade the operating system?" Why not indeed. Well, that royally screwed up the works, and since no upgrades ever go smoothly, we became a case study in frustration. 2 tech support calls to RedHat later, we were back on the network, and ready to serve up webpages again.

Cut to yesterday - we're planning to move to the new building when suddenly our server is off the network again. It appears that they just unplugged it, without even calling us... I haul ass down to the server, grab it and move it to the new building.

Our ISP decided to let us upgrade the number of IP's we have to half of a Class C (128 IPs total) for free (shweeeeeet!!!!), but we had to add a router to our half rack and some other sundry items. During the switch over of our old IP's to our new subnet, our routing table was corrupted, resulting in yesterday's and today's downtime.

The plus thing is that we are now in a much more stable environment. The minus is that our throughput is limited for right now as we have 10 channels of a T-1 allocated into the server - and we're already using 98% of that. That will be changing in the next 2 days as they are installing a dedicated T-1 pipe for us which will lead into the OC-48 backbones of the ISP.

Our Daily Traffic
According to the router we now have in place, between BladeForums.com and TFL, we're using 98% of our current 10 channels. That means that our current max level of traffic is 640K per second, which we're already hitting now. The T-1 has 24 channels, which is more than double of our current capacity - which should be more than enough.

For those who would say "I don't think they get *that* much traffic!", here's a pretty good relative indicator. At the top of the main categories listings there is a list of active topics for today.

Here's the <a href="http://www.bladeforums.com/cgi/search.cgi?action=getdaily" target="_blank">direct link</a> for ours. Notice the number of active topics, and we've only been up today since, oh, 2 PM. Typically it's around 350 or so total for the day.

Now, here's a <a href="http://wbboards.warnerbros.com/ultimate/cgi/search.cgi?action=getdaily" target="_blank">direct link</a> to the same feature on Warner Brothers UBBs.

Check out any of the other sites out there and see how many active topics they have, then compare them to ours.

Now, this is just the number of active threads, that is, the number of threads that are being created and being replied to. That doesn't even count the number of pages that people are just reading and not responding to.

Short of publishing our log files (something no reputable site would do), there's no extremely accurate way to show the exact amount of actual traffic that we have. Compare those threads though, and you'll see what I mean. Better yet, check out the posts that have been crossposted to the various forums out there, and see the number of replies....

Thanks for all of your patience, guys, we appreciate your sticking with us while we experience our growning pains which are inevitable with any successful site. We've cleared the last major hurdles, and with only a minor bump in the near future (when they hook up the T-1), so it should be pretty cool from now on.

Spark

------------------
Kevin Jon Schlossberg
SysOp and Administrator for BladeForums.com

Insert witty quip here

 

[Bob Irons]

Thanks for the story Spark. The slowdown I've been seeing fits with need for more channels.

T-1 heck, I can't wait for the T-3 and then multiple T-3s. Onward and upward.

BladeForums Ho!

 

[Walt Welch]

Well, Spark, old buddy, I don't pretend to understand all of the jargon you used, but it appears that you finally got the right size penny to put into the fuze box!

Good thing, too, as you were dopplered down into the IR earlier today.

Have fun, and remember, mistakes are opportunities to learn. Walt

 

[Mike Turber]

With the new T-1 in place we should start screaming here in the next couple of days. We tried to see if 10 channels would work and obvioulsy, based on the slowdown we all experienced, it did not.

So I took the plundge! Yup a FULL T-1 direct loop to our server into an OC-48 backbone. This will mostly benifit BFC and TFL, the other sites we host will notice an increase in speed to but serving web pages is cake. This is not a T-1 to my business folks. This is a T-1 connected directly to our server and only 50 feet from the OC-48 pipe room (which looks really sweet!).


Pics of our server coming soon

So what is a T-1?
The T-carrier system, introduced by the Bell System in the U.S. in the 1960s, was the first successful system that supported digitized voice transmission. The original transmission rate, 1.544 Mbps(*1), in the T-1 line is in common use today in Internet service provider (ISP) connections to the Internet. Another level, the T-3 line, providing 44.736 Mbps, is also commonly used by ISPs. Our ISP uses a OC-48 which is equal to 1,611 T-1's! Another commonly installed service is a fractional T-1 line, which is the rental of some portion of the 24 channels in a T-1 line, with the other channels going unused. We had 10 channels out of the available 24 going to our box.

The T-carrier system is entirely digital, using pulse code modulation (*2) and time-division multiplexing(*3). The system uses four wires and provides full-duplex (*4) capability (two wires for receiving and two for sending at the same time). The T-1 digital stream consists of 24 64-Kbps channels (*5) that are multiplexed(*6). (The standardized 64 Kbps channel is based on the bandwidth required for a voice conversation.) The four wires were originally a pair of twisted-pair copper wires, but can now also include coaxial cable, optical fiber, digital microwave, and other media. A number of variations on the number and use of channels are possible.

Voice based T-1.
In the T-1 system, voice signals are sampled 8,000 times a second and each sample is digitized into an 8-bit word. With 24 channels being digitized at the same time, a 192-bit frame (24 channels each with an 8-bit word) is thus being transmitted 8,000 times a second. Each frame is separated from the next by a single bit, making a 193-bit block. The 192 bit frame multiplied by 8,000 and the additional 8,000 framing bits make up the T-1's 1.544 Mbps data rate. The signaling bits are the least significant bits per frame.

• *1 Mbps stands for millions of bits per second and is a measure of bandwidth (the total information flow over a given time) on a data transmission medium such as twisted-pair copper cable, coaxial cable, or optical fiber. Depending on the medium and the transmission method, bandwidth may also be in the Kbps (thousands of bits or kilobits per second) range or the Gbps (billions of bits or gigabits per second) range.
• *2PCM (pulse code modulation) is a digital scheme for transmitting analog data. The signals in PCM are binary; that is, there are only two possible states, represented by logic 1 (high) and logic 0 (low). This is true no matter how complex the analog waveform happens to be. Using PCM, it is possible to digitize all forms of analog data, including full-motion video, voices, music, telemetry, and virtual reality (VR).
  
  To obtain PCM from an analog waveform at the source (transmitter end) of a communications circuit, the analog signal amplitude is sampled (measured) at regular time intervals. The sampling rate, or number of samples per second, is several times the maximum frequency of the analog waveform in cycles per second or hertz. The instantaneous amplitude of the analog signal at each sampling is rounded off to the nearest of several specific, predetermined levels. This process is called quantization. The number of levels is always a power of 2 -- for example, 8, 16, 32, or 64. These numbers can be represented by three, four, five, or six binary digits (bits) respectively. The output of a pulse code modulator is thus a series of binary numbers, each represented by some power of 2 bits.
  
  At the destination (receiver end) of the communications circuit, a pulse code demodulator converts the binary numbers back into pulses having the same quantum levels as those in the modulator. These pulses are further processed to restore the original analog waveform.
  
• *3 TDM (time-division multiplexing) is a scheme in which numerous signals are combined for transmission on a single communications line or channel. Each signal is broken up into many segments, each having very short duration.
  
  The circuit that combines signals at the source (transmitting) end of a communications link is known as a multiplexer. It accepts the input from each individual end user, breaks each signal into segments, and assigns the segments to the composite signal in a rotating, repeating sequence. The composite signal thus contains data from all the end users. At the other end of the long-distance cable, the individual signals are separated out by means of a circuit called a demultiplexer, and routed to the proper end users. A two-way communications circuit requires a multiplexer/demultiplexer at each end of the long-distance, high-bandwidth cable.
  
  If many signals must be sent along a single long-distance line, careful engineering is required to ensure that the system will perform properly. An asset of TDM is its flexibility. The scheme allows for variation in the number of signals being sent along the line, and constantly adjusts the time intervals to make optimum use of the available bandwidth. The Internet is a classic example of a communications network in which the volume of traffic can change drastically from hour to hour. In some systems, a different scheme, known as frequency-division multiplexing, is preferred.
  
• *4 In telecommunication, duplex communication means that both ends of the communication can send and receive signals at the same time. Full-duplex communication is the same thing. Half-duplex is also bidirectional communication but signals can only flow in one direction at a time. Simplex communication means that communication can only flow in one direction and never flow back the other way.
  
  An ordinary telephone conversation is a duplex communication. Most inexpensive speakerphones in conference rooms are half-duplex communication. (If you're speaking, you can't hear anyone else interrupt. You have to pause to let others speak.)
  
• *5 1) In telecommunications in general, a channel is a separate path through which signals can flow.
  
  2) In the public switched telephone network (PSTN), a channel is one of multiple transmission paths within a single link between network points. For example, the commonly used (in North America) T-1 line service provides 24 64 Kbps channels for digital data transmission.
  
  3) In radio and television, a channel is a separate incoming signal or program source that a user can select.
  
  4) In optical fiber transmission using dense wavelength-division multiplexing (DWDM), a channel is a separate wavelength of light within a combined, multiplexed light stream.
  
  5) On the World Wide Web, a channel is a preselected Web site that can automatically send updated information for immediate display or viewing on request. See Webcasting (push technology).
  
  6) In computer and Internet marketing, a channel is a "middleman" between a product creator and the marketplace. Value-added resellers (VARs) and retail store chains are examples of channels in this context.
  
  7) Using Internet Relay Chat (IRC), a channel is a specific chat group.
  
  8) In IBM mainframe systems, a channel is a high bandwidth connection between a processor and other processors, workstations, printers, and storage devices within a relatively close proximity. It's also called a local connection as opposed to a remote (or telecommunication) connection.
  
• *6 Multiplexing is sending multiple signals or streams of information on a carrier at the same time in the form of a single, complex signal and then recovering the separate signals at the receiving end. Analog signals are commonly multiplexed using frequency-division multiplexing (FDM), in which the carrier bandwidth is divided into subchannels of different frequency widths, each carrying a signal at the same time in parallel. Digital signals are commonly multiplexed using time-division multiplexing (TDM), in which the multiple signals are carried over the same channel in alternating time slots. In some optical fiber networks, multiple signals are carried together as separate wavelengths of light in a multiplexed signal using dense wavelength-division multiplexing (DWDM).

Now with a T-1 connect we can transfer up to 1.544 Mbps. 1.544 Million Bits Per Second. In real numbers you would devide by 8 to get the Bytes per second which is what us comuter nerds use or 193,000 Bytes per second. Right now we are passing nearly 5 Gigs per day. Using the math that Spark and I have had to learn over the last year, we can now pass up to 16.675 Gigs a day.

So now you know our stats that Spark eluded to. In fact our log file is over 350 megs each 5 days.

So in a couple of days we will be @ 1/3 the capacity of our T-1.

Yes this costs money and yes I guess I am whining when I am the one paying the bills

Actually I am letting you guys know what is takes to run a site as busy as our is.

Here is where we are colocated. http://www.fdn.com/internet/dedicated-ptp.cfm

Here is what our server is.
Prosignia Server 740 6/450
Intel® Pentium® II processor 450MHz
576MB 100MHz ECC SDRAM Total Memory(1x64, 2x256)
Integrated Dual Wide Ultra SCSI-3 Controller
2X 9.1GB Wide Ultra2 SCSI Hard Drives @ 10,000 RPM (1")
1.44MB Floppy Disk Drive
32X CD ROM Drive
Compaq 10/100 TX UTP (Embedded)
SmartStart & Compaq Insight Manager
Country Kit-Keyboard & Mouse

I will get pics for those interested tomorrow

------------------
Best Regards,
Mike Turber
BladeForums Site Owner and Administrator
Do it! Do it right! Do it right NOW!
Show Your Support -Visit Our Sponsors - Click On The Banners!

 

[Alberta Ed]

Carry on, O Wearers of the Red Hat.
Like a lot of other forumites, you don't miss the water 'til the well goes dry. Glad you're back on.

 

[swball]

Of course you're gonna prop up some BF knives alongside the server when you take pics, right?

Mmmmm...knives and servers..

------------------
Shawn
"Earth has its boundaries, but human stupidity is limitless."

 

[DonL]

Geez, Mike, that made my head hurt.

Almost like information overload, sorta like hearing about your parent's sex life. All you need to know is that it worked well enough for your own benefit.

(Looking for aspirin and making an appointment with a therapist...)

------------------
Don LeHue

Rome did not create a great empire by having meetings...they did it by killing all those who opposed them.

 

[PhilL]

Wow Mike, was any of that in English? I thought I was reading HTML for awhile, I tried to put it through my Decoder. As for me, I don't care how Blondes get to be Blonde, as long as they end up Blonde, and I have no idea how this place works and just as long as I don't have to get out and push I'm happy.

 

[Hammer]

I'm going to start hanging out at GeekForums.com so I can understand all this stuff.

Mike K.

 

[Snoopy]

Before this thread I thought I was fairly computer savvy - sheesh.

 

[Mike Turber]

Welcome to the net.

Sit back and relax, enjoy the ride, as Spark and I take you on this wonderful tour.

Don't worry about how the engine works, just remember where the gas station is

------------------
Best Regards,
Mike Turber
BladeForums Site Owner and Administrator
Do it! Do it right! Do it right NOW!
Show Your Support -Visit Our Sponsors - Click On The Banners!

[This message has been edited by Mike Turber (edited 03-08-2000).]

 

[not2sharp]

Mike,

Good presentation.

 

[pyro_1]

Well being a computer tech and well i guess a hardware geek i like the setup unfortunatly i will not be able to make use of the upgrades much as i am looking my nice fast cable connection and going back to 56k this weekend

bayden

 

[Matt Shade]

Thanks for letting us in on whats happening guys. I have no idea what that meant

but it sounds good I think. The only upgrades I'm familiar with are lift kits, bigger tires, and stuff like that

Whatever you guys just wrote definitely wasn't in redneck lingo.
I emailed you guys earlier when I had problems getting on the forums. Both Mike and Spark replied to me in short order. I appreciated that and wanted to say thanks, but figured you had enough emails to read and things to do as it was and didn't want to bother you. So thanks for everything, and keep up the good work.

------------------
Fix it right the first time, use Baling Wire !

 

[Brian_Turner]

To help give some perspective, I manage a 2000 user, 50 site network, which also hosts several "public" web sites. All of this usage generates approximately 25-30% utilization on our internet T-1.

That's one cookin' site you've got there, Spark! Congrats!

------------------
AKTI Member #A000832

"A man's got to know his limitations." - Dirty Harry