If you want cheap and durable, military surplus is often the best way to go.
Get a Medium ALICE pack (with or without frame), a couple of Army Wool blankets, a couple of canteen kits (canteen, cup and stove in the pouch -- can ride on a belt or on the pack).
Get some decent cutlery (for about 20 years my outdoors blades were a Buck 110, a Ka-Bar USMC, and an Estwing hatchet).
Raid your medicine cabinet for a basic first aid kit. Raid the cupboards for camp food (crusty bread like baguettes last longer in the field, peanut butter, SPAM, rice, mashed potato flakes, etc).
Raid your closet for seasonally appropriate clothing.
If you are staying overnight or longer "in the weather" a Wal-Mart tarp. some 550 cord or even clothesline, or a surplus pup tent work.
Some strike anywhere matches, Bic lighter and cotton balls or dryer lint make a good fire kit.
Cook with your canteen cups, or get a surplus mess kit (about the same price and won't fall apart the first time like the Wal-Mart kits), or bring the smallest pot you own.
No biggie.
Best reason to start cheap, is you may not know how you like to camp, or what you like to use. Better to dial in your preferences on cheap stuff that doesn't cost a lot and is easier to flip when the time comes to get rid of it. After some experience, you'll come up with a bunch of "It'd be nice if [piece of gear] di this, or had that feature", then you save you money while wringing things out until you find the better piece of gear that fits what YOU want it to do.
Which is a large part of why you see some of the high $ equipment here. Someone's been doing something for umpteen years and finally dropped the bank on what they really want, rather than just making do. I see no more virtue in making do with something when you can buy what you want, than I do with buying something expensive, just because it's a status thing, not because it's what you really want.
Some things you may never change. I've tried a lot of high-end stuff, but I haven't found anything that can replace my cheap-ass GI Poncho. I've tried the new-fangled cooksets and bottles and such, but when it comes down to it, I almost alwasy go with my GI canteen kits instead.
Bought a Kifaru backpack. When I got done accessorizing it and stood back to look, I said "Great, I just made a $1000 ALICE pack."
When I go light, I don't use an ultralight backpack, I use my GI web gear. I even drew up a set of custom gear I was going to make or have made. But I didn't. Why?
Because it would be a custom leather set of WEB GEAR!! 
Some things are worth ditching. I love the quaintness of my pup tent. But since I almost always go alone, carrying the 12 pounds of it gets old, fast. A Sil Tarp is much nicer. But a polyurethane tarp would work just as well, just not be as small. And those don't have to be blue, either. Wal-Mart had a nice one that was brown with a sort of dull silver on the other side -- good for stringing up benind you to reflect the fire's warmth back at you.
