I think that the biggest problem, or rather the weak link in my setup, was that I have a really crappy sleeping bag. I have some others I am looking at the are rated for much lower, and I think that should do it. I have an Alps Mountaineering 2" sleeping pad that worked really well, and was very comfortable, and I was hoping the rest of their gear might be decent as well. Here is the tent I am heavily considering, mainly because it looks to be able to be adjusted as far as the venting goes.
Alps Mountaineering Vertex 2
Like I said, I wasn't uncomfortably cold, just wouldn't have minded being warmer, and I think like you guys are a saying a better sleeping bag would do me better. Thankfully I don't need to worry about weight too much since we usually are able to drive where we are going. It was a lot of fun, which I don't think it would have been had I been freezing, but it was educational in that I need to rethink some of my gear choices and upgrade in a few key areas.
Location-wise we are around the Beckley area for most of our campouts.