If you find the right type of class, Yoga is a great thing to incorporate into a larger health and fitness regimen.
I'm not a "fitness buff," but I have been teaching Yoga for about six years and actively practicing for 10.
There are lots of types of classes and lots of really bad instructors, so you have to be careful and put time in up front to find what and who works for you.
And if you're serious about giving it an honest try, don't give up just because you don't like a particular class or instructor. My recommendation is to try out a few different ones before making a final judgement on whether you're a "yoga person," which is what I hear some people say. (Yoga is for everyone -- so to me, everyone is a "yoga person.)
I don't do any of the "spiritual" instruction in my classes -- it's purely focused on the body and quieting the mind. The more you do of both -- strengthen the body, quiet the mind -- the more you help your spirit. I say this because there are a lot of nut jobs who teach Yoga and you shouldn't feel like you have to put up with them just to take a yoga class.
Regardless of whether you check out Yoga, my biggest piece of advice is to pick what works for you -- not just in exercise, but food, sleep, work, etc.
It's unrealistic to force anything upon yourself -- short of a bootcamp environment. So really figure out a routine that you can tolerate and maintain and won't want to flake out on constantly.
For example, I hate being inside a gym running on a treadmill -- so I don't do it. I'd rather play 50 minutes of a pick-up soccer game, or walk the 2 miles home from work sometimes, instead of taking the subway.
As an alternative to the monotonous weight machine routine, again in the confines of a gym, I have a couple dumb bells at home and use the fitness trail of a local park for pull-ups, push-ups and other strength training.
Not forcing your diet is just as important. I'm not the healthiest eater by far -- I'm addicted to sweets and drink two cups of coffee a day, at least. But I'm much healthier than the average American b/c I limit my indulgences and gravitate toward healthy foods when I can.
I still eat fries, burgers, pizza, candy bars and other junk. But I balance it out with healthy stuff. You can tell when your body has had too much of that stuff.
Actually, you can tell a lot about your own body, once you start paying attention to it -- often more than doctors can. But that's a whole other topic.