Somehow this short conversation strikes me as right on; like two strangers whispering to one another at the edge of the firelight outside a crowd of folks gathered there who are all friends.
There are a whole lot of years of great information posted on this site from remarkably talented, and even more generous, knifemakers on what tools you need/can upgrade to, and how to use them to make knives. Try a few different searches, with words like you used in your post: "Budget starting grinder"; like you'd search google. Lots of the big guys, the most gifted and experienced and respected knifemakers in the world, frequent this board and have posted answers to questions like yours before. Right here, and on other similar sites, you'll find by searching a library of information that will astonish you, and that you wouldn't ever be able to accrue at Amazon.
Personally, I've only just started. I made my first knife-shaped piece of metal (which was garbage steel that I never hardened) with a bench grinder to profile and a hand-held belt sander clamped to the top of my workbench. In the end, it looked alright, but was messy as hell. I then picked up a small belt sander from Harbor Freight (that takes 30"x1" belts) for like $30 (the 2x42 Craftsman is a similar option for around $100), with which I fashioned a few more that also looked like they'd been made sloppily by a moron with inferior tools. A little grinder like that will take steel away, but it turns that belt so fast that it's very hard to control.
Then, after falling completely under the spell, I went for a KMG (takes 2"x72" belts and can be set up for various speeds). The KMG set me back over a grand all told, so it's no small investment. But if you want to turn out blades (I reasoned), each one better than the other, that seems to be the way to go. I've been working on that grinder for several months now and have made some pieces that I'm quite proud of. But the truth is I spend a HELL of a lot more time with an old-fashioned file in my hand than I do hogging steel with the grinder.
When you first put steel to stone you'll do it with this idea in your head that it is an incredibly dense material that takes great presure and speed to shape. We trust that the substance that holds up our buildings and highways and comprises so much of the tangible world around us is, once shaped, immuable. We've all used the phrase, "iron-clad". But in truth, steel, in its annealed state as you'll be using it, is as yeilding as clay, and is better thought of by the beginning knifemaker in that regard. The word malleable, which means capable of being shaped of formed, comes from the Latin, malleus, which means hammer, refering metals plasticity (to use the modern vernacular). Point being, you don't really need a machine that moves at 4000rpm to shape steel. In fact, you'll learn a whole lot more, if you really want to know, about how steel behaves by laying into it with a file than you will with a 1x30 Harbor Freight.
And you'll never get as clean a product with a 1x30 as you will with a sharp set of files. You'll be much more prone, as I am, to making mistakes and removing too much as a novice (and steel, unlike clay, can't be stuck back on!), and it will therefore be vital for you to slow all the processes down in order to finish with something you can be proud of.
Figure out your design on paper. Transfer it to posterboard or wood or plexiglass (something thicker is best for giving a realistic idea of how the blade will feel in the hand) as a template. Pick up some 1095 from Admiral or O1 from MSC, draw the outline on the steel and then follow this tutorial:
http://hossom.com/tutorial/jonesy/
There are guys out there that have made, and still do make, world class knives without the use of machinery beyond handfiles and a drill press. Your hands will certainly be sore (they'll be sore and cut up and scabbed no matter how much you spend on tools!), but you'll learn a whole lot more about steel that way; I guarantee your learning curve will be steeper via that route than any other.
I hope you take the plunge, because personally I've found it as rewarding as anything I've ever done. Take your time, consider your steps before you take them and you'll be floored at the result.
And make sure you post it, because folks around here love pics!
Good luck!
Bukiewicz