I think mills are like motorboats or motorcycles. You always wish you had gotten a bigger one. If you want precision you will want a larger one. The more mass - the more stability. I have a 9 X 42 Jet knee mill. A lot of money but a nice machine and I don't think I'll outgrow it.
Also, look into the cost of tooling up. The mill itself is just a part of the cost. You'll need a vise, collets, cutters, etc. It really adds up.
My machine is big enough that I can make other tools on it. All kinds of little things that make the job easier.