I went through basically this same thought process when buying a mill about 13 years ago. I started looking at Harbor Freight's options, then Little Machine Shop, Grizzly, and finally Precision Matthews. PM seemed like the best option all around if you wanted the nicest machine right out of the box, but the prices started adding up quick. I figured for what even the new "lower cost" mills would run me, I'd probably be able to find something really nice in the used market.
I ended up finding a Kondia knee mill (bridgeport clone) at a used machinery dealer about 2-1/2 to 3 hours north of me. They had their own truck with a lift gate and we negotiated a fair cash price that included lift gate delivery. It came on a pallet, so I had to tilt the head 90 degrees to get it into my garage door, but the driver was able to roll it with a pallet jack nearly all the way to where it sits to this day. I used a 4 foot prybar to slide it the rest of the way, and carefully knocked the pallet out in pieces from under the machine with a sledge hammer. It only almost tipped over once.
All that to say, if you could get a similar delivery from a relatively local dealer, I imagine you could put some plywood down and use a pallet jack right off the lift gate. I've moved equipment over grass and gravel like this lots of times. If needed, you could even partially disassemble the machine with an engine hoist and a flat cart and then move the pieces one at a time into your shop. It's a pain, but it's one and done (at least until you decide to move...haha)
I'll just add that the final price of my mill was well under 1/2 of what the average brand new benchtop mill would cost me from PM, and still cheaper than even the smallest new models from places like LMS. I should add, however, that over the years I've spent well over the cost of my mill in tooling and accessories, but these are things you can add as you need them. At least get a vise, a collet set, a drill chuck, and a DRO is money well spent.
I don't know what kind of work you plan on using your mill for, but I'll also add that there are times I wish my full sized mill was even bigger. That said, if you're just doing folders and guards and that kind of thing, you'll probably be fine with just about any sized mill at the end of the day. It is nice having some extra size and power when you need it though.