Tough call... IMO, both tools are just about as necessary (for me) as the other, but I don't particularly enjoy spending all day long cutting out blanks, or burning through hacksaw blades at an alarming rate. That said, I also use a drill press for quite a lot as well.
You MIGHT be able to swing both a small benchtop drill press and a portaband (old and well used most likely), but you'll have to shop around and be patient.