I think its a conditioning thing. By that I mean that our sub-conscious filters massive amounts of info before telling us what it thinks is going on. When you are in the city, it knows that the fast moving things are big, and will hurt, and that slow things are just people to be avoided. Take that mindset, and all the narrow vertical things (trees) should be moving, but they are not (trees don't move, people do. parts of our brains can be really stupid) and the fast moving things (sparrows and squirrels) are little and avoid us, but the danger filter doesn't know that. Add to that a new set of patterns to interpret, and it can put you on edge. The brain tells the body to keep ready. Our brains also pull another nasty stunt. They cheat. Our brains need to have a path of narration, things have to happen in order, and for a reason. So if you suddenly notice something, then your brain adds in all the motion that got it to there. Even if there was none, this can then be interpreted as a hallucination, which ups the adrenalin level, and makes it more likely to happen again, or just an optical illusion. There are two cool proofs of this. First one is to look at an analog clock with a second hand. If you look at it at just the right time, the second hand will hang, and then jump ahead. Thats because your brain goes, "yep thats a clock, okay, anything else going on here, nopeHOLY CRAP IT MOVED.... okay.... yep it moved, I don't think he noticed.... um, yeah, moving on then" The other one is that for over a hundred years, people have been convinced that there is such a thing as a curve ball, even though the human arm cannot spin a baseball fast enough to get it to walk. But the spinning causes the brain to think its moving sideways a bit, and then when the ball moves from the batters peripheral (motion based) vision to the central (fine detail) the brain realizes that the ball is in the wrong spot. So instead of a vast correction, it takes a half second to draw the ball moving to where it needs to be.
So little details that your brain needs to explain get added to bigger and bigger patterns. And that can make you feel uneasy. The reverse is often true, go from familiar suburban or rural surroundings to a packed urban center, and all of a sudden none of the patterns make sense, and the body and brain go on high alert.
Each person is effected differently, so its hard to know what exactly is happening. There have been people who have had auditory hallucinations within minutes of entering a sound-proof room, and visual hallucinations are very common when the brain can't figure out the static of our vision. Our brains want patterns and want things to make sense, even if that includes just making crap up.
I will say, I've been in deep bush paintball games and been scared near to death by a squirrel farting or a mouse tripping. Its like the brain looses calibration and just cranks the amp to 11 to try to hear something, then a paintball going past sounds like a nascar. And the other day I had to set up a ballroom for a wedding. White drape around the room covered in white mesh. Lit from behind with white lights, white tables and chairs, and white flowers. So many shades of white that my eyes couldn't really figure out what was going on, and even the color in the carpet started to fade, and the room looked hazy. The pic I took on my cell phone is a rainbow, LED lights are blue, ceiling halogens are yellow, tables are gray. But it was enough to screw up my eyes.