Pretty safe since I don't have any brass stuff I carry around.There are different types of brass but the stuff used in common keys can be up around 2.5% lead. Small amounts can rub off upon surface contact. It isn't huge but remember, no amount of lead is good for you. This is why many brass keys are now coated and we have advisories to not let small children play with them.
Exactly how dangerous this is will depend on a lot of factors. For instance, how much lead is in your EDC brass? How often do you fondle your brass gear? How often do you touch your face or eat with your hands after using that gear? How often do you wash your hands after using your gear? What else gets stored in those pockets? An important question is how much lead you are picking up from other sources of exposure since it is cumulative.
I do have some copper stuff.