Coleman,
you cleared that out perfectly.
Indie_Guy,
again I think it's not really about Nu.
I think I sort of understand your way of thinking on masculinity, and would like to add that a man always (!) has something to say, even when received those "ya wearing women's perfume!" kind of comments.
Courteous people are unlikely to come up with any of this kind.
For others, why don't you just develop a line of answering as, say, Basenoters' weapon of defence for uncomfortable situations in question.
Let's just think it over a bit.
1. Reflection time
What does matter for the perfect answer?
a. who's making a comment?
If such comment is made by a near relation/close friend then it most likely a joke and there's nothing wrong about it. If that person is just an acquaintance of yours or someone you don't know at all - that's the case that the thinking is for.
b. does the person who comments (and, probably, people participating the conversation) know of your scented hobby? Some scentaholics are quite open with their hobby, others are not, but if a person knows of you and makes a comment, that's most probably a joke too, so nohing to worry about.
So what do we got. The comment is most likely to be made by a passing acquaintance or a stranger that has no clue of your passion with scents. If you're in public (working, partying or something), let's think the public is mixed type (some people know you, some don't).
2. Possible lines, for answering questions like "YOU wearing women's perfume?" or "Who's wearing women's perfume?" (with strange looks towards you)
NOTE: of course, you shouldn't react (let alone answer) when it's a question just "into the air", not aimed at you personally.
- Could be me, I've probably mixed up the bottles in the bathroom this morning with my GF's,
- Funny you noticed, I made a bottle exchange with my GF, just for kicks,
- You think so? I have some strange feeling about my cologne too, hmmm... didn't strike me when I sprayed it on. Got it as a gift, you know, it seems colognes are much girlier these days.
- You still can smell it? A while ago I took an elevator where some woman had it too much on (cough!).
- I've been choosing a gift for my GF/wife (experts advise to try scents on skin, you know).
...it seems to me there's A LOT more of these. I'm from somewhat different background so I don't know all the nuances of humane treatment in your neck of woods.

I hope the guys who "don't give a sh!t" will add a little; those who do might want to be of help too.
Regards, Odor.