Proposers of cross-scenting (if I may call it that) argue that scent don't naturally have gender.
Funnily, colors, fibers, fabric, etc, doesn't have gender either, but people are much more willing to accept that certain clothes are "female" clothes and others are "male" clothes.
Why don't you also adhere to the "if it looks good on me I'll wear it" the same way you adhere to "if it smells good on me I'll wear it"?
Notice that only because the social norms about male/female dress codes are more ingrained in people's minds and more widely shared it doesn't mean that they are less of social norms than the gender of scents.
So, to me the argument "perfume has no genitalia" is not good to reject the thesis that males (females) wearing female (male) scents is akin to cross-dressing.
Is cross-scenting more accepted in society than cross-dressing? It looks like it is and this is why most people around here so promptly accept it and adhere to it.
Not only that, but since most people are NOT educated on fragrances as much (and from as early in life) as they are on dressing, cross-scenting becomes more difficult to detect.
Because it is more difficult for most people to detect, cross-scenting is not as big a social threat as cross-dressing and therefore many people who would not even remotely consider the possibility of cross-dressing feel comfortable cross-scenting.
This is my opinion and answer to the OP.