I think the answer is pretty simple: a lot of the way people distinguish "masculine smells" from "feminine smells" or vice versa is directly related to perfume marketing that we've all been exposed to.
For example, most people take it for granted that Brut, and all of the fragrance notes it contains, is a masculine scent. That's how Faberge wanted to market it: as a machoman scent. Consequently, other perfumes that have similar notes as Brut will often be regarded as "masculine".
Now what if they decided to market Brut as a woman's fragrance? If it turned out that women wore Brut, it would have become a hit, and I believe our ideas of what a "feminine" scent is could have been turned on its head.
Again, I picked Brut as just one example.