I'm a bit of a fragrance hermaphrodite. Cruising through the forums, I find as many women as men wearing my favorite fragrances, and most of my favorite frags tend to be niche unisex creations.
It seems to me tuberose, gardenia and the other white flowers get classed as female, as does anything that's purely sweet and powdery. There are no exclusively male notes per se, but conifers - e.g. pine, cypress, spruce and to a much lesser extent cedar - are often considered masculine.Of course, things like the hemlock spruce in Ormonde
Woman - emphasis mine

- prove that statement at least partly wrong.
I think the division is cultural. Flowers are cliched as "female" things, soft, delicate and pretty, which are considered female traits in many cultures. Pines are big tough trees... so I think you can see where this is going. I notice gender-neutral notes are often from gender-neutral objects - spices like cinnamon, resins like benzoin, green notes like oak moss; citrus notes like orange.
I also see other cultural parallels. The decline of traditional gender roles is paralleled by the rise of unisex fragrances, as well as the olfactory division between mens and womens fragrances shrinking.
While there are likely currently unknown biological factors in how men and women perceive smell, I do believe that gender in fragrance is almost entirely based on our cultural attitudes towards gender.
As for myself, I've never had trouble from others wearing tuberose... of course I am 6'8" and not in the least feminine, so that probably helps.
