For me personally, no color is "too feminine" (or too masculine, young/old, casual/formal, ...) to feel comfortable on me. There are colors that don't work well with my skin tone, but that's another issue.
However, there are definitely cuts of clothing that I can't pull off. No matter what Jean Paul Gaultier might think, I'm not going to be sporting a skirt anytime soon. Here's a more extreme case: if you're a man, wearing a floor-length Valentino gown isn't "pulling it off," it's cross-dressing or drag.
Fragrance, in my opinion, falls in between these cases. Some fragrances marketed to women (e.g. Kingdom) would be as comfortable for me to wear as a pair of pink trousers. Others (e.g. Guerlain Cherry Blossom) are more like a pink tutu -- nice, maybe, but not for me. [Edit: Still, if I had to choose between spending a day at work wearing Cherry Blossom or a tutu, the fragrance would be an easy choice.]
I don't really feel compelled to match my fragrance to my clothing choices. In fact, I like to wear formal fragrances like Bois de Portugal with jeans and a t-shirt. I suppose on the rare occasions that I dress "formal," I'll skip my sportier colognes -- if I'm dressing up, it's for a special event, and I'll pick a fragrance that suits the event as well.