Absolutely, I do the same thing. I posted a thread on this a long time ago. I actually feel like it clashes to wear the wrong colours with the wrong fragrance!
I don't know if my main influence is the colour of the bottles or the actual properties of the fragrance.
A few of my combos
GIT goes with purples and blacks
Cuir Mauresque for oranges and reds
Vetivers go with Browns and Greens
Things like Bois du Portugal and Equipage also browns
Black Aoud with purples and blacks
Chergui, Red Vetyver, Fou d'Absinthe with reds
Tea for Two with greens, browns, reds
Encens et Lavande and Gris Clair with turquoise, grey, blue
I could waste all day on this!