I do this. I have also made it a point to buy something new when I travel.
I wasn't the one wearing it, but Aimez-moi brings back more memories of Paris than all the pictures I took.
I'd estimate that about half of my bottles have a strong and specific geographic association for me. They also have a time; Las Vegas has been Rive Gauche, Terre D'Hermes, Magnetism, and 10 Corso Como for me. Each time association might as well have come from a different place.
Feeling some strong nostalgia now...