You can't compare the cost of a fragrance to the cost of a vacation, they work on very different levels. Apples and Oranges.
I have trips that stand in my memory as very significant, I wouldn't trade them for anything still, even though they are over. I don't want that money back, it wasn't a waste.
But you're right about it being a luxury, no one needs even one bottle. The only value will be to yourself, as with any hobby. Other people can't put a value on someone else's interests. My parents ask me all the time "are you still wasting money on all that cologne?" I have to politely say "no, I've never wasted any money on any of my fragrances." I can't counter them with anything though, they aren't the collecting type. Unless you count my Dad's continued desire to take college classes even though he's retired now, he collects knowledge. I would call that a waste of time, but that's for myself, not him.
One thing is certain, depending on how deep you get into fragrance collection, it has the option of being very VERY expensive.
But, you can't put a price on memories. Especially holidays.