I've never had anyone challenge my hobby. "Challenge" is such an interesting word choice. Are people really challenging you? If so, why? Why does the topic even come up in the first place?
A few nights ago, I had a date with a woman who mentioned I smelled different than I had the previous time we'd met. I said I owned a bunch of colognes that I enjoy (and I used the word cologne because, in the real world, that's what people mistakenly think scents for men are). Anyway... she asked "How many do you own?" I told her I didn't know. "A bunch, really." I could tell the idea of a man collecting perfume seemed odd to her, and I could tell she was interested... so... I asked if she liked the smell of the bread aisle at a grocery store. Of course she said yes. Then I asked if she ever noticed the smell of the rain, or the smell of newfallen leaves in the fall. And, of course, she said yes. Then I said "I'm going to treat you to something really good. I'll be right back."
I fetched some cotton balls and a few bottles of perfume. I handed her a cotton ball sprayed with Gucci Pour Homme II and asked her "What do you think?" Her eyes got really wide, and she grinned. Then, I handed her a cotton ball sprayed with Gucci Pour Homme 1, and her eyes got even wider, and she didn't want to hand the cotton ball back so we could move on to the next sample. We ended up sampling a whole bunch of things and it was a ton of fun. I asked her "What kind of guy would smell like this?" and I loved getting her input. So! Much! Fun! I'm seeing her again this week and she picked the one she wants me to wear. Did I mention somuchfunsomuchfun?
That's how I handle it. If asked, I discuss my interest to the extent that the other person is interested, which brings me to the second part of my answer. I can't help noticing you've posted various threads with
this same topic, which suggests to me that your hobby is often an issue in your everyday life. Could it be that you talk about about your hobby a lot with people who aren't interested? Here's an example. I'm an audio guy. I could easily go off on a tangent about a relatively unknown kind of amplifier you can buy for as little as $25 (seriously) that outperforms rack mount pro studio amps. I know because I swapped out my rack mount for one a few years ago and... well... you're not interested, are you? I can't blame you for not having any interest at all. It's not your thing. No worries. I could also go off on how amazing the new Sony SLTs are compared to SLRs, but unless you're really into photography, comparisons between an electronic viewfinder and a pentaprism would bore you to tears. Spoiler alert: a pentaprism is an amazing pieces of glass, but compared to an EVF viewfinder (and an open mind regarding new technology), it's just a piece of glass. Among photographers, that comment is like declaring war. Here, it's irrelevant because this isn't a place to talk photography. Different audience = Different results.
My point?
It's important to know your audience. If people are often "challenging" your hobby, it means you're making your hobby an issue. Maybe you wear too much, thus causing the topic to come up again and again? Or (this is what I suspect) maybe you want their approval, so you bring the topic up again and again. That doesn't work. I bought L'Air du Desert Marocain recently and was very excited about it, but the friend I was hanging out with that day wouldn't have been interested in the least. We were drinking beer, eating food that's bad for us and watching football. It was a great day, but not a day to rave about L'ADDM. You've got to know your audience.