One year ago, I had one bottle. I went to get a replacement for it, and discovered what was out there. Within a few weeks, my collection was up to 11 and growing. Since I was still interested in others, I set a mental limit of around 30 - one for every day of the month is how I justified it. Then when it kept growing, I thought: "ok 60 would be the max"... Then I set my limit at 100. No way I need more than that!
Well, now, one year into this, I have over 130 full bottles, many splits, and a few hundred samples. My goal is still the same: to stop. My point is, how many is too many depends on you personally. To me, it's not about needing to justify if I will ever use a full bottle of it or not, it's about if I really like the way it smelled - if so, I added it to my collection. I have some I could live without, and most of my samples will slowly fade away, but I am sure over this next year I will continue purchasing more (but hopefully at a much slower pace), and will get a few more samples - but I feel there is less to get and less to explore as time goes on, so that is why I think my collection will soon reach a max number - and maybe shrink sometime soon.
I realized I needed to stop thinking of a specific number - and rather set a spending limit / month and try to stick to that to curb my obsessiveness. I guess you should stop if you can't afford to buy any at the moment, or when nothing else intrigues you.