Likely the musk. Musk in niche fragrances can be very harsh and dirty. I can't see how any of the other notes can smell dirty at all. Stronger musks tend to have a mossy, dirty, earty and sometimes animalic smell.
Haven't tried IB personally, so I can't say for sure though.