I think most of the 'infamous' animal-derived ingredients such as musk and civet have been phased out now, though be careful with vintage frags.
Ambergris is whale vomit, so I can't see why you'd have a problem with it in theory. I have seen mentions of 'beach-harvested' ambergris, so that makes me suspect there might be more than one way of obtaining it.
Interestingly, as a vegan, you might have more difficulty with honey in fragrances. Some vegan friends of mine eat honey, others don't. Ditto beeswax.
I don't have a clue about other ingredients in perfumery - the iso-E super molecules and suchlike. And I have no idea where these things originate from. However, I have no doubt the clever people on this board will know about them and be able to warn you away from any particular ingredients that are animal-derived. (And I bet there will be some.)
I just checked on the Fragrance Industry Discussion board and there's a thread on there about musk from musk deer that you might find interesting. Hirch Duckfinder posted a link to this website which I've just been looking at and which has pages about musk and ambergris.