Welcome!
Hedonist and alfarom's definitions should be the ideal ones. But niche in general is used simply to mean restricted distribution, ie stuff you won't find at Macy's or department stores, and often (but not always) from firms that specialize only in perfumes. Many niche houses are low quality. So don't assume that niche is better than mainstream, use your nose.
In NYC, reference niche stores are Aedes de Venustas, Min, and certain counters at Barneys, Henri Bendel, and Bergdorf Goodman (many niche houses also have standalone stores).
About the brands, everybody will have their own favorites, and there are of course dozens of firms. Perhaps the most famous, and one of the earliest, is Serge Lutens. The first, programmatic fragrance was Feminite de bois, which, as the name suggests, brought woods (cedar) into women's perfumes. In general his style veers towards big, spicy, dried fruity, theatrical orientals. Unfortunately, many of his perfumes are sold only in his boutique in Paris.
The oldest niche house is perhaps Diptyque, characterized by a fresh, natural style (reference: l'Eau de Diptyque). Another older one is l'Artisan Parfumeur (on average, but not always, characterized by a certain lightness). Amouage was instead the first firm in the west to go into superluxury (it was, and likely still is, owned by royals in the Gulf); unfortunately recent reports suggest that some of their former glories (Amouage Gold being the signature, an enormous, oversized floral) are being watered down, so one should test and proceed with caution here before dumping hundreds of dollars into it.
Incidentally, most niche firms avoid the distinction man/woman, all their offerings are described as unisex. So, again, use your nose to find out what is for you.
cacio
