I suppose you could have an allergy to something contained in lavender oil, or it could have been the strength of the oil. I don't usually have skin contact with the concentrated oils. Many, many men's fragrances have lavender in them, so be careful. Looks at the pyramids to see if a frag may have lavender in it.
You can also try applying to fabric.