I'd probably get mad myself. I'm an OCD/germaphobe person, and in general I don't like people touching my stuff.
Then again if I was expecting a stranger employee to be in my personal area while I'm not there, I'd prob. lock/close doors, put away stuff that I wouldn't want to go 'missing' and stuff before the person even arrives.