I do it all the time. Just like you, I am not happy using the washer to ruin my expensive clothes. I prefer to handwash them for the same reasons as yours. I could choose to dry clean too but then you don't want to pay $5 to dry clean your Ralph Lauren polo each time you wear it.
Taking this a step ahead, I also am particularly careful ironing these 'designer' clothes. I almost always iron them inside out to avoid the 'shine' that a steam iron can leave on the clothing. I hate those little shine marks.