i would think that regular washing and light scrubbing should help keep the surface clear of bacteria.
Think about the sources of bacteria and how you might be creating an environment for them to grow (dark, wet, hot):
Do you go to the gym and sweat in a cotton t-shirt that stays damp and pressed to your skin instead of a wicking fabric? Shower afterwards?
Are you wearing cotton t-shirts as undershirts under your dress shirts?
Do you wear a shirt to sleep, or are you skin on contact with the same sheets all night, where you probably sweat a lot... how often do you wash your sheets? once a week? that's like wearing the same t-shirt for 8 hours for a week... try wearing a shirt to bed and washing/switching every few days.
just a couple of ideas