I've seen a lot of men wear v-necks, and it often looks "off" somehow; they look self-conscious. Also, the v-neck visually breaks the line across the shoulders, snipping out a wedge and making the shoulders look narrower, which most men prefer to avoid.
But try it. If you look at your reflection and love it, well, it works for you.
If you find yourself trying to work out its relative merits and drawbacks intellectually, take it off.