Button-down = less formal (than straight collar). So actually I think that would be quite suitable for the layered look you propose. Very preppy.
But think also of grey button-down under that polo. Tone-on-tone.
Or bold colour: rich blue, bright green, even orange or pink if you can pull it off.