Does it matter? I'm not talking ballcaps--I mean fedoras, flat caps, etc. Is there some rule about matching your hat with...shirt? Pants? Shoes? No rule?
As with ties, I think one can experiment and be a little bold, so there's no need to match colors.
Of course, straw or similar seem appropriate only for the summer or very warm days. And, as with all clothes, pure white can only be worn between memorial and labor day (but this applies only to pure white, yellowish or straw are not considered pure white).
I'd say, from my own little experience with hats, that colors like black and dark gray are matched with pretty much any outfit, preferably of course office style, formal etc. outfits.
Of course, during summer, warmer weather and for casual wear, I could imagine light colors like white and beige also being quite appropriate and easily accessorized.
Personally, I think matching with your shoes/belt would be considered quite tasteful. However, today's fashions are quite a bit freer than when I was growing up. I'd say that provided it doesn't overtly clash, you should be good, but I wouldn't recommend a brown hat with black shoes, for example.