5% is a very weak dilution for something like Sandalwood which is not a strong smelling oil. The books you're referring to are probably addressing safety when using oils for aromatherapy (in aromatherapy you may be applying essential oils over a larger area of skin than you would apply a perfume).
In perfumery, you would typically dilute your composition to 15 to 25% or higher for parfum concentration, but of course you still have to be aware that certain oils can't be used above a certain percentage due to skin irritation, sensitization risk, etc.
An all-natural perfume, very generally speaking, isn't expected to last as long on your skin as one containing synthetics, and in fact 4 hours is good performance for a natural perfume. To make it last longer, try applying a bit to fabric (since you're using oil you'll want to choose an inside seam or somewhere the stain won't show) or to your hair or beard.
Another consideration is oil vs. alcohol as your diluent - you won't get much lift or diffusion with oil, it damps down many notes, so you may want to consider experimenting with alcohol as well.