Glad to here you are embracing the DE safety razor. I think in time you will find that as your technique improves, your skin will feel much better. I am a firm believer that the multi blade cartridge razors are harsh and irritating on your skin.... partly because of the multiple blades all at once on your skin..... but mostly because I feel guys are less likely to use a sharp blade because the cartridges cost so much that they try to use them as long as possible (which is usually too long).
To answer your question, simply rinse well. I use all different kinds of shave soap and cream.... some lightly scented and some heavily scented. I have never used one that the scent of the soap or cream lingers very long after the shave when rinsed well after the shave. I have a post shave routine that consists of a warm water rinse, a cold water rinse, spraying witch hazel on my face and neck, applying an aftershave lotion (splash), and then an aftershave balm. It is overkill.... but it works for me. The aftershave scent can sometimes linger depending on what I use..... but shave soaps and creams typically do not contain enough fragrance oil for the scent to stick around very long. If it is, you just need to rinse more after your shave.
You could also try an unscented soap or cream.
Hope this helps.