I know EXACTLY how you feel. I used to walk under a subway under a railway line on my walk back from UCT. I never went into the subway if there was anyone under there - be they black or white. One day I was in my own world and got half way down when I noticed three coloured guys in there. I didn't want to look like a racist, so instead of going back out and waiting until they'd passed through (what I used to do, receiving glares), I carried on. I got to the bottom and the three of them mugged me. Hurray.
They had been waiting at the bottom for an easy target.
I suppose that the bottom line is that you can't change that little voice in your head, but you can change the way you act towards others.