Cracked explains why:
You may have noticed after your little impression of the Flash that your whole body’s wreathed in fire. It’s the friction of your body rubbing against a whole bunch of air molecules. The rest of us mortals aren’t bothered by that because we move at the pace of a narcoleptic snail compared to you.
…the sense of sight works by having the waves of visible light reflect from an object onto our retinas, where it forms an upside-down image and is transmitted to the brain where it’s converted into the image of whatever you were looking at.
You being invisible means that all those light waves are passing through your body. So, sure, they aren’t bouncing off you to return to someone else’s eyes, but this in turn means that they aren’t hitting your own retina, either. So, whenever you’re invisible, you’re also blind.