It is because of ignorance. Or it is because of your body type and vibrations.
You may have limited vibrations, or you may not be so centered that you feel comfortable and at ease with all kinds of people. You have limited range of vibrations and not the total range of vibrations.
If you are strongly centered then you are comfortable with the entire 360 degree range of vibrations, because just like in a circle, every point becomes equidistant from the centre.
But if you are on some chord of this circle, then you are not equidistant from everything. So you are not comfortable with every vibration.