Do NOT feel guilty for assuming someone might be dangerous and treating him coldly. Get over it. An innocent and decent man easily understands a man or woman's safety concerns in a vulnerable location.
We don't need to be aggressive toward someone to make them feel unsafe. We are unknown to them and therefore do not present a "safe" feeling. It is only natural to avoid eye contact and not talk to strangers...we even teach out children not to talk to strangers. We want them safe. We want everyone safe.