Help please this problem has been bothering me for a while someone keeps messing with me?

So a while back i was walking home with my gf and there was this guy who lives a couple house down from me and hes like some wanna be gangster idiot who thinks he tough anyways as we passed by he said ''damn girl'' to my gf so yea he was just trying to disrespect us anyways i went up to him and i told whats his problem and hes like dont worry about and i just said whatever man screw u so yea i regret not fighting him that day because he always stares at me andi think he thinks im afraid what can i do to get him to stop staring at me every single damn time he sees me idk whats his problem with me. everytime i see him i get filled with anger and that feeling you get right before ur about to fight im not really scared of him im like 2 years older than him im taller than him and i work out its notice able what can i do so he'll leave me alone he lives a couple house down from me so i see him alot should i fight him? and i know i know fighting doesnt help if i do fight how do i get over this nervous feeling i get?