Want to learn to fight/ self defense... One small dilemma?

My Sis and I want to start learning Martial arts &/or kickboxing (not the cardio one, the actual fighting one) anyways I'm worried cause even if we find a good place I don't want to be made fun for being short (5'1") and female. My sister isn't worried b/c she's got plenty of confidence. Not looking to go pro or anything but I just can't get past the thought of that. Be honest, am I just being paranoid or do I actually have a reason?