Well, you don't even have to go by the manga. in the anime, jirayai talks about the 4th a lot. and he clearly stated that he was a man.