Why wouldn't they???
It doesn't matter that they are heroes as well.
It matters that they believe, or are being told, that the inhabitants of that world honor the heroes.
If that's the case, they wouldn't accuse them of doing something that they didn't do.
For all they know, he's just some weird virgin who wants to abuse his new power to get women.
"yeah right...your armor, sure."Naofumi shouted it's his armour, so unless they have gone deaf, they would know.