He's a miserable sucker, and the shopkeeper woman knows exactly how to handle such a sucker. You can't actually say it wouldn't be realistic.
I can understand why the sister was so disappointed when he failed the sprint and peed himself, but it's indeed quite a stretch she would be so disappointed by the dude quitting the terrible game. Rather, you'd think she would have used the opportunity to have the dude quit all the games and go back to sports. It would be a bit too deep to have the younger sibling be proud of the brother as long as he can dedicate himself to anything.
I guess it was there mainly to explain the depth of his trauma. However, the flashback scene didn't really have anything that would make it seem like it was just his perception of what happened. It was presented as objective flashback, which makes the other people's reactions a bit off, like you said. Unless he indeed was an arrogant bastard and nobody liked him. So, when he failed, they were all not-so-secretly glad he did. Except for the little sister who had decided to externalise her source of pride to her brother.