If the one being slapped is a stronger fighter than you are and very obviously stronger than the slapper, what business could you possibly have to interfere? Well, other than in the "get a room, don't make a public spectacle" kind of way. One thing is clear: You aren't defending or saving anybody. Unless, of course, Ayato wants to claim the geezer's role and take care of Toudou for the rest of her school days (and possibly the rest of her days, being the gentleman he is). Poor Julis in that case!
There's simply no intelligent excuse for Ayato not to know after all this time. He isn't anymore a transfer student of the same day. He's simply totally disinterested unless it momentarily suits his ego.