That's just fine. The royal court has given him nothing but grief and trouble, on multiple occasions and right from the beginning. It's not just the princess, it's also the king and the rest of the folks there as well. In fact they made the whole capital city view him dimly. Let's not forget the first princess as well treated him better initially, all to trick and entrap him later, naturally. So, what possible reason could he have to listen to this second princess? Even if she wasn't trying to make him fall into another trap, just associating with her means being in closer contact with the rest of the bastards. It ended up being one of his biggest mistakes to return to the castle after the previous wave, to get his minor portion of the reward money. Definitely not worth it. It even made the fricking pope crack jokes about it.