Although Sword and Bow did allow Naofumi to be throw to the wolves once, they ought to still possess a lingering doubt about the wisdom of allowing one of only four heroes in the world to be treated like shit. It's still four vs the whole world if it comes down to it. In other words, if Naofumi being a hero means nothing, then what guarantee is there that the world wouldn't get tired of the bow or sword as well, should something happen beyond their power? Naofumi's case shows simply being a "hero" doesn't carry any sacred prestige.

On the other hand they saw Naofumi could kick the Spear Hero's ass, despite receiving zero support from the crown. That would make them wonder if it was smart to make an enemy out of Naofumi.