Yes, that would've been the most optimal thought process and decision.

But this is why I tried to use that weird example with the 5c on the ground to try and explain Naofumi's thought process, considering what we've seen of the orb until now, his status, his mindset about royality and their schemes and the situation itself.

Basically I'm thinking that Naofumi had reason to not bother and didn't question it further even though they certainly could've used additonal lines there such as "Shall I capture them, Master?" - "Why even bother, whatever they are trying to think up, they'll paint me as the villain anyway" or "No, we should see to it that we get somewhere safe, they probably called for assistance using that magical orb and this place will be full of soldiers soon"

etc. etc.