The true punishment was always for Vesta. He even took the thing Vesta most wanted from him, by stating he was never to show his face to the king ever again, but didn't exile him. He said it pretty clearly, he waited years for Vesta to admit he was the one at fault, and framed Kaijin.
I think the king knew Kaijin was going to leave no matter what. The insultingly impossible task Vesta gave him was sure to drive him away one way or another. Frustration, exhaustion, or forced disgrace. The whole trial was a farce, designed to give Vesta one last chance to repent. He didn't.
It's also noteworthy that Vesta was shocked when Kaijin was exiled. That's not the outcome he wanted. He wanted Kaijin around, he's a huge asset to the kingdom. Like Kaijin said, Vesta is a good man at heart, who believes in the good of the kingdom. He just lost his way for some reason. Maybe it was resentment, maybe it was frustration, maybe he just lost his way.
Rimuru was just there kind of, when you take a step back and look at this as mostly being about dwarf politics, which it was.
Rimuru's existence alone unnerves the king, not the fact that Kaijin is going to the slime. A slime shouldn't be that powerful. Shouldn't be able to produce perfect potions when their research has gotten them only close after decades. Someone that powerful shouldn't have appeared without his spy network knowing about Rimuru before he came to the city unannounced. And from the earlier episode...everyone know Veladora disappeared. Add all those pieces together, and any competent leader who is labeled the Hero King, and has been around as long as he has should be suitably freaked the fuck out.