Ah, I just realize what it is that always tips me off on filler characters.

Its robes.

Almost every ninja in the Naruto world wears those vests. But the filler characters always seem to be wearing some kind of plain kimono.