There were men left behind at Castle Black, as well as men at Eastwatch-by-the-Sea and the other one I don't know the name of. But this is a heavy blow for the Night's Watch. Who knows how many of these men who remain loyal will even make it back alive, and Castle Black is the great Night's Watch stronghold. Also the big problem, as you state, is that it used to be a great honor to join the Night's Watch, and now the only people who do it voluntarily are those who are lied into believing it was some grand defense of the kingdom like Jon. The rest are the worst dregs of society, hardly worthy of forming the "shield to guard the realms of men".