Mist Hashira was stated to be a direct descendant.

Tanjiro's flashback suggests that he is not blood related. Tanjiro's ancestors (Sumiyoshi) were rescued by the First Breathing dude, and in gratitude they asked if they could pass First Breathing's family name because he has no heir.

First Breather then says:
-no need, those who master their technique will end up "in the same place" anyway..
-he failed to protect what's important to him
-his name is worthless

I guess from rewatching that, it's he's actually saying that if you strive to get good / improve your swordsmanship/breathing technique, you'll eventually arrive back to the peak - which is what he discovered as First Breathing.

Also, despite the claim that he has no heir, perhaps he thought his offspring was dead when they were actually alive - thus Muichiro. That or the wife just lied to Muichiro so they'd train with her, and all he had was raw talent. If it's the latter, he's never had a chance to demonstrate it though, so they shouldn't know about that unless he IS from the line of swordsman.