Well, I myself see the sports of aspects of the manga as a mere plot device to move the story forward. The actual drama and story takes place outside the sports portion. For example, the Adachi mangas though they have a heavy dosage of sports, the actual story of interest are friendship, romance, love triangles, and how the characters overcome their own adversities, etc. So yeah, I think there's a bit of misconception on what these series have to offer. For example, I don't care nor do I know how to play Mahjong and Go, still I found the story itself for Hikaru no Go and Akagi very good.

So yeah, I think your reasons given are very valid and very true as to why they don't read sports manga or watch sports anime. I'd say that most that are not already attracted to sports finction in general won't enjoy most that is out there, still I believe that there's a handful of series that would be of interest if only they didn't have sports as a genre and they give them a chance. Major, for example, has some very powerful dramatic scenes that are outside the sports portion, so I think that would be something that some would like even with the heavy dosage of baseball in the manga, I think once the story progresses you would get used to the sports portion and would be genuinely intrigued by the story itself.