Maybe the fan love is actually because of the game. I watched the anime first, and I loved Saber and Rin, but that was about it. I hated Shirou and found the anime itself to be mediocre. But after playing the game, I found myself liking it much more than before.

If I had to say something about the anime alone, maybe people simply like some of the characters a lot, and that translates to being a fan of the show in most cases.