What I mean is, the very fact that Shichika is human is the poison. It is because he is a human sword that his user Togame fell in love with him.

I was speaking in a figurative sense rather than a mystical sense when I used the term poison, but I bet that is what the author is going for as well.