It doesn't seem like he was interested or had a relationship before he was transferred to this world.
His hatred is stronger for the moment, it's not like his mind is available for love or relations relying strongly on trust.
It's entirely possible he sees himself as a father figure for Raphtalia and Filo, for age reasons and also because he's like a family support at the moment.