They never used it because it was supposed to be a hidden palace of refuge. Muzan's goal ultimately was to have immortality. Destroying the demon slayers would help, but ultimately it probably doesn't bother him all that much. Like he says, he's got time.
The Biwa demon also stays in the castle, so if they got a Hashira in there and killed her then that's a huge asset lost to Muzan without much gain. Killing a Hashira or two doesn't mean much to him.
Now. Now that Nezuko is sun-resistant, Muzan has something to play for. Demon Slayers are hiding her, so he's now out to wipe them out in order to get to her. When he had the other demons out and about to infect people + hopefully find/develop into a cure, the castle was mostly empty.