What downsides?

The only downside shown so far is that the monster you name might kill you because he gets to be stronger than you at some point under certain circumstances

Other than that, nothing has been shown so far or even suggested.