Ahh lol, completely forgot that little detail :P
Guess it isn't that much of surprise that "powerless" people can become super powerful now that i think about it, most of the 'swordsmen' in OP seem to not have fruit powers but are instead just crazy skilled and powerful anyway, so i guess training and willpower can get you pretty far without the need of fruit powers.