I was under the impression that, before he was critically injured, All Might's "puffed up" form was just his normal body.
Like, he started out a normal kid, just like Midoriya. Once he received OFA, he had to train his body constantly in order to be able to use it without wrecking his body. And by the time he'd trained enough that his body could use 100% of OFA without damaging itself, he'd just grown to his All Might proportions.
But once he received his critical injury, his health began to fail. He's already stated that he can't eat properly, and I assume he also can't physically train hard anymore either. This has caused his body to atrophy to the point that it's at now. However, he was still able to take his old form temporarily, but it was basically like holding a flexed muscle.
So it's not OFA itself that makes you big. It's the physical training required to fully use OFA that makes you big. Deku can only use, like, 5% of OFAs power without hurting himself atm. So he's only, like, 5% as swole as he needs to be.