Well.....the speed of the attack on the town should hardly be an excuse to downplay blackbeard's power. Besides, even though it was only a few frames in the manga, it couldn't have been that much faster time-wise, because I believe most of the villagers still managed to escape in time.

Anyway....Blackbeard's power doesn't seem too impressive mostly because it's significance is greatest against logia type. If you take Enel, for example, who seemed to have no physical weakness throughout most of skypeia...he seemed like a true example of an elite opponent.
Of course luffy just happened to be the only devil fruit user capable of harming him, but you could attribute that to luck, if you allow suspension of disbelief of the incident being completely circumstantial. However Enel would not have stood any better of a chance against Blackbeard as Ace did.

While with Luffy, the victory lied in a rock/paper/scissors match, however Blackbeard turns every match into a rock vs rock match....with the scales tilted a little in his favor as the added bonus of still having his powers. This is especially significant if you consider that most of the deadliest characters in the series are the deadliest because of their powers (I have no idea how Luffy could ever hope to stand up against Smoker or err...that ice guy)