I believe Wiles' proof requires the case of n=3 (Euler), and n=4 (Fermat) separately. That is, Wiles' proof starts with n=5 for nontrivial reasons.

So it is more likely that Fermat saw n=4, and thought the rest would be similar.