I used to think the same as you, but I would suggest giving this a read before you pass off his success as luck as it's a direct rebuttal to the normal argument. Unless by luck you mean him meeting Warren Buffett.
Warren Buffett may have been lucky to meet Munger. He credits Munger with getting him off buying “cigar butts” and instead focusing on excellent businesses (pricing power, barriers to entry, etc) at fair prices which is largely where Buffett’s wealth expanded the most.
They're both lucky, and I'm not a fan of Warren Buffett either. He constantly throws out his "money has no utility to me" as part of his "aww shucks, I'm from Nebraska" schtick, which is a slap in the face of the other n-billion people poorer than him, less lucky than him, and to whom money does have utility, as in life or death utility.
Jim Simons has mentioned explicitly about how most people underestimate the role of luck in success, and he attributes quite a bit of his success to luck.
https://www8.gsb.columbia.edu/sites/valueinvesting/files/fil...