Having a high average comment score helps a lot (maybe a bit too much as you've said) but AFAICT the average is only among relatively recent posts.
I'm pretty sure it is something like the average of your last $X posts that matters rather than average score of every post you've ever made. So while it may still result in a "rich get richer" situation, it isn't quite as dire as it would be if your entire post history worked into the average.
I have strong recent evidence of this in my own score, and It was an interesting experience for me. Lets just say my average was less than one for a while, but then climbed back quickly over the last 72 hours. Its an interesting system that really punishes the type of comments that get downvoted, and while I was upset about it for a few days I can see that it is a good system.
I'm pretty sure it is something like the average of your last $X posts that matters rather than average score of every post you've ever made. So while it may still result in a "rich get richer" situation, it isn't quite as dire as it would be if your entire post history worked into the average.