Do you have an idea of a site that you consider fast? I just checked a few sites around the web, and 1.x seconds seems pretty average (I'm looking at the onload time, as that seems like a somewhat-maybe scientific way of looking at things.)
Most Google sites (less so lately since there is more Ajax), craigslist, reddit of 4 years ago, HN of 2 years ago. Basically all the bare-bones content-driven sites, since people tend to write JS and CSS in a way that makes the site at least 100% slower.
Amazon is pretty decent. Their full page loads are slow, but they concentrate on above-the-fold time which is OK. I don't use Facebook much, but I think they have pretty good latency.
I think Microsoft's new mail is pretty fast but I haven't used it much.
I would say 1.x seconds for initial HTML is average. And 3-4 seconds for the full page with all assets is average. Github is in the 3-4 seconds range for onload time.
So Github is not particularly bad -- it's just average. It's certainly not "fast".
I feel like Github's perceived latency is pretty bad. I think they end up showing spinners and redrawing various parts of page to try to hide latency, but it makes it seem janky to me.