It depends on what you consider "low-rise". Different parts of the city have different densities and building heights. Some parts near the center have real skyscrapers (and new ones are under construction now), and other more-central areas are full of buildings that are 5-10 stories or so. Many places are full of apartment blocks, and the newer ones are usually 10-15 stories. Some places are full single-family homes or buildings that have just a handful of apartment units in them. Of course, the taller buildings tend to cluster near train stations more, and the single-family homes tend to be farther from stations since the land is less valuable the farther from a station it is.

But yeah, "tiny roads and tightly packed buildings" summarizes Tokyo's layout to a good degree. If you look at Tokyo from a satellite view and compare with typical American cities, the biggest thing you'll probably notice is just how little space is wasted on things like wide multi-lane roads (or stroads), parking lots, and lawns. That's what makes it so walkable, and why public transit works so well here. A 5-minute walk can take you to a very large number of buildings, which equates to a huge number of apartments or businesses. A 5-minute walk in a typical American city like Houston will probably only get you across the street.