From what I understand, it's the culture. Watch VICE documentaries on Japan's sex culture. No one wants to have kids when the price of living is high, 80 hour work weeks, both male and female are the work force, and sex is viewed as a non-romantic "come and go" as you please pass time. Having kids in Japans high level business/work culture is a tremendous setback. They can't afford that kind of downtime.