Since the cytogenetic identification of azoospermia factor regions 40 years ago, the Y chromosome has dominated research on the genetics of male infertility. We hypothesized that hotspots of structural rearrangement, which are dispersed across the genome, may mediate rare, recurrent copy number variations (CNVs), leading to severe infertility. We tested this hypothesis by contrasting patterns of rare CNVs in 970 Han Chinese men with idiopathic nonobstructive azoospermia and 1661 ethnicity-matched controls. Our results strongly support our previous claim that sperm production is modulated by genetic variation across the entire genome. The X chromosome in particular was enriched for loci modulating spermatogenesis—rare X-linked deletions larger than 100 kb were twice as common in patients compared with controls (odds ratio [OR] = 2.05, P = 0.01). At rearrangement hotspots across the genome, we observed a 2.4-fold enrichment of singleton CNVs in patients (P < 0.02), and we identified 117 testis genes, such as SYCE1, contained within 47 hotspots that may plausibly mediate genomic disorders of fertility. In our discovery sample we observed 3 case-specific duplications of the autosomal gene MAST2, and in a replication phase we found another 11 duplications in 1457 patients and 1 duplication in 1590 controls (P < 5 × 10⁻⁵, combined data). With a large, polygenic genetic basis, new ways of establishing the pathogenicity of rare, large-effect mutations will be needed to fully reap the benefit of genome data in the management of azoospermia.