A major focus in evolutionary ecology lies in explaining the evolution and maintenance of social systems. Although most theoretical formulations of social system evolution were initially inspired by studies of birds, mammals, and insects, incorporating a wider taxonomic perspective is important for testing deeply entrenched theory. Here, we review the contribution of studies of habitat-specialist coral reef fishes to our understanding of the evolutionary ecology of animal social systems. These fishes are ecologically similar but display remarkable variation in mating systems, social organization, and sex allocation strategies. By reviewing recent research, we demonstrate their amenability for experimental testing of key concepts in social evolution and for generating novel insights, including the ultimate reasons for female reproductive suppression, group living, and bidirectional sex change. Habitat-specialist reef fishes are a tried and tested group of model organisms for advancing our understanding of the evolution and ecology of social systems in animals.