The maintenance of genetic variation in traits closely associated with fitness remains a key unresolved issue in evolutionary genetics. One important qualification on the observation of genetic variation in fitness-related traits is that such traits respond asymmetrically to selection, evolving to a greater extent in the direction of lower fitness. Here we test the hypothesis that standing genetic variation in fitness-related traits is principally maintained for unfit phenotypes. Male Drosophila bunnanda vary in mating success (the primary determinant of male fitness) due to female mate choice. We used competitive mating success to partitioning males into two groups: successful (high fitness) and unsuccessful (low fitness). Relative to successful males, unsuccessful males harbored considerably greater levels of additive genetic variation for sexual signaling traits. This genetic asymmetry was detected for a multivariate trait that we demonstrated was not directly under stabilizing sexual selection, leading us to conclude the trait was under apparent stabilizing selection. Consequently, our results suggest genetic variance might be biased toward low fitness even for traits that are not themselves the direct targets of selection. Simple metrics of genetic variance are unlikely to be adequate descriptors of the complex nature of the genetic basis of traits under selection.