Selfish genetic elements (SGEs) are ubiquitous in animals and often associated with low male fertility due to reduced sperm number in male carriers. In the fruit fly Drosophila pseudoobscura, the meiotic driving X chromosome “sex ratio” kills Y-bearing sperm in carrier males (SR males), resulting in female only broods. We competed SR males against the ejaculates of noncarrying standard males (ST males), and quantified the number of sperm transferred by SR and ST males to females. We show that SR males are very poor sperm competitors, which is partly related to transfer of fewer sperm during mating. However, sperm numbers alone cannot explain the observed paternity reduction, indicating SR males’ sperm may be of reduced quality, possibly due to damage during the killing of the noncarrying Y-sperm. The reduction in sperm competitive ability due to SR is large enough to potentially stabilize the spread of sex ratio drive through populations. The poor sperm competitive ability of SR males coupled with their low fitness as mates could favor increased remating by females to reduce paternity by SR males. Given the generally poor performance of SGE-carrying males in sperm competition, this may generate strong selective pressure favoring polyandry in many species.