Many obligately intracellular symbionts exhibit a characteristic set of genetic changes that include an increase in substitution rates, loss of many genes, and apparent destabilization of many proteins and structural RNAs. Authors have suggested that these changes are due to increased mutation rates, or, more commonly, decreased effective population size due to population bottlenecks at the symbiont or, perhaps, host level. I propose that the increase in substitution rates and accumulation of deleterious mutations is a consequence of the population structure imposed on the endosymbionts by strict host association, loss of horizontal transmission and potentially conflicting levels of selection. I analyze a population genetic model of endosymbiont evolution, and demonstrate that substitution rates will increase, and the effect of those substitutions on endosymbiont fitness will become more deleterious as horizontal transmission among hosts decreases. Additionally, I find that there is a critical level of horizontal transmission below which natural selection cannot effectively purge deleterious mutations, leading to an expected loss of fitness over time. This critical level varies across loci with the degree of correlation between host and endosymbiont fitness, and may help explain differential retention and loss of certain genes.