The effect of inbreeding and outbreeding depression on the evolution of assortment are often considered separately. For instance, inbreeding depression is usually thought to shape selfing rates whereas outbreeding depression is commonly thought to affect the evolution of assortative mating. In this article, we consider the evolution of assortment in a context of local adaptation and we show that it is a typical situation in which both effects act simultaneously to shape the degree of selfing or assortative mating. More specifically, we show that selection on a modifier of mating can be partitioned into three distinct effects: a transmission advantage, an association to heterozygosity (proportional to inbreeding depression), and an association to beneficial alleles (proportional to outbreeding depression), so that random mating may evolve even with strong local adaptation. In addition, we show that it is necessary to carefully delimit the conditions for polymorphism at local adaptation locus to study the evolution of assortment. In particular, the range of parameters most favorable to the maintenance of polymorphism corresponds to situations favoring less assortment.