Understanding speciation requires discerning how reproductive barriers to gene flow evolve between previously interbreeding populations. Models of sympatric speciation for phytophagous insects posit that reproductive isolation can evolve in the absence of geographic isolation as a consequence of an insect shifting and ecologically adapting to a new host plant. One important adaptation contributing to sympatric differentiation is host-specific mating. When organisms mate in preferred habitats, a system of positive assortative mating is established that facilitates sympatric divergence. Models of host fidelity generally assume that host choice is determined by the aggregate effect of alleles imparting positive preferences for different plant species. But negative effect genes for avoiding nonnatal plants may also influence host use. Previous studies have shown that apple and hawthorn-infesting races of Rhagoletis pomonella flies use volatile compounds emitted from the surface of fruit as key chemosensory cues to recognize and distinguish between their host plants. Here, we report results from field trials indicating that in addition to preferring the odor of their natal fruit, apple and hawthorn flies, and their undescribed sister species infesting flowering dogwood (Cornus florida), also avoid the odors of nonnatal fruit. We discuss the implications of nonnatal fruit avoidance for the evolutionary dynamics and genetics of sympatric speciation. Our findings reveal an underappreciated role for habitat avoidance as a potential postmating, as well as prezygotic, barrier to gene flow.