Gene flow between crop fields and wild populations often results in hybrids with reduced fitness compared to their wild counterparts due to characteristics imparted by the crop genome. But the specifics of the evolutionary outcome of crop-wild gene flow may depend on context, varying due to local environmental conditions and genetic variation within and among wild populations and among crop lines. To evaluate context-dependence of fitness of F₁ hybrids, sunflower crop lines were crossed with nine wild populations from across the northern United States. These crop-wild hybrids and their wild counterparts were grown under agricultural conditions in the field with and without wheat competition. Hybrids were far less fecund than wild plants, yet more likely to survive to reproduce. There was considerable variability among wild populations for fecundity and the specific crop line used to generate the crop-wild hybrid significantly affected fecundity. The fitness deficit suffered by crop-wild hybrids varied by population, as did the rankings of the crop-wild hybrids from three different crop lines. Wheat competition decreased fecundity and survival considerably and hampered seed production of wild plants more than that of hybrids. Genotype × environment interactions indicated that the response of fitness to competition differed by population. Consequently, the fitness of hybrids relative to wild plants varied considerably among wild populations and was not consistent across environments. Notably, relative fitness of hybrids was greater under competitive conditions. This research is the first study of its kind to demonstrate that the consequences of crop-wild gene flow are context dependent and contingent on the genetics of the specific wild populations and the local biotic and abiotic conditions.