In pea, high temperatures during reproductive development lead to severe yield loss. Although the ovule is the seed precursor, studies elucidating the effect of heat on this plant structure are scarce. We investigated the impact of heat in the field and growth chamber on ovules 4 days after the open flower (4DOF) stage. Objectives were to identify associations between ovaries and plant performance, and to evaluate seed set and ovule abortion of heat-treated plants for six cultivars from a diverse range of seed-to-ovule ratios. In the field, plants were seeded at early (control, [early seeded pea, ESP]) and late (stress plant [late seeded pea, LSP]) periods in the season. In growth chambers, plants were exposed to heat (35/18 °C) at early flowering for 4 days and then evaluated at maturity. Stressed plants (LSP) displayed twice as many aborted ovules than ESP during early embryo growth (pro-embryo to globular stage) in synchrony with reduced ovaries, ovules, and embryo sac size. Cultivars with reduced ovary size at 4DOF were related to a high number of reproductive nodes and pods in LSP (r = −0.44 to −0.48). Similarly, under growth chamber conditions, heat caused seed reduction by increasing the abortion of immature ovules (early embryonic stages) at various reproductive nodes. Collectively, our results indicated that pea seed loss from heat in the field is largely due to early embryo abortion, a novel finding, rather than disruption of pre-fertilization events. Compensatory effects on plant performance infer plant resource adjustment. Our findings contribute to the assessment and selection of high-yielding pea cultivars for future warming seasons.