The most common sexual system in animal-pollinated plants is hermaphroditism, while some species are dioecious or gynodioecious and a very few are androdioecious. In this paper, I attempt to explain this pattern by extending previous models for the evolution of sexual systems to incorporate two main features: (1) a portion of investment in pollinator attraction contributes to only female or male function, because one sexual function of a flower is saturated with pollinator visitation earlier than the other sexual function; and (2) there are trade-offs between the size and number of flowers. The analysis was conducted to determine the conditions when females and males can increase in frequency in a hermaphroditic population, assuming either concave or convex pollinator gain curves (relation between investment to attractive structures of a flower and frequency of pollinator visits to the flower). The results suggest that both of the main factors play important roles in the evolution of plant sexual systems: uneven contribution of pollinator-attractive structures and nonlinear trade-offs between flower size and number can destabilize hermaphroditism. When a convex pollinator gain curve was assumed, the effect of nonlinear trade-offs can produce accelerating compensation from the elimination of one sexual function, allowing males to increase for large regions of parameter space, where females could not increase. The last prediction obviously conflicts with the observed rarity of androdioecy in nature, indicating the necessity of exploring pollinator gain curves in more detail.