Weeds remain the most commonly cited concern of organic farmers. Without the benefit of synthetic herbicides, organic farmers must rely on a host of ecological weed management (EWM) practices to control weeds. Despite EWM's ability to improve soil quality, the perceived rate of integrated EWM strategy adoption remains low. This low adoption is likely a result of the complexity in designing and evaluating EWM strategies, the tendency for outreach to focus on the risks of EWM strategies rather than their benefits, and a lack of quantitative measures linking the performance of EWM strategies to farmers' on-farm objectives and practices. Here we report on the development and deployment of an easy-to-use online decision support tool (DST) that aids organic farmers in identifying their on-farm objectives, characterizing the performance of their practices, and evaluating EWM strategies recommended by an expert advisory panel. Informed by the principles of structured decision making, the DST uses multiple choice tasks to help farmers evaluate the short- and long-term trade-offs of EWM strategies, while also focusing their attention on their most important objectives. We then invited organic farmers across the United States, in particular those whose email addresses were registered on the USDA's Organic Research Integrity Database, to engage the DST online. Results show considerable movement in participants' (n = 45) preferences from practices focused on reducing weeding costs and labor in the short term to EWM strategies focused on improving soil quality in the long term. Indeed, nearly half of those farmers (48%) who initially ranked a strategy composed of their current practices highest ultimately preferred a better-performing EWM strategy focused on eliminating the weed seedbank over 5 yr.