Exploring the driving mechanism and factors influencing farmers' decisions regarding farmland abandonment is fundamental for the targeted control of land abandonment and the adjustment of land management policies, especially for ecologically fragile areas. This study used Guyuan, a typical hilly and gully region, as a study area and employed the framework of a participatory assessment workshop, key informant interviews, and questionnaires to obtain data from stakeholders at multiple levels. The descriptive statistical analyses were carried out from social, economic, and natural aspects to explore the factors affecting farmers' decisions to abandon farmland in the hilly and gully regions. By combining multiple levels of stakeholders and multiple sources of data, the multi-spatial-temporal analysis effectively allowed us to conduct a comprehensive and thorough exploration of the potential factors affecting farmers′ land abandonment decisions using this framework. The results revealed important aspects of the main social, economic and environmental factors. (1) Among the social factors, the influence of neighbours is the most influential, owing to the herd mentality of farmers and the negative impact of abandoned farmland on surrounding farmland. Additional factors are declining employment opportunities and grain subsidies. (2) The main economic factors affecting farmers' abandonment of farming are land productivity, farming costs, and grain price. (3) The main environmental factors include road accessibility, slope and terrain relief, and farmers living in hilly and mountainous areas are more sensitive. This research provides comprehensive knowledge about the trade-offs associated with land abandonment and local stakeholders in the Loess hilly and gully region of China, a reference for finding possible pathways to halt the negative impacts, and a solid foundation for the statistical and spatial model building to simulate the abandonment scale, spatial-temporal evolutionary process, and the risk of abandonment for political reference.