The increase in extreme climate events in recent years has been considered as an important factor affecting forest pests. Understanding the responses of forest pests to climate is helpful for revealing the trends in forest pest dynamics and proposing effective control measures. In this study, the relationship between the dynamics of all forest pests, independent forest diseases, and forest insect pests with the climate was evaluated in China, and the corresponding differences among forest pests, diseases and insect pests were assessed. Based on cross-wavelet transform and wavelet coherence analysis, the influences of teleconnection factors on the relationship between climate and forest pests were quantitatively analyzed to determine the roles of these factors. The results indicate that (i) three types of disasters in most parts of China have decreased from 1979 to 2019, while forest pests and forest insect pests in the southwestern region have increased; (ii) the relationship among Forest Pest Occurrence Area Rate and climate factors such as the Multivariate ENSO index, Southern Oscillation index, Arctic Oscillation (AO), Atlantic Multidecadal Oscillation (AMO), and Sunspot is more significant; (iii) the cycle is short in most regions, with oscillations in 2–4 years bands being the main variation periods of disasters in East, Central, and South China; (iv) There is a significant correlation between climate and disasters in the periods of 2–4 or 8–10 years. The AO, AMO, and Sunspot were important driving factors affecting the relationship between climate and disasters. Specifically, the Sunspot had the greatest impact among these factors.