Invasive Alien Species (IAS) are known to be an important driver of environmental changes, yet the social impacts of such invasions are understudied, particularly among vulnerable groups. In this article we study the ways Rabari pastoralists from Kutch (Gujarat, India) deal with the invasion of Prosopis juliflora, a widely-spread, invasive bush tree. First, we analyze how the Rabari pastoralists perceive the presence of P. juliflora among a range of environmental changes and problems they are faced with today. Second, we focus on the dynamics of their Local Ecological Knowledge (LEK) system by studying the knowledge and uses that are implemented in daily interactions with the tree. Our results show that P. juliflora invasion is not perceived as a major problem by the pastoralists, despite being mostly associated with negative impacts. We relate these results to several intertwined factors, such as the importance of other changes, the shifting baseline syndrome, the adaptive capacity of LEK systems, and the slow rate of environmental change directly attributable to P. juliflora invasion. We then suggest that the ability of LEK systems to adapt to environmental change may in turn influence the perception of environmental changes, such as P. juliflora invasion.