Specificity in parasitic interactions can be defined by host genotypes that are resistant to only a subset of parasite genotypes and parasite genotypes that are infective on a subset of host genotypes. It is not always clear if specificity is determined by the genotypes of the interactors, or if phenotypic plasticity (sometimes called acclimation) plays a larger role. Coevolutionary outcomes critically depend on the pervasiveness of genetic interactions. We studied specificity using the bacterial parasite Pasteuria ramosa and its crustacean host Daphnia magna. First, we tested for short-term adaptation of P. ramosa lines that had been rapidly shifted among different host genotypes. Adaptation at this time-scale would demonstrate the contribution of phenotypic plasticity to specificity. We found that infectivity was stable across lines irrespective of recent passage history, indicating that in the short term infection outcomes are fixed by genetic backgrounds. Second, we studied longer-term evolution with two host clones and two parasite lines. In this experiment, P. ramosa lines had the possibility to evolve adaptations to the host genotype (clone) in which they were serially passaged, which allowed us to test for a genetic component to specificity. Substantial differences arose in the two passaged lines: one parasite line gained infectivity on the host clone it was grown on, but it lost infectivity on the other host genotype (this line evolved specificity), while the other parasite line evolved higher infectivity on both host clones. We crossed the two host genotypes used in the serial passage experiment and found evidence that the number of host genes that underlies resistance variation is small. In sum, our results show that P. ramosa specificity is a stably inherited trait, it can evolve rapidly, and it is controlled by few genes in the host. These findings are consistent with the idea of a rapid, ongoing arms race between the bacterium and its host.