Abstract: The human visual system is able to distinguish naturally occurring categories with exceptional speed and accuracy. At the same time, it exhibits substantial plasticity, permitting the seamless and fast learning of entirely novel categories. Here we investigate the interplay of these two processes by asking how category selectivity emerges and develops from initial to extended category learning. For this purpose, we combine a rapid event-related MEG adaptation paradigm, an extension of fMRI adaptation to high temporal resolution, a novel spatiotemporal analysis approach to separate adaptation effects from other effect origins, and source localization. The results demonstrate a spatiotemporal shift of cortical activity underlying category selectivity: after initial category acquisition, the onset of category selectivity was observed starting at 275ms together with stronger activity in prefrontal cortex. Following extensive training over 22 sessions, adding up to more than 16.600 trials, the earliest category effects occurred at a markedly shorter latency of 113ms and were accompanied by stronger occipitotemporal activity. Our results suggest that the brain balances plasticity and efficiency by relying on different mechanisms to recognize new and re-occurring categories.