Implementing the modern technologies of light-emitting devices, light harvesting, and quantum information processing requires the understanding of the structure-function relations at spatial scales below the optical diffraction limit and time scales of energy and information flows. Here, we distinctively combine cathodoluminescence (CL) with ultrafast electron microscopy (UEM), termed CL-UEM, because CL and UEM synergetically afford the required spectral and spatiotemporal sensitivities, respectively. For color centers in nanodiamonds, we demonstrate the measurement of CL lifetime with a local sensitivity of 50 nm and a time resolution of 100 ps. It is revealed that the emitting states of the color centers can be populated through charge transfer among the color centers across diamond lattices upon high-energy electron beam excitation. The technical advance achieved in this study will facilitate the specific control over energy conversion at nanoscales, relevant to quantum dots and single-photon sources.