The interdisciplinary synergy of deep learning (DL) and metasurface shows great power in revealing the hidden complicated functions between physical structures and electromagnetic responses of the meta-atoms. But the conventional DL-based method is facing a conflict between computational efficiency and high demand for precise predictions, which consumes substantial time and computing resources to collect the specific dataset due to its data-hungry nature. Herein we report a complex-valued spectral transfer-learning-based hybrid metasurface design method to alleviate both ends of this dilemma. The proposed approach robustly improves the spectrum predicting accuracy on a small dataset with only 1000 homemade samples in the target terahertz (THz) waveband by migrating the knowledge learned from the open-source data in the infrared band, where the scale invariance of Maxwell’s equations is no longer applicable due to the material dispersion in different bands. Several typical THz meta-devices are demonstrated by employing the hybrid inverse model consolidating this trained target network and a global optimization algorithm as proof-of-concept applications. The simulated results clarify the reliability and scalability of our spectral transfer-learning-based metasurface design methodology assisted by complex deep neural networks, which is of great significance for balancing the efficiency and accuracy of the DL-based method, hence promoting the automated metasurface design in arbitrary bands.