Optimal Splitting of Language Models from Mixtures to Specialized Domains

This paper addresses how to optimally train multiple specialized language models for different domains by determining how to split computational resources between general pretraining and domain-specific fine-tuning. The authors develop a method using scaling laws to predict model performance and find the best allocation of computing power, showing improvements in reasoning and knowledge tasks across different model sizes.