New Survey Highlights Data Mixing Strategies for LLM Pretraining
A comprehensive survey explores domain-level data mixing techniques for optimizing large language model pretraining. The research underscores the importance of strategic data composition in improving efficiency and generalization under constrained resources.