Megatron-Energon

Megatron-Energon serves as a multi-modal data loader designed to facilitate the training of large models by efficiently loading and blending diverse datasets. It supports distributed processing across multiple nodes, ensuring reproducibility and resumability of training sessions. The package includes dataset-related classes and a command line utility for data preprocessing, making it suitable for users working with large-scale machine learning projects.