How to Generate Synthetic Pretraining Data? with Joël Niklaus from Hugginface

today we are taking a deeper look at of the secret behind LLM pretraining which is synthetic data pipelines with Joel Niklaus machine learning engineer at Hugging Face! joel an his team ran 90 controlled experiments and burned over a trillion tokens to figure out what actually makes good pretraining data and in huggingface fashion provided all of their findings/artifacts openly! 👉Joel Website: https://niklaus.ai 👉 Twitter: https://x.com/joelniklaus 👉 Paper : arXiv: https://arxiv.org/abs/2604.13977 👉 Blog: https://huggingface.co/spaces/HuggingFaceFW/finephrase Check out Arcee (American o