Open Source Inference at Production Scale

Get more out of your AI infrastructure. Carlos Costa, Distinguished Engineer at IBM Research and co-founder of the llm-d project will share some exciting news about llm-d: how the project changed the way the open source AI community scales inference at PyTorch Conference North America. A little over a year after the project was founded, llm-d is reaching a new level of maturity, delivering state-of-the-art performance for generative serving, while also enabling the capabilities needed for day two operations, including multi-tenant quality of service, zero downtime operations, and all of that