Optimizing LLM Inference: Challenges and Best Practices

This presentation delves into the world of Large Language Models (LLMs), focusing on the  efficiency of LLM inference. We will discuss the tradeoff of latency and bandwidth, followed by a deep dive into techniques for accelerating inference, such as KV caching, quantization, speculative decoding, and various forms of parallelism. We will compare popular inference frameworks and address the challenge of navigating the multitude of design choices. Finally, we'll introduce Nvidia Inference Microservices as a convenient one-stop solution for achieving efficient inference on many of the popular models.