DISA

Where? Onsite: D2272 and via zoom
Registration: Please sign up for the seminar via this link https://forms.gle/7LK5jZfjVwvYAf4L8 by February 18. This is especially important if you plan to attend onsite so we can make sure there is fika for everyone.

Abstracts
Accelerate ML: Overlap of computation and collective communication in multi-GPU systems – Minyu Cui 
The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental building block of modern ML systems. As model sizes continue to increase and computational throughput improves, communication overhead has emerged as one of the dominant performance bottlenecks in multi-GPU computing paradigms. Conventional training pipelines in multi-GPU systems perform computation and communication sequentially, which leads to idle compute resources, limited scalability, and inefficient hardware utilization. 

In my research plan, I aim to accelerate multi-GPU ML by overlapping the two dominant operations: computation (such as GEMM) and collective communication. I will explore two complementary and efficient directions. First, my research will explore overlapping computation and communication kernels to hide communication latency. Second, it will further investigate fusing computation and communication into a single GPU kernel to enable efficient fine-grained overlap. These efforts will initially focus on improving operator-level performance and will subsequently be extended to enhance end-to-end training performance. 

I used to love Python… – Morgan Ericsson  
Some 15 years ago, when I did a lot of NLP, I learned Python 2, because it was the language that made the most sense (that was not Perl). I found it to be a beautiful language that made it fast and easy to translate thoughts into code. The rich ecosystem often turned coding into “gluing” things together, and since the things were often written in, e.g., C, it was fast enough. These days, I find it a lot more frustrating. The things were always silos, but a few years ago, I never found it to be a problem. These days, you are stuck with things that might play well together, if the authors were aware and took the time to integrate. If you are lucky, the things will support the platform (cpu, cuda, mps, …) that you are running on, but if not, well, then it’s not so much fun. You are also hitting all kinds of performance issues and bugs in the various things and gaps between them. So, for some work, I don’t like Python very much these days. My talk will rant about the problem but also try to find a way forward, looking at helpful tools for today and ideal solutions for tomorrow…