Back To Schedule
Wednesday, October 27 • 12:30pm - 2:30pm
Democratizing Deep Learning with Commodity Hardware: How to Train Large Deep Learning Models on CPU Efficiently with Sparsity - Auditorium

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
GPUs are expensive, require premium infrastructure, and are hard to virtualize. Furthermore, our models and data are growing faster than GPU memory. The communication cost of distributing the models over GPUs is prohibitively expensive for most workloads.

Wouldn't it be nice if we could train extensive models with commodity CPUs faster than GPUs? CPUs are cheap, well understood, and ubiquitous hardware. The main memory in CPUs can quickly run in Terabytes (TB) with minimum investment. For extensive models, we can fit both the model and the data in the CPU RAM.

This tutorial will focus on a new emerging paradigm of deep learning training using sparsity and hash tables. We will introduce the idea of selectively identifying parameters and sparsity patterns during exercise. We will demonstrate the integration of these algorithms in existing python codes. As a result, we demonstrate significantly superior deep learning capabilities on CPU, making them competitive (or even better) than state-of-the-art packages on some of the best GPUs. If time permits, we will briefly discuss multi-node implementation and some thoughts on how to train outrageously (Tens of billions or more) large models on small commodity clusters.

avatar for Anshumali Shrivastava

Anshumali Shrivastava

Professor, Rice University; Founder, ThirdAI Corp
Anshumali Shrivastava's research focuses on Large Scale Machine Learning, Scalable and Sustainable Deep Learning, Randomized Algorithms for Big-Data and Graph Mining.

Nicholas Meisburger

Rice University

Shabnam Daghaghi

Rice University

Minghao Yan

RIce University

Wednesday October 27, 2021 12:30pm - 2:30pm CDT

Attendees (3)