Using Dask and Many GPUs to Train a Neural Network with PyTorch

Dask Distributed Summit 2021 | May 16, 2021

I took a neural network I had trained on a single CPU to generate pet names and tried retraining it with tons of connected GPUs using Dask, PyTorch, and the package dask-pytorch-ddp. I learned a lot about when is the right time to use multiple GPUs and what the pitfalls can be. In this talk I discussed what these lessons mean for training with GPUs and Dask.