Professional Documents
Culture Documents
Lecture 08 Dataset and Dataloader
Lecture 08 Dataset and Dataloader
Lecturer : Hongpu Liu Lecture 8-1 PyTorch Tutorial @ SLAM Research Group
Revision: Manual data feed
……
Lecturer : Hongpu Liu Lecture 8-2 PyTorch Tutorial @ SLAM Research Group
Terminology: Epoch, Batch-Size, Iterations
# Training cycle
for epoch in range(training_epochs):
# Loop over all batches
for i in range(total_batch):
One forward pass and one The number of training Number of passes, each
backward pass of all the examples in one forward pass using [batch size]
training examples. backward pass. number of examples.
Lecturer : Hongpu Liu Lecture 8-3 PyTorch Tutorial @ SLAM Research Group
DataLoader: batch_size=2, shuffle=True
Batch 1
Sample 2 Sample 7 Sample 7
Batch 2
Sample 4 Sample 1 Sample 1
Batch 3
Sample 6 Sample 2 Sample 2
Batch 4
Sample 8 Sample 3 Sample 3
Lecturer : Hongpu Liu Lecture 8-4 PyTorch Tutorial @ SLAM Research Group
How to define your Dataset
import torch
from torch.utils.data import Dataset
Dataset is an abstract class. We
from torch.utils.data import DataLoader
can define our class inherited from
class DiabetesDataset(Dataset):
def __init__(self):
this class.
pass
def __len__(self):
pass
dataset = DiabetesDataset()
train_loader = DataLoader(dataset=dataset,
batch_size=32,
shuffle=True,
num_workers=2)
Lecturer : Hongpu Liu Lecture 8-5 PyTorch Tutorial @ SLAM Research Group
How to define your Dataset
import torch
from torch.utils.data import Dataset DataLoader is a class to help us
from torch.utils.data import DataLoader
loading data in PyTorch.
class DiabetesDataset(Dataset):
def __init__(self):
pass
def __len__(self):
pass
dataset = DiabetesDataset()
train_loader = DataLoader(dataset=dataset,
batch_size=32,
shuffle=True,
num_workers=2)
Lecturer : Hongpu Liu Lecture 8-6 PyTorch Tutorial @ SLAM Research Group
How to define your Dataset
import torch
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
DiabetesDataset is inherited from
class DiabetesDataset(Dataset):
def __init__(self): abstract class Dataset.
pass
def __len__(self):
pass
dataset = DiabetesDataset()
train_loader = DataLoader(dataset=dataset,
batch_size=32,
shuffle=True,
num_workers=2)
Lecturer : Hongpu Liu Lecture 8-7 PyTorch Tutorial @ SLAM Research Group
How to define your Dataset
import torch
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
class DiabetesDataset(Dataset):
def __init__(self):
pass
The expression, dataset[index],
def __getitem__(self, index):
pass will call this magic function.
def __len__(self):
pass
dataset = DiabetesDataset()
train_loader = DataLoader(dataset=dataset,
batch_size=32,
shuffle=True,
num_workers=2)
Lecturer : Hongpu Liu Lecture 8-8 PyTorch Tutorial @ SLAM Research Group
How to define your Dataset
import torch
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
class DiabetesDataset(Dataset):
def __init__(self):
pass
Lecturer : Hongpu Liu Lecture 8-9 PyTorch Tutorial @ SLAM Research Group
How to define your Dataset
import torch
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
class DiabetesDataset(Dataset):
def __init__(self):
pass
Lecturer : Hongpu Liu Lecture 8-10 PyTorch Tutorial @ SLAM Research Group
How to define your Dataset
import torch
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
class DiabetesDataset(Dataset):
def __init__(self):
pass
def __len__(self):
pass
Lecturer : Hongpu Liu Lecture 8-11 PyTorch Tutorial @ SLAM Research Group
How to define your Dataset
import torch
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
class DiabetesDataset(Dataset):
def __init__(self):
pass
def __len__(self):
pass
Lecturer : Hongpu Liu Lecture 8-12 PyTorch Tutorial @ SLAM Research Group
Extra: num_workers in Windows
Lecturer : Hongpu Liu Lecture 8-13 PyTorch Tutorial @ SLAM Research Group
Extra: num_workers in Windows
train_loader = DataLoader(dataset=dataset,
batch_size=32,
shuffle=True,
num_workers=2)
……
if __name__ == '__main__':
for epoch in range(100):
for i, data in enumerate(train_loader, 0):
# 1. Prepare data
Lecturer : Hongpu Liu Lecture 8-14 PyTorch Tutorial @ SLAM Research Group
Example: Diabetes Dataset
class DiabetesDataset(Dataset):
def __init__(self, filepath):
xy = np.loadtxt(filepath, delimiter=',', dtype=np.float32)
self.len = xy.shape[0]
self.x_data = torch.from_numpy(xy[:, :-1])
self.y_data = torch.from_numpy(xy[:, [-1]])
def __len__(self):
return self.len
dataset = DiabetesDataset('diabetes.csv.gz')
train_loader = DataLoader(dataset=dataset, batch_size=32, shuffle=True, num_workers=2)
Lecturer : Hongpu Liu Lecture 8-15 PyTorch Tutorial @ SLAM Research Group
Example: Using DataLoader
Lecturer : Hongpu Liu Lecture 8-16 PyTorch Tutorial @ SLAM Research Group
Classifying Diabetes
import numpy as np
Prepare dataset
import torch
1
from torch.utils.data import Dataset, DataLoader
class DiabetesDataset(Dataset):
def __init__(self, filepath):
xy = np.loadtxt(filepath, delimiter=',', dtype=np.float32)
self.len = xy.shape[0]
self.x_data = torch.from_numpy(xy[:, :-1])
Dataset and Dataloader
self.y_data = torch.from_numpy(xy[:, [-1]])
def __len__(self):
return self.len
class Model(torch.nn.Module):
inherit from nn.Module
def __init__(self):
super(Model, self).__init__()
self.linear1 = torch.nn.Linear(8, 6)
self.linear2 = torch.nn.Linear(6, 4)
self.linear3 = torch.nn.Linear(4, 1)
self.sigmoid = torch.nn.Sigmoid()
3
x = self.sigmoid(self.linear1(x))
x = self.sigmoid(self.linear2(x))
x = self.sigmoid(self.linear3(x))
return x
Training cycle
# 2. Forward
4
y_pred = model(inputs)
loss = criterion(y_pred, labels)
print(epoch, i, loss.item())
# 3. Backward
Lecturer : Hongpu Liu Lecture 8-17 PyTorch Tutorial @ SLAM Research Group
The following dataset loaders are available
• MNIST
• Fashion-MNIST
• EMNIST
• COCO
• LSUN
• ImageFolder
• DatasetFolder
• Imagenet-12
• CIFAR
• STL10
• PhotoTour
Lecturer : Hongpu Liu Lecture 8-18 PyTorch Tutorial @ SLAM Research Group
Example: MINST Dataset
import torch
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision import datasets
train_dataset = datasets.MNIST(root='../dataset/mnist',
train=True,
transform= transforms.ToTensor(),
download=True)
test_dataset = datasets.MNIST(root='../dataset/mnist',
train=False,
transform= transforms.ToTensor(),
download=True)
train_loader = DataLoader(dataset=train_dataset,
batch_size=32,
shuffle=True)
test_loader = DataLoader(dataset=test_dataset,
batch_size=32,
shuffle=False)
Lecturer : Hongpu Liu Lecture 8-19 PyTorch Tutorial @ SLAM Research Group
Exercise 8-1
Lecturer : Hongpu Liu Lecture 8-20 PyTorch Tutorial @ SLAM Research Group
PyTorch Tutorial
08. Dataset and DataLoader
Lecturer : Hongpu Liu Lecture 8-21 PyTorch Tutorial @ SLAM Research Group