In this article, we are going learn how to use ridge regression algorithm using neural networks and then compare the results we get with an RidgeRegressor model that comes built-in sklearn module.
import numpy as np
import matplotlib.pyplot as plt
Creating Dataset
A simple dataset using numpy arrays
x_train = np.array ([[4.7], [2.4], [7.5], [7.1], [4.3],
[7.8], [8.9], [5.2], [4.59], [2.1],
[8], [5], [7.5], [5], [4],
[8], [5.2], [4.9], [3], [4.7],
[4], [4.8], [3.5], [2.1], [4.1]],
dtype = np.float32)
y_train = np.array ([[2.6], [1.6], [3.09], [2.4], [2.4],
[3.3], [2.6], [1.96], [3.13], [1.76],
[3.2], [2.1], [1.6], [2.5], [2.2],
[2.75], [2.4], [1.8], [1], [2],
[1.6], [2.4], [2.6], [1.5], [3.1]],
dtype = np.float32)
View the data
There seems to be some relationship which can be plotted between x_train and y_train. A regression line can be drawn to represent the relationship
plt.figure(figsize=(8,8))
plt.scatter(x_train, y_train, c='green', s=250, label='Original data')
plt.show()
import torch
Converting data to pytorch tensors
By defualt requires_grad = False
X_train = torch.from_numpy(x_train)
Y_train = torch.from_numpy(y_train)
print('requires_grad for X_train: ', X_train.requires_grad)
print('requires_grad for Y_train: ', Y_train.requires_grad)
Output:
requires_grad for X_train: False
requires_grad for Y_train: False
Set the details for our neural network
Input, output and hidden layer sizes plus the learning rate
input_size = 1
hidden_size = 1
output_size = 1
learning_rate = 0.001
Create random Tensors for weights and biases.
Setting requires_grad=True indicates that we want to compute gradients with respect to these Tensors during the backward pass
w1 = torch.rand(input_size,
hidden_size,
requires_grad=True)
w1.shape
Output:
torch.Size([1, 1])
b1 = torch.rand(hidden_size,
output_size,
requires_grad=True)
b1.shape
Output:
torch.Size([1, 1])
w1
Output:
tensor([[0.6612]], requires_grad=True)
b1
Output:
tensor([[0.8730]], requires_grad=True)
alpha = 0.8
Training
Foward Pass:
- Predicting Y with input data X
- we use y = ax+b as its a regression problem
Finding Loss:
- Finding difference between Y_train and Y_pred by squaring the difference and then summing out, similar to nn.MSELoss and adding a penalty of L2 norm as it is ridge regression
For the loss_backward() function call:
- backward pass will compute the gradient of loss with respect to all Tensors with requires_grad=True.
- After this call w1.grad and b1.grad will be Tensors holding the gradient of the loss with respect to w1 and b1 respectively.
Manually updating the weights
- weights have requires_grad=True, but we don't need to track this in autograd. So will wrap it in torch.no_grad
- reducing weight with multiple of learning rate and gradient
- manually zero the weight gradients after updating weights
for iter in range(1, 5001):
y_pred = X_train.mm(w1).add(b1)
ridge_regularization_penalty = (w1*w1)
loss = ((y_pred - Y_train).pow(2).sum()) + (alpha * ridge_regularization_penalty)
if iter % 500 ==0:
print(iter, loss.item())
loss.backward()
with torch.no_grad():
w1 -= learning_rate * w1.grad
b1 -= learning_rate * b1.grad
w1.grad.zero_()
b1.grad.zero_()
Output:
500 6.147125720977783
1000 6.1442108154296875
1500 6.144202709197998
2000 6.144202709197998
2500 6.144202709197998
3000 6.144202709197998
3500 6.144202709197998
4000 6.144202709197998
4500 6.144202709197998
5000 6.144202709197998
print ('w1: ', w1)
print ('b1: ', b1)
Output:
w1: tensor([[0.1735]], requires_grad=True)
b1: tensor([[1.4123]], requires_grad=True)
Checking the output
Converting data into a tensor
x_train_tensor = torch.from_numpy(x_train)
x_train_tensor
Output:
tensor([[4.7000],
[2.4000],
[7.5000],
[7.1000],
[4.3000],
[7.8000],
[8.9000],
[5.2000],
[4.5900],
[2.1000],
[8.0000],
[5.0000],
[7.5000],
[5.0000],
[4.0000],
[8.0000],
[5.2000],
[4.9000],
[3.0000],
[4.7000],
[4.0000],
[4.8000],
[3.5000],
[2.1000],
[4.1000]])
Get the predicted values using the weights
Using final weights calculated from our training in order to get the predicted values
predicted_in_tensor = X_train.mm(w1).add(b1)
predicted_in_tensor
Output:
tensor([[2.2280],
[1.8288],
[2.7139],
[2.6445],
[2.1586],
[2.7660],
[2.9569],
[2.3148],
[2.2089],
[1.7768],
[2.8007],
[2.2801],
[2.7139],
[2.2801],
[2.1065],
[2.8007],
[2.3148],
[2.2627],
[1.9330],
[2.2280],
[2.1065],
[2.2454],
[2.0197],
[1.7768],
[2.1239]], grad_fn=<AddBackward0>)
Convert the prediction to a numpy array
This will be used to plot the regression line in a plot
predicted = predicted_in_tensor.detach().numpy()
predicted
Output:
array([[2.2280006],
[1.8288374],
[2.7139387],
[2.644519 ],
[2.1585813],
[2.7660036],
[2.9569077],
[2.3147755],
[2.2089105],
[1.7767726],
[2.8007135],
[2.2800655],
[2.7139387],
[2.2800655],
[2.1065164],
[2.8007135],
[2.3147755],
[2.2627106],
[1.932967 ],
[2.2280006],
[2.1065164],
[2.2453558],
[2.0197415],
[1.7767726],
[2.1238713]], dtype=float32)
Plotting
Our training has produced a rather accurate regression line
plt.figure(figsize=(8,8))
plt.scatter(x_train, y_train, c='green', s=250, label='Original data')
plt.plot(x_train, predicted, label = 'Fitted line ')
plt.legend()
plt.show()
We check the weights and biases with sklearn ridge regression to see if they match the weights and biases calculated by neural network
from sklearn.linear_model import Ridge
ridge_model = Ridge()
ridge_reg = ridge_model.fit(x_train, y_train)
print("w1 with sklearn is :", ridge_model.coef_)
w1 with sklearn is : [[0.17317088]]
print("b1 with sklearn is :", ridge_model.intercept_)
b1 with sklearn is : [1.4142637]
predicted = ridge_reg.predict(x_train)
predicted
Output:
array([[2.2281668],
[1.8298738],
[2.7130454],
[2.643777 ],
[2.1588986],
[2.7649965],
[2.9554844],
[2.3147523],
[2.2091181],
[1.7779225],
[2.7996306],
[2.280118 ],
[2.7130454],
[2.280118 ],
[2.1069472],
[2.7996306],
[2.3147523],
[2.2628012],
[1.9337764],
[2.2281668],
[2.1069472],
[2.2454839],
[2.020362 ],
[1.7779225],
[2.1242642]], dtype=float32)
plt.figure(figsize=(8, 8))
plt.scatter(x_train, y_train, c='green', s=250, label='Original data')
plt.plot(x_train, predicted, label = 'Fitted line')
plt.legend()
plt.show()
Thanks for reading :)