CELU() and GELU() in PyTorch – ██FR█████ █INTELL███████████

This content originally appeared on DEV Community and was authored by Super Kai (Kazuya Ito)

*Memos:

My post explains FReLU, ELU, SELU and CELU.
My post explains GELU, Mish, SiLU and Softplus.
My post explains heaviside() and ReLU().
My post explains LeakyReLU() and PReLU().
My post explains ELU() and SELU().

CELU() can get the 0D or more D tensor of the zero or more values computed by CELU function from the 0D or more D tensor of zero or more elements as shown below:

*Memos:

The 1st argument for initialization is alpha(Optional-Default:1.0-Type:float). *It’s applied to negative input values.
The 2nd argument for initialization is inplace(Optional-Default:False-Type:bool): *Memos:
- It does in-place operation.
- Keep it False because it’s problematic with True.
The 1st argument is input(Required-Type:tensor of float).

import torch
from torch import nn

my_tensor = torch.tensor([8., -3., 0., 1., 5., -2., -1., 4.])

celu = nn.CELU()
celu(input=my_tensor)
# tensor([8.0000, -0.9502, 0.0000, 1.0000, 5.0000, -0.8647, -0.6321, 4.0000])

celu
# CELU(alpha=1.0)

celu.alpha
# 1.0

celu.inplace
# False

celu = nn.CELU(alpha=1.0, inplace=True)
celu(input=my_tensor)
# tensor([8.0000, -0.9502, 0.0000, 1.0000, 5.0000, -0.8647, -0.6321, 4.0000])

my_tensor = torch.tensor([[8., -3., 0., 1.],
                          [5., -2., -1., 4.]])
celu = nn.CELU()
celu(input=my_tensor)
# tensor([[8.0000, -0.9502, 0.0000, 1.0000],
#         [5.0000, -0.8647, -0.6321, 4.0000]])

my_tensor = torch.tensor([[[8., -3.], [0., 1.]],
                          [[5., -2.], [-1., 4.]]])
celu = nn.CELU()
celu(input=my_tensor)
# tensor([[[8.0000, -0.9502], [0.0000, 1.0000]],
#         [[5.0000, -0.8647], [-0.6321, 4.0000]]])

GELU() can get the 0D or more D tensor of the zero or more values computed by GELU function from the 0D or more D tensor of zero or more elements as shown below:

*Memos:

The 1st argument for initialization is approximate(Optional-Default:'none'-Type:str): *Memos:
- 'none' or 'tanh' can be selected.
- The results of 'none' or 'tanh' are almost the same.
The 1st argument is input(Required-Type:tensor of float).

import torch
from torch import nn

my_tensor = torch.tensor([8., -3., 0., 1., 5., -2., -1., 4.])

gelu = nn.GELU()
gelu(input=my_tensor)
# tensor([8.0000e+00, -4.0499e-03, 0.0000e+00, 8.4134e-01,
#         5.0000e+00, -4.5500e-02, -1.5866e-01, 3.9999e+00])

gelu
# GELU(approximate='none')

gelu.approximate
# False

gelu = nn.GELU(approximate='tanh')
gelu(input=my_tensor)
# tensor([8.0000e+00, -3.6374e-03, 0.0000e+00, 8.4119e-01,
#         5.0000e+00, -4.5402e-02, -1.5881e-01, 3.9999e+00])

my_tensor = torch.tensor([[8., -3., 0., 1.],
                          [5., -2., -1., 4.]])
gelu = nn.GELU()
gelu(input=my_tensor)
# tensor([[8.0000e+00, -4.0499e-03, 0.0000e+00, 8.4134e-01],
#         [5.0000e+00, -4.5500e-02, -1.5866e-01, 3.9999e+00]])

my_tensor = torch.tensor([[[8., -3.], [0., 1.]],
                          [[5., -2.], [-1., 4.]]])
gelu = nn.GELU()
gelu(input=my_tensor)
# tensor([[[8.0000e+00, -4.0499e-03], [0.0000e+00, 8.4134e-01]],
#         [[5.0000e+00, -4.5500e-02], [-1.5866e-01, 3.9999e+00]]])

This content originally appeared on DEV Community and was authored by Super Kai (Kazuya Ito)