Periodic Function as Activation Function for Neural Networks

Abstract

In this paper, we explore the periodic function as alternative activation function for neural network. Previously sigmoid function is used as standard activation function for neuron and now linear rectifier function are used. Even max-out function can be learned as a general form convex activation function. We explore the possibility in the other direction, where we use periodic function as activation function. We expect network with less layer and less neuron can capture the target distribution. The experiments verify our expectation and show that period function can act as an alternative activation function.