With the rapid increase in the data volume of modern communications, optical fiber infrastructures are required to have a larger capacity and higher transmission rate. However, due to the nature of the optical fiber, the signal will be interfered by the interaction of chromatic dispersion (CD), nonlinear effects, and amplified spontaneous emission (ASE) noise during optical fiber transmission, and these effects limit the performance of optical fiber transmission. One promising strategy for nonlinearity mitigation is digital backpropagation (DBP).  It can be governed by the inverse nonlinear Schrödinger equation and the split-step Fourier method (SSFM) is used to solve this equation numerically.  Although DBP is able to achieve high accuracy, it requires high computational complexity and power consumption as well.  In this seminar, we will first introduce the basic concepts in optical communication systems and the existing difficulties of nonlinear compensation. Then we briefly review the previous efforts to this problem in the last decade. Finally, we will present the proposed Neural Network-based algorithm to compensate for the nonlinear effect in optical signal propagation. This approach can reduce the computational complexity largely. We show that our proposed method achieved a promising performance in a 100 Gbaud/s 16QAM transmission across 1600 km with a much lower computational cost compared to traditional DBP.