With the two-stage fourth-order temporal evolution of the gas distribution function and Weighted Essentially Non-Oscillatory (WENO) reconstruction, a high-order finite difference gas-kinetic scheme is proposed. Different from the previous high-order finite volume gas-kinetic methods, which uses the discontinuous initial reconstruction at the cell interface, the present scheme is the conservative finite difference method with a continuous flow distribution at the grid point. And the numerical fluxes are obtained by the kinetic splitting method, instead of the traditional flux splitting based on the approximate Riemann solver. Many numerical tests in solving one and two-dimensional Euler and Navier-Stokes equations demonstrate the current scheme is highly stable, accurate, and efficient, capturing discontinuities without oscillations.