The precise time step integration method proposed for linear time-invariant homogeneous dynamic systems can provide precise numerical results that approach an exact solution at the integration points. However, difficulty arises when the algorithm is used for non-homogeneous dynamic systems, due to the inverse matrix calculation and the simulation accuracy of the applied loading. By combining the Gaussian quadrature method and state space theory with the calculation technique of matrix exponential function in the precise time step integration method, a new modified precise time step integration method (e.g., an algorithm with an arbitrary order of accuracy) is proposed. In the new method, no inverse matrix calculation or simulation of the applied loading is needed, and the computing efficiency is improved. In particular, the proposed method is independent of the quality of the matrix H. If the matrix H is singular or nearly singular, the advantage of the method is remarkable. The numerical stability of the proposed algorithm is discussed and a numerical example is given to demonstrate the validity and efficiency of the algorithm.