A high performance algorithm is proposed to provide efficient separation of transmission paths in single carrier duplex satellite systems. The algorithm is based on direct estimation of Wiener-Hopf equation solution on each iteration in real-time. Several approximations and fast calculation techniques are implemented to reduce computational complexity of the algorithm proposed. In particular, the algorithm uses predefined autocorrelation matrix, fast procedure of cross-correlation vector calculation and fast procedure of Toeplitz matrix multiplication by vector. The approximation of cross-correlation vectors array is also implemented. Algorithm efficiency is been studied on simplified model of earth stations and satellite. The study results demonstrate that the algorithm provides echo interference suppression more than 40 dB in typical for bandwidth optimization techniques conditions and allows to use QAM modulation with 256 levels. The proposed algorithm was compared for robustness with other commonly used adaptive filtering algorithms, such as: conventional RLS, QR-decomposition RLS, Householder RLS, NLMS, frequency domain LMS and affine projection algorithm. In considered application, the proposed algorithm outperforms other algorithms in terms of echo interference suppression.