The present paper proposes the optimal transmission parameters for non-return-to-zero (NRZ), return-to-zero differential phase shift keying (RZ-DPSK), carrier-suppressed RZ-DPSK (CSRZ-DPSK) and duobinary (DB) modulation formats in 16-channel 10 Gbit/s wavelength division multiplexed passive optical network (WDM-PON) transmission system. The paper investigates influence of laser output power, length of dispersion compensating fiber (DCF) and application of erbium-doped fiber amplifiers (EDFA), as well as impact of optical components configuration on received signal quality. It compares demand of these modulation formats for amplification and dispersion compensation with the increase of transmission distance, determining the optimal conditions for achieving maximal network reach for each of them. It is shown, that DB is the most resistant to chromatic dispersion (CD) and is able to achieve the longest transmission distance (100 km) without dispersion compensation. Whereas RZ-DPSK and CSRZ-DPSK modulation formats demonstrated maximal achievable reach (180 km) using DCF and EDFA.