The authors have investigated the optical signal transmission over the wavelength division multiplexed (WDM) fiber optic transmission systems (FOTS) that are offered as a model for the future design of backbone optical networks. The proposed model is fully characterized by the modulation formats used for the manipulation of optical signal freedom degrees and particular channel’s bitrate. In this paper investigated mixed-WDM system complies to the following configuration scheme: [1st, 4th and 7th channels: NRZ-OOK, R= 10 or 40 Gbit/s] – [2nd, 5th and 8th channels: 2-POLSK, R= 10 or 40 Gbit/s] – [3rd, 6th and 9th channels: NRZ – DPSK, R= 10 or 40 Gbit/s]. This research revealed not only the optimal configuration of such FOTS that secures the lowest in system’s channels detected signals bit-error rate but also the frequency plan for channels central wavelength that allows achieving the maximum allowable systems spectral efficiency. In addition, the maximum permissible transmission distance for the spectrally efficient configurations of mixed-WDM FOTS and the impact of nonlinear effects to optical signal transmission are evaluated.