The authors have studied the possibilities to obtain the maximum spectral efficiency for a combined wavelength division multiplexing (WDM) system by minimizing the channel spacing. The fiber optic transmission systems under study can be considered in the context of next generation optical networks and is offered as a model for designing the future backbone networks. In the case of different telecom operators’ optical networks merging, in the nearest future the necessity may arise to transmit variously modulated optical signals over a single optical fiber (even with different per channel bitrates). In this paper the authors show an optimal combined WDM system configuration that provides the least bit-error-rate values for the signals detected in system’s channels. It is found that the minimal channel spacing for differently modulated optical signals in the investigated mixed data rates WDM system is 75 GHz if these signals are transmitted with 10 Gbit/s and 40 Gbit/s per channel bitrates. It is revealed that such a system’s average spectral efficiency depends also on a combined system’s configuration in the case of channel separation with equal frequency intervals.