Due to continuous growth in Internet traffic requests, optical networks operators are becoming more interesting in solutions that allow to increase a spectral efficiency. However, the increase of spectral efficiency could sufficiently reduce the transparent reach of the wavelengths. Extra 3R regenerators lead to additional power consumption required to transmit the certain amount of aggregated traffic, i.e., worse energy efficiency. Therefore, for the certain quality of transmission, the trade-off between spectral efficiency, length of fiber optical link and energy efficiency exists in wavelength division multiplexing (WDM) optical networks. This doctoral thesis covers a study about possibilities of increasing spectral efficiency in WDM-based fiber optical links and its impact on energy consumption required to transmit 1 bit of aggregated traffic between two optical network nodes with defined signal quality guarantee. Research is performed for the WDM-based fiber optical links with different data rates, modulation formats and single-mode optical fiber span lengths where forward error correction might not be used. As a result, this doctoral thesis overviews the research results that could be considered as completed. They have been used for five journal publications and presented at more than 20 international scientific conferences in more than 15 countries.