Thesis describes the wavelength division multiplexed fiber optic communication systems, i.e., assessment of the development of PON and DCI. Experimentally evaluate the application of spectrally efficient multi-level pulse amplitude modulation formats to increase the performance of PON, DCI and RoF solutions for 5G and beyond implementation. The evaluation of the applied optical transmission environment for the implementation of a high-speed spectrally efficient broadband transmission systems is performed. Application of SDM technology for the implementation of a spectrally and spatial efficient wavelength division multiplexed FOTS is evaluated. Spectrally efficient next-generation optical system for the transmission of millimeter-wave radio signals over fiber for cellular mobile communications and new hybrid fiber optic transmission system architecture capable of broadband Internet data transmission and radio signal transmission over single optical fiber was developed. Implementation of a novel type of transmission with a novel WGMR-based Kerr-OFC multi-wave light source are shown and presented for first time.