In recent years, the information technology sector has developed rapidly and there has been a rapid increase in the amount of information transmitted. This can be explained by the increase in Internet users, multimedia services, and devices registered on the global network. The demand for larger telecommunication network capacities is growing, therefore it is necessary to increase the number of channels and transmission speed in wavelength division multiplexed transmission systems. Various optical amplifiers can be used to compensate for the attenuation of the accumulated signal (over a wide wavelength range). The most popular type of amplifier is the erbium ion-doped fiber amplifier (EDFA), which operates in the limited C-band (1530-1565 nm). Therefore, it is necessary to look for new amplification solutions that would allow to expand of the amplification spectrum and increase the achievable optical signal amplification. The Doctoral Thesis did research on the use of EDFA, Raman, FOPA, and combined optical amplifiers in wavelength division multiplexing systems. The study was performed using computer simulation programs (Rsoft OptSim and VPIphotonics Design Suite). In the work, special emphasis is placed on erbium and ytterbium alloy fibers with the study of computer modeling of optical amplifiers in cladding pumping technology. The use of such a broadband amplifier in WDM transmission systems makes it possible to achieve higher and more uniform gain and to improve the performance of fiber optic communication systems. The results obtained from the EYDFA computer simulation will be further used for experimental research in the RTU ComTech Research Center. The Doctoral Thesis reflects the results of completed research, which evaluates the performance of both existing and new amplification solutions in WDM systems.