Rutile to anatase phase transition in Nb-doped thin films grown by RF magnetron sputtering method, annealed in H2 atmosphere at 460°C for 30 min using X-ray diffraction (XRD), X-ray photoelectron spectroscopy (XPS), UV-VIS spectroscopy and Raman analysis was found. XRD study reveals transformation of rutile fraction phase presented in mixed structure of Nb:TiO2 thin films before annealing in H2 environment to anatase and increases the crystallites size from 21 nm to 32 nm. The UV-VIS spectroscopy shows that the optical band gap of the films increases as the Nb concentration increases. Moreover, annealing in H2 additionally increases the band gap. The increase of the optical band gap of the films is explained by the presence of both anatase phase and Nb2O5 inclusions as shown by analysis of Raman spectra.