A search for the nonresonant production of Higgs boson pairs (HH ) via gluon-gluon and vector boson fusion processes in final states with two bottom quarks and two tau leptons is presented. The search uses data from proton-proton collisions at a center-of-mass energy of recorded with the CMS detector at the LHC, corresponding to an integrated luminosity of 138. Events in which at least one tau lepton decays hadronically are considered and multiple machine learning techniques are used to identify and extract the signal. The data are found to be consistent, within uncertainties, with the standard model (SM) predictions. Upper limits on the HH production cross section are set to constrain the parameter space for anomalous Higgs boson couplings. The observed (expected) upper limit at 95% confidence level corresponds to 3.3 (5.2) times the SM prediction for the inclusive HH cross section and to 124 (154) times the SM prediction for the vector boson fusion HH cross section. At 95% confidence level, the Higgs field self-coupling is constrained to be within −1.7 and 8.7 times the SM expectation, and the coupling of two Higgs bosons to two vector bosons is constrained to be within −0.4 and 2.6 times the SM expectation.