Charge-dependent azimuthal particle correlations with respect to the second-order event plane in p-Pb and PbPb collisions at a nucleon-nucleon center-of-mass energy of 5.02 TeV have been studied with the CMS experiment at the LHC. The measurement is performed with a three-particle correlation technique, using two particles with the same or opposite charge within the pseudorapidity range |η| < 2.4, and a third particle measured in the hadron forward calorimeters (4.4 < |η| < 5). The observed differences between the same and opposite sign correlations, as functions of multiplicity and η gap between the two charged particles, are of similar magnitude in p-Pb and PbPb collisions at the same multiplicities. These results pose a challenge for the interpretation of charge-dependent azimuthal correlations in heavy ion collisions in terms of the chiral magnetic effect.