The first search for the Z boson decay to ττμμ at the CERN LHC is presented, based on data collected by the CMS experiment at the LHC in proton-proton collisions at a center-of-mass energy of 13 TeV and corresponding to an integrated luminosity of 138 fb^{-1}. The data are compatible with the predicted background. For the first time, an upper limit at the 95% confidence level of 6.9 times the standard model expectation is placed on the ratio of the Z→ττμμ to Z→4μ branching fractions. Limits are also placed on the six flavor-conserving four-lepton effective-field-theory operators involving two muons and two tau leptons, for the first time testing all such operators.