Measurements of gravitational waves from the inspiral of a stellar-mass compact object into a massive black hole are unique probes to test general relativity (GR) and massive black hole (MBH) properties, as well as the stellar distribution about these holes in galactic nuclei. Current data analysis techniques can provide us with parameter estimation with very narrow errors. However, an extreme-mass ratio inspiral (EMRI) is not a two-body problem, since other stellar bodies orbiting nearby will influence the capture orbit. Any deviation from the isolated inspiral will induce a small, though observable, deviation from the idealized waveform which could be misinterpreted as a failure of GR. Based on conservative analysis of mass segregation in a Milky-Way-like nucleus, we estimate that the possibility that another star has a semimajor axis comparable to that of the EMRI is non-negligible, although probably very small. This star introduces an observable perturbation in the orbit in the case in which we consider only loss of energy via gravitational radiation. When considering the two first-order non-dissipative post-Newtonian contributions (the periapsis shift of the orbit), the evolution of the orbital elements of the EMRI turns out to be chaotic in nature. The implications of this study are twofold. From the one side, the application to testing GR and measuring MBH parameters with the detection of EMRIs in galactic nuclei with a millihertz mission will be even more challenging than believed. From the other side, this behavior could in principle be used as a signature of mass segregation in galactic nuclei.