More directly, they usually would encounter "time travel," but there's also people who encounter people who are from very similar worlds to theirs which aren't quite the same. Anyway, in relation to the "time travel" - it is very possible to encounter someone from one year in your past or fifty years in your past, or maybe five years in your future. And then those people who went home to your past you didn't notice gone when they were here and return without memories, not changing your "present" and their "future."