Correct. But you're not alone in that regard. Everyone who's been taken from their world will forget once they return home.

It also means you can do whatever you want without the consequence carrying over to your original time. Isn't that nice?