That's the premise of Sekai kara neko ga kieta nara, a 2012 short novel. The author is Genki Kawamura, who more recently co-produced the anime film Kimi no na wa (Your Name).
A friend recommended the book as light reading, manageable for N2/N3 readers. I read 10 to 20 pages a day without a dictionary and understood enough to enjoy it.
I also watched the 2016 movie version, which I recommend for listening practice. It's a good adaptation, with a few changes for more realism. With Japanese subtitles on, it shouldn't be hard to enjoy the movie if you're used to listening to daily conversation in Japanese.
Sekai kara neko is the last book I read in Japanese -- which was two whole months ago, erp!
I've been caught up reviewing games and, plus, am going real easy on the studying until end of winter. Japanese learning isn't my priority this season. But I've just picked up the translation of C.S. Lewis' Prince Caspian (カスピアン王子のつのぶえ) and use it occasionally for bedtime reading.