I worked in Japan for 4 years and have spent another couple of years there when you sum up various trips. My wife is Japanese, 20 years married this year.
I don't have much time and I can tend to waffle so I'm going to bullet point some thoughts
I moved to Japan after living in various other Asian countries for a short while (I was working in the oil industry) and I was blown away by the place, it was easily the coolest place to be.
The lustre wore off after about 6 months. A friend of mine who still lives there mentioned that it must feel the same if you move to Disney Land, sooner or later it becomes all too obvious that the things which seem magical at first are far more banal when you figure out how or why they operate. That's not to say it's not nice, just that the initial glow will dim. A wet Tuesday afternoon in Japan is like a wet Tuesday afternoon anywhere else.
Being a foreigner can be tough depending on where you live and how thick skinned you are. If you live in Toyko or Osaka you won't get a second look, if you live in a rural town you'll get lots of looks. I only encountered kindness but I know foreigners who have had a tough time in smaller cities/towns. In some cases you can absolutely embrace this and some flamboyant foreigners have done really well in Japan....despite seeming to be almost completely talent less to non-Japanese people
As a foreigner in the workplace you will be given more slack than a Japanese person, but you'll also be excluded from stuff and the Japanese way of doing business can drive you crazy. The hours are long, the salaries are usually excellent.
If you live in Tokyo, you'll be living small, living space is a premium.
Osaka, Hiroshima, Kyoto are nicer places to live than Tokyo and will offer a better standard of living while still providing lots to see and do
You know about the food already, but I'll point out that wherever you go in Japan you'll get great food and service.
I'm not sure the EFL route is as easy as it used to be. There are a lot more homegrown English teachers now, plus things like the JET scheme dropped lots of unprepared young Westerners into the rigid Japanese education system and the reputation of EFL teachers took a big hit. The fact that you're older will certainly help in that regard. I knew someone who recently went to Vietnam as an EFL with the intention of getting experience there before applying to a school in Japan. It turns out he wants to stay in Vietnam now, but that was his strategy.
Good luck. You'll only really know if you try it, and IMO it's better to try than regret not going years later.