Maybe try shadowing? I found that my ability to understand spoken Japanese went through the roof once I started doing this (although I’ve yet to speak to other human beings in Japanese… so yeah…). Probably any material will do, but I find shadowing the dialogues on Japanesepod101 (which despite engaging in slightly sketchy marketing tactics is actually very good) particularly useful. Many of the lessons have informal tracks that might prove especially valuable in your case. If you haven’t done it already, consider getting something like Pimsleur or Michel Thomas audio courses - while they are not the best things in the world, they will get you used to thinking on your toes a bit and just force you to spit out phrases.
But yeah, part of it is just using the language - I had a a similar experience to you living in Italy for several months. At first everything went over my head, despite the fact that I had spent a considerable amount of time studying beforehand, but I was able to get into the swing of things after a week or two. Now granted, I speak French, so Italian was much easier for me to learn than Japanese, but just being forced to speak everyday, things will slowly start clicking in place.