Very real. I haven’t mastered these things myself, but I think I’m in a similar enough boat (moved to Japan 3 years ago, been studying fairly consistently ever since, think I just passed N1 pretty easily last week, handle all my immigration stuff on my own), that my perspective won’t be totally useless.
Basically, I think contextual knowledge is going to serve you as much as, if not even more than language ability alone.
For example, I had a complicated immigration process recently to get a new visa as half-Japanese. It required about 20 different documents issued between my home country and Japan. This included the usual ones that everyone has to know (住民票、納税証明書) and then some specialized ones that most people never have to deal with (除籍謄本、生産届出受理証明書). Sure, you can probably intuit their meaning from the kanji, but if it’s said to you all of a sudden on the phone or at city hall, with no visible kanji, I’d say even a fluent speaker could get tripped on what じょせき means in this context unless it was clear you were already talking about 戸籍 immediately before. Certainly it would’ve been far above my level unless I already knew to expect it. So having a lot a contextual knowledge would help there a lot more than raw ability.
So even though my Japanese ability is nothing special, I’ve been able to handle this more-complicated-than-average immigration process on my own because I’m very aware of all the forms and what they’re called, (in no small part because it took a lot of effort to track them down and get copies of them). I know what files the immigration office has under my name, I know what things they’re going to want to verify next time, basically, I know most of what to expect walking in or if the phone rings suddenly. So I don’t have to rely on my vocabulary and listening ability to understand rare documents by their name alone.
I’ve only been here a few years and have been lucky enough to not have had to go the hospital yet, so maybe I’ll be in for a bit of a language shock when I do go. But by reading my 健康診断 results every year I’ve tried to develop some background awareness of what a doctor might be interested in from my files. Especially if there are any 要観察 areas; apparently my cholesterol is lower than average, so I can expect some questions about if I’m getting enough fat in my diet, and I won’t be caught off guard. Of course, I may still be totally underprepared in other areas, but I have enough contextual knowledge about my health and what the doctor will see, to walk in even though my Japanese ability is a lot lower than where I hope to be after another three years.
Basically, try and get briefed on all the documents and items that relate to you. Especially with matters of bureaucracy or procedure, you can anticipate about 80% of the conversation before it happens (high redundancy can be a real blessing for us language learners), with only about 20% being new information or surprises, and that 20% is deeply related to the other 80% so you have a good shot at picking it up on the first try.