Just like @Ditto20 I can’t speak for Japanese, but I can also speak for English, although I would never call myself native level, only fluency.
My first question would be: how easily do you produce Japanese?
Can you think in Japanese naturally, aka it isn’t noticeably slower than your native language, perhaps even same speed? Are there things you actually prefer to do in Japanese? Have you had dreams in Japanese (I don’t actually know which language I’m dreaming in, but I know other people who are)? Would changing your OS language (Windows/Mac/Linux/etc.) to Japanese be a problem? Can you search for things online with Japanese and how well can you read the websites that come up?
I could put more questions here, but I’m sure you got the idea about question #3. 
If producing Japanese is not easily done (N1 doesn’t necessarily mean that), then that would be the next step I’d think.
But assuming you have that, next step would be to learn more things that natives do, more specifically meaning: idioms, sayings, particular structures (English example: the answer to the question “Do you mind?” is backwards. “Yes” when you mean no I don’t mind and “no” when you mean yes I mind.), slang, recognizing dialects (aka where they are from, not necessarily producing them), common stories (aka fairy tales (and similar) just like more or less everyone knows who little red riding hood is in English) and more such things.
Basically the things that gives language more of its cultural roots, more of its color and the things that tend to stump non-native speakers.
As for my tips when it comes to learning that? Consume, consume, consume. Produce, produce, produce. It takes a native speaker all their life or at least all of their childhood to learn this by hearing the fairytales read to them every night by parents, to watching kid cartoons (/animes), to learning in school, from friends, from family, etc.
Read widely, but start with your favorites (manga or novels or a specific genre of novels). Watch widely, but start with your favorites (movies, anime, tv-shows, whatever). Write and speak as much as possible, that way you also get replies. Osmosis basically.
I guess some of it can be studied too, and there is nothing wrong with reading/watching/listening to resources on idioms, sayings, slang, etc. if you find that interesting.
I can produce English without thought. In fact I stopped translating even a little in my head a decade ago or so. In fact words just mean things, they aren’t in a specific language in my head and that produces its own problems if I speak with someone who only knows one of my two main languages (Swedish (mother tongue) and English (learned to fluency)). (A great problem to have and it only happens occasionally.) I never have to think and structure what I say or what I write. I sometimes change it, just like I do in Swedish. But then perfection is not fluency.
Remember that even natives produce their own language wrong, by choice and by mistake, in writing and in speaking. But if you can understand and produce idioms/sayings/culture-references, then that is the next level of fluency after being able to use the language with ease. (MY OPINION, just wanted to mark that with caps to make it very visible.)
PS. I should really learn to be brief. >.<