I have still only read through Level 2 of the Graded Readers, so I’m not sure how I would find Levels 3 and 4, but I think that part of the key is realising that when you make the switch to native material it is simply going to be harder.
The philosophy behind the Graded Readers (and I think it’s a good one) is to read lots, easily. Once you are stuck with native material though, it becomes extremely hard to find anything easy enough to read, so you have to accept the ‘slog through it’ approach! Children’s books are often still difficult, because being child-friendly is not the same as being ‘foreign language learner friendly’ - they tend to eschew kanji a lot of the time, focus on simple concepts, and use relatively complex grammar, because children pick up an innate understanding of grammar and this is therefore not such an issue for them.
I second the manga suggestion - it can have more slang, which is unhelpful, but the sentences are much shorter and the pictures aid your understanding with context.
I managed to grind my way through the first volume of Flying Witch way back last year by accepting that I would have to look up lots of words, and I’ve now read the second volume as well by taking a more “ignore what I don’t understand” approach. Yotsuba was easier, and I’d imagine things like Shirokuma Cafe and Chi’s Sweet Home are more on that level. Otherwise I’d also recommend slice-of-life stories, if you won’t find them too boring, because they don’t tend to have too many niche words.
I’ve also got several bilingual books of Japanese folk tales, which are pretty good practice but maybe not for you if you’re looking to avoid childish content. Tuttle publish quite a few.
I would also suggest looking at the picks read through by the beginner book club, because there are often vocab lists available in the associated threads, and your questions may have been asked and answered in the past.