I think you’re worrying too much - I stopped using graded readers after just level two and I was trying to read native things even before that, so if you’ve completed all of level four you’re probably in a good position to get stuck in.
Why don’t you look at some of the sample pages from the various clubs? Take a look at the sample pages from コンビニ人間, generally regarded as the easiest pick the Intermediate Club have done so far. If that’s too much, then I’m afraid you are probably looking at children’s books, but personally I’ve really enjoyed the ones I’ve read, and the absolute beginners’ club has done some non-fiction ones.
Below the コンビニ人間 level I would highly recommed Zenitendou, read by the Beginners’ Club. It was pretty entertaining and very readable, and it’s essentially six short stories, so much less intimidating than a full-length book.
Otherwise, check out the upcoming absolute beginners’ book, or (and I’m biased here) join Kitty Detectives!