for kids:
Look for the grade level on the cover of the book. As an example, here’s a book that one of the book clubs here read recently. On Amazon there are some pictures of inside pages so you can see the kanji usage (fairly minimal, at this grade level).
for language learners:
This thread has a list and cover pictures of common graded readers for learners of Japanese (there are not as many as there are for Japanese kids):
日本語多読 means extensive reading in Japanese, so a lot of these books will have that on the cover somewhere.
Sorry, I can’t help you with what section to find them in the bookstore because I’ve only bought books online. Maybe somebody else has more information on that…?