That’s no crime! My Japanese mangas and books are my textbooks and rarely is there a bit of blank space left in them after I’ve written down all the translations and grammar notes! But yes, addressing the main discussion, I tend to read too closely, looking up everything, and need to learn to relax more and, like you, look up less and so read faster.
But, however you read, the best idea is to read together in a club! It’s fun and provides a bit of discipline and a real boost!