Should I slow down? (Advice appreciated)

In order to make sure you use the language long term, you need to make it part of your life as early as possible. That means reading manga/books and/or communicating with people in Japanese regularly, once you can. In can be a bit hard to start reading without a decent foundation in grammar and vocab, but joining a book club on the forums is a great way to start early because you can get help and motivation from others reading with you.