It’s OK. In my opinion, it’s worth it, and will pay off. I’m a Chinese native speaker, and the biggest reason I remember many characters well is that I know how to write them. You’ll also get faster over time once you start to see the patterns in how kanji are written.