Any time, and don’t be afraid of settling for partial understanding. The only way to get better is by tackling stuff that’s too hard for you at you current level, until it doesn’t feel too hard anymore.
Whatever level you’re at, I also second the notion of turning on Japanese captions for Japanese media that’s still somewhat difficult for you. (It isn’t as good as raw listening practice then, but it can aid comprehension and help you pick up new phrases and vocab that might otherwise have flown by you. Right now I generally try watching shows without captions first, then if I feel like there’s enough I’m not picking up (high-register dialogue, fast delivery, specialized vocab, etc.), I’ll turn Japanese captions on.) This is probably more a down-the-line thing (I’m sorry; I did elementary Japanese years and years ago, and recently came back to learning it starting from a low-intermediate level, so other members can give you better ideas for what to start with at the very beginning), but … whenever you feel like you have enough vocab and grammar under your belt to not be utterly lost, give it a try.