Leebo already mentioned SOV structure, which is a big one. And there’s stuff like how much a language leaves implicit vs how much is made explicit, politeness levels which may or may not exist, ways of referring to people, cultural foundations, and a whole host of other things.
I’d say in the end the difficulty of any language just boils down to what you’re used to. German is easy for me to learn because I know Dutch - the vocabulary is similar, the basic culture it’s based on is similar, the grammar is similar (nearly identical, even, cases notwithstanding)… But for a Japanese speaker German would be a lot harder.
In turn, Japanese is a fairly hard language for me to learn. Not necessarily because of the grammar - I think that’s more of an initial hurdle and it really is relatively simple - but because the language as a whole is so different from what I’m used to.
It may be, but it does cover the basic idea of why Japanese grammar is fairly simple, IMO.
Sure, longer sentences look more complex, but I’d say in most cases they’re not actually more complex, just longer. And because you’re translating them in your head to something you’re familiar with, with an entirely different structure, that feels complex because you have to juggle a lot of information at once. The structure itself, however, is quite simple.
Even if I say something like 昨日は忙しかったからなかなか疲れていて今日も忙しいからよく休めないけど、明日は全然忙しくなくてきたから何もしないと思う is that really more complex than 象は鼻が長い or is it just more of the same? It’s a bunch of clauses connected together in honestly very clear ways by particles. What makes it harder to parse is not that it’s complex, just that it’s unfamiliar. But if you’re familiar enough with the grammar, you can spot the relationships between clauses even before you’ve parsed the clauses themselves.
It takes time to reach that point, and I’m nowhere near it myself. But it’s how you read English too - sentence structure and certain signal words tell you how to connect the meaning of words together, and you end up with an understanding of what a sentence said even if it’s very long. You’re just used to the way English does that, and not to the way Japanese does it. Having looked at both German and Japanese from a language learner’s perspective… yeah, I really do feel Japanese has simpler grammar between the two.
Eh, I’d say hard has more to do with effort than time. It’s just that people tend to conflate the two 
Learning Japanese takes both time and effort, so in that sense, sure it’s hard. But that’s no reason not to do it unless you don’t have enough of an interest to (which is also fine).
