When you use the volitional, you are expressing your own desire.
行こうと思わない directly says you have no will of going.
It actually sounds a bit strong, so while you could use it with a third person talking about an invitation you got but you have no interest in, most people would consider it rude directly answering 行こうと思わない to an invitation.
行かないと思う, on the other hand, is way more neutral. You just think the most likely outcome is you not going, the reason being anything. Schedule overlap, lack of money, no interest… You can use it together with the reasons for not going, too.
And for the exact same reason, 行かないと思う is a lot more related with what will actually happen than 行こうと思わない.
You could say 行こうと思わないけど、行くしかない。 I don’t wanna go, but I have no choice.
But 行かないと思うけど、行くしかない is non-sense, because the first sentence clearly states i’ts unlikely you will go.
Searched the internet a bit and found a nice example:
It’s always so crowded I don’t feel like going. But once it becomes less popular, I might give it a try.
If you wanted to say the same sentence with 行かないと思う the only reasonable way of doing so would be adding a time expression, to make it clear that 行かないと思う refers to a specific moment and not the final outcome.