Okay, so drumroll I talked to my Japanese friend, and
No, you are absolutely right on all accounts 
Akase - The Real Story
嬶も因果な奴さ、
→ my wife is a poor fellow,
俺が道楽して居る間中
→ I was out and about (I checked back with him, and 道楽 indeed means gambling, drinking, sex and these things)
苦労をさせて、
→ and therefore I caused my wife to suffer, (because I spent all the money and left her alone)
とうとう赤瀬の親方にひどい迷惑をかけて
→ finally I even got into trouble with my boss,
お詫びかたがた
→ as an apology (this means to fulfill a duty as apology, e.g. doing the dishes in a restaurant when you can’t pay for the bill)
この山の番人みたいになった
→ I went to the mountains to serve as a guard, (that was the duty he picked/that was chosen for him)
が、
→ but
百姓仕事ばかりさせて
→ this way I imposed all the farm work (at home) on my wife (because I was away)
碌な目にも合わせず、
→ [to summarize all of the above points] he could not do anything good for her (e.g. give her money or something), see also 「ろくな目にあわない」とは？意味や使い方を解説 | 意味解説辞典 (it’s a fixed expression)
揚句にゃ狐にまで取っ憑かれやがった。
→ and on top of it all she went mad.
So bottom line, he knows full well that his past behaviour is the root cause of his wife’s mental illness. (And by the way, that’s why 彦太郎 suddenly runs away - and like Belerith said, we will learn more about that slightly later.)
By the way he said that it’s pretty common in this book that one doesn’t know who the actor of a given part is 
Looks like we’re in for some big learning 