API data incorrect?

The app is working again! I just refreshed and everything went back to normal ^^ Thanks!

Great! Fix isnā€™t out yet thoughā€¦ Working its way up the pipeline, but looks like the cache expired for you and generated the correct info :slight_smile:

2 Likes

You guys seem to have a lot of load issues (Iā€™m assuming the cache project was to reduce API load), are you trying to downsize your instance(s), or is WK just getting that much more popular?

Hotfix is live now.

3 Likes

Iā€™ve been looking into this. The API data is fine. But it seems like something else is going on. Normally, my script replaces the ā€œdata-contentā€ attribute of the progress items, and the WK javascript uses (or used to use) that attribute to populate the auto-popover box. But now, it doesnā€™t seem to use that attribute anymore. When I try to manually change the data-content attribute, some event seems to detect the change, replaces what Iā€™ve put in there with an incomplete piece of html. And when I hover over an item, whatever was in data-content attribute is not placed in the popup box.

Really? Oo thatā€™s funny because Iā€™ve been having this problem since yesterday. The website was working fine to me but the app simply wouldnā€™t refresh after doing the reviews. The moment you commented, I tried and everything went back to normal.

Oh, well ^^

Thatā€™s consistent with the caching viet is talking about (that they implemented yesterday). If they set the cache TTL long enough, it may have been over a day before it was cleared/refreshed.

1 Like

The updates the last few weeks dramatically reduced response times and error rates and increased throughput. Adding additional caching is also helping with that.

However, we are now running into a different issue. As you probably know we use Heroku to run WK. We procure X numbers of dynos to run the site. Every once and a while one dynoā€™s memory footprint would just explode (havenā€™t found the source issue yet) and itā€™ll be forced into memory swap, which makes the dyno very slow. We have no indication of memory leaking since all other dynos follow a nice memory profile (very very slight bloat, but thats about it). The dyno gets autorestarted once it reaches 5x the allocated memory space. Really wish this was user configurable so we can configure it to 1x.

Another unfortunate thing with this is Heroku does not do smart load balancingā€¦ it does a weird random algorithm. You can read more about it here. This sucks because that means requests are getting funneled into the problematic dyno. All it takes is one dyno out of many to reduce performance dramatically ;\

2 Likes

Gotcha, so the goal is to reduce load which in turn reduces response times because you have a hardware bottleneck somewhere in your heroku sizing.

We had a problem like this in our data center before because the default JVM garbage collector was waiting too long to run, so it was a big job everytime it finally did run. Increasing the memory for the instance fixed this, although I canā€™t explain how.

Good info about Heroku, I havenā€™t used it before (only AWS) and I was considering it for a side project.

I use the wanikani mobile and since last night itā€™s been stuck on 20 reviews and even when I actually do have reviews it just sends me to the summary of my last review session rather than start the new reviews.

Awesome, my API calls are now returning the right data for next_review_date. Thanks! :smile:

Oh, I see!! Sorry for my ignorance :stuck_out_tongue: I clearly have 0 knowledge in this subject D: Iā€™m more of a Food guy (majoring in Nutrition Sciences).

Appreciate the explanation ^^

No worries man, always happy to explain what I know (and occasionally a little I donā€™t >.>)

1 Like

Yep, all fine now. Thanks for the prompt attention!

A coworker of mine recommended this to find app memory leaks in heroku:

I just now learned what a dyno is (I had assumed it was similar to an AWS EC2 instance, which is seems to be). Good teachable moment.

I always thought it was something you find in a natural science museum. :stuck_out_tongue_winking_eye:

1 Like

I think I found it. Was looking at the request logs every time it happens and think I identified the pattern and the problem. Looks to be an n+1 issue, which is an easy fix.

I miss troubleshooting :cry:

I just had my next 2 years scoped out for data migrations. I just spent the last year moving Oracle to Postgres and Iā€™m totally burned out -.-

Congrats on your issue though, sounds like that was a tough find!

Too early to celebrateā€¦ Will relax when it ships and nothing explodes x.x

5 Likes