This is geared towards the authors, but those savvy enough may be able to answer.
According to the analytic tools we use the site has experiencing sharp increases of throughput (majority API hits pulling item information) at every quarter hour mark (:00, :15, :30, :45).
Any idea what extensions/apps hit the site at those marks?
My scripts only run API queries when the user loads a page.
On a side note, I’m working on a data framework for us script authors to use. If other scripters adopt it, it could reduce a LOT of load on the WK servers. Basically, it caches API data until it is determined to be out of date. So, any scripts that use the framework will get instantaneous data from cache most of the time, and redundant API queries will be eliminated.
http://wk.aincrad.cc/ apparently does refreshes at a regular interval, though I can’t imagine it being so specific except by freak coincidence. Also, it defaults to 10 minutes, though has options for other timeframes.
i believe the mobile apps update on the quarter hour, but i’m not sure.
ShotgunLagoon said... i believe the mobile apps update on the quarter hour, but i'm not sure.I believe it wouldn't be going on only recently (last couple of weeks). The app has been popular for a lot longer.
WK Mobile (android) is a possible candidate (but I think similar apps may behave similarly). Here is an explanation: the app learns when new reviews are available and sets an alarm. When the alarm goes off, it checks the queue again to see if they are really available. Since next review time is given as an absolute timestamp, lack of clock synchronization (or the phone set to a wrong timezone) may result in not having reviews available. So the only possible thing to do is polling. I tried to be as gentle as possible: WK Mobile waits a bit before performing the first check and polls after something like 20 secs or so. So you get at least one API call for each device + some extra calls from misconfigured devices. If this is really an issue, the only solution I can think of is to provide the next review time as a relative timestamp (I believe I suggested this eons ago, but I’m not 100% sure – well, not even 20% sure…).
Not likely me. KW syncs up all users every 12, and synchronizes the database every 3 hours, or whenever a user attempts to access an item we don’t have synced in our db.
rfindley said…doesn’t this just move the load from their API to yours? Or are you discussing a framework that each app would make use of personally and hold onto their own cache on a per-app level?
My scripts only run API queries when the user loads a page.
On a side note, I’m working on a data framework for us script authors to use. If other scripters adopt it, it could reduce a LOT of load on the WK servers. Basically, it caches API data until it is determined to be out of date. So, any scripts that use the framework will get instantaneous data from cache most of the time, and redundant API queries will be eliminated.
Tadgh11 said...Doesn't this just move the load from their API to yours? Or are you discussing a framework that each app would make use of personally and hold onto their own cache on a per-app level?The framework is a userscript, and essentially acts as a data broker. Client scripts ask the framework for data, and if that data is not yet cached, the framework gathers it and caches it in localstorage for common use by all scripts that use the framework. It also passively harvests data from WK's ajax requests, so it has a better idea of when items change status. If you use more than one device for WK, there are gaps in its tracking, but I can still deduce much of what has transpired on other devices, and minimize API queries accordingly (e.g. knowing which items have come up for review since the last time the framework updated).
The result is that client scripts get instantaneous data most of the time, and when the framework does need to run some WK API queries, the client can optionally choose to display older data first, then receive notification when the new data is available. For example, by converting the Timeline script to use the framework, it now loads instantly, and provides a plethora of new data and analysis. And it rarely needs to run any WK API queries at all if you access WK from the same device most of the time, due to passive ajax monitoring.
I would love to make use of this outside of WK, for autopopulation of user’s API key, so that when they sign up for kaniwani, they don’t have to go look up their API key. Is this something I could get my eyes on before it’s released so I can work on integrating it?
Tadgh11 said... I would love to make use of this outside of WK, for autopopulation of user's API key, so that when they sign up for kaniwani, they don't have to go look up their API key. Is this something I could get my eyes on before it's released so I can work on integrating it?You are welcome to email me. But the framework only runs while the user is logged into WK, and you would have to find a way to get data across domains. Off the top of my head, I think you'd have to have your own script that runs when the user is on WK, fetches the API key, and uploads it to your server. Sounds more difficult than just having the user get their API key.
ah yeah probably not worth it in that case. I was hoping the key would go into the user’s browser storage. Oh well! I look forward to seeing this released anyhow.