@#& *&#$%#$% #$%$#%!!! (recommendation system killed) · 4:06pm Oct 24th, 2014
After the recent site update, the list of users who favorited a story is NO LONGER AVAILABLE.
That's a catastrophe for my recommendation system. With favorites by story, I could just pull the favorites on the top 1000 stories, and that provided enough data to bootstrap the system. More importantly, I could get the favorites off new stories as they come out.
About a year ago, the site stopped giving you the full list and truncated it at the most-recent 200. That was pretty bad. It meant the lists were biased in unpredictable ways.
But now, I have to get favorites from user's pages. That means that to have daily updates, instead of querying 100 new stories a day for a list of who favorited them, I'd have to query 100,000 user pages a day. I can't do that.
I don't think my recommendation system can work anymore. Or, if it did, it wouldn't be able to recommend a story until a month or two after it came out.
I can get crude estimates by taking the current user's list of favorites, finding a list of "similar" users, and querying the favorites off of them. But I don't think that will scale well.
Sounds wonderfully painful.
Everything is horrible forever.
Ouch. That sucks.
I remember needing to use a seemingly similar solution for a problem a few years back, and in that case scaling up actually made it easier as well as more accurate. But of course the devil's definitely in the details with this sort of thing, so I have no idea if it's true in this case too, and it'd certainty be harder to write. Here's to hoping the mentioned upcoming site tweaks render the issue moot; I've really been looking forward to this system.
Have you tried asking knighty to give you access to this data?
I'm pretty sure he's interested in having a good recommendation system for this site.
Man, I'm sorry :-\
2553248
I second the suggestion: he might be interested in it
Xaqueseg at the very least would be intrigued
Can't you just recommend shit you like?
Also I still am pissed they removed the ability to track your own comments.
How were you getting similar users? A dot product of boolean vectors each containing the stories a person has favorited? Was there some weighting associated with each fic? Or an intermediate representation/translation?
I'm asking for two reasons: (1) I'm curious about this and what you tested and found to work/not work, and (2) I don't see why that wouldn't scale well. My laptop can do 40 million floating point multiplications in 50 milliseconds, and that's without a dedicated graphics card.
EDIT: I feel stupid now. Checking 100k users with maybe 100k possible stories would definitely not scale well. Disregard (2).
Is this a bad time to mention that I was the first person to favorite True Colors after that feature went live?
Actually, it probably is. I'm not sure that's relevant at all. I've just kinda wanted to brag it before I forget about it.
2553873
That was ever a feature?
If it interests you, some other user (Nadnerb? I have no idea if I spelled that correctly) created a firefox addon to track their own comments. I expressed interest, and they put it up somewhere public. I can point you towards it if you want.
2553873
How could you do that?
2557526
Yes, Nadnerb .. brendaN
Interested. Do you need to use firefox with each comment it tracks?
2560405
It was always very limited but you use to be able to have a module on your page that displayed your last 5 comments with links. It was not ideal but it was better than nothing.
2560405
I'm not sure what you're asking, and I'm not sure I'd know the answer if I did. But:
It's a firefox addon, so it only tracks comments it sees in firefox. But I gather that it doesn't care when you made the comment (ie, whether the comment was made before you started using the addon), so that it just has to see a comment by you to know about it.
And, of course, you'll only get the list of comments within firefox. It's not like it can export it.
And it seems to only bother with story comments. No comments on blogs or whatever. I'm sure that could be changed, by a determined individual, but you'd have to code it.
You and
2560973
If you want to look into this, hit up https://www.fimfiction.net/blog/376250/my-comments#comment/2498332. Xe's got some explanation/demonstration, a link to the addon package, and a link to the GitHub repo. I ain't done anything with it, so beyond this I think you're on your own.