Jul
10th
2017
Site Update » Fimfiction API · 12:18pm Jul 10th, 2017
If you're not a developer you can probably ignore this post.
It's been like 6 years, but hey, things take time. The API is currently very WIP still but it's ready for people to get working on in our development chat room.
API documentation can be found at https://www.fimfiction.net/developers/api/v2/docs and you should join the Discord Chat and PM me to add you to the private API channel and I can help you get started. The functionality is very limited right now but I'm dedicating all my time to it at the moment and would love to have people add their input to the process.
Gosh darn it, all the cool stuff happening while I'm on holiday
Can't wait to give it a try when I get back
I don't even know what an API is.
4597534
I think it's actually spelled "apple".
Well, that basically kills off my interest. My main use for an API would be re-implementing the browse functionality so I can implement a "hide stuff I've already read" filter while still guaranteeing a minimum pagination increment.
(Something which can only be implemented in a userscript by sending new requests until enough unread stories have been accumulated... which would violate the "one request per click" rule unless I violated the spirit of it by having the script ask me to click a button five or ten times to queue up "technically allowed XHRs" before it goes to work.)
(Another interesting but disallowed tweak would be to implement infinite scroll using history.replaceState to make the approximate scroll position bookmarkable in a site-friendly way.)
4597541
Doing that would also require potential request spam so I'm not sure how the API would help there specifically? The rule regarding replicating existing functionality is really there to prevent people leeching off Fimfiction as a backend. I want to make it clear that that's not okay upfront so we don't get a situation like Twitter had where they had to screw over all their app developers because for years they let them use Twitter without people ever needing to visit the site.
No API for messages? You are dead to me.
I jest. It looks like a good start.
Wooo!
4597543
It wouldn't be a perfect fix unless you chose to expose read/unread information as a filterable attribute, but it'd be a great improvement if I could combine the higher pagination limit of List View with the presence of some kind of read/unread indicator from Full View. (Actual read/unread checkboxes or bookshelf membership would both work acceptably.)
(Heck, I could implement something I'd consider satisfactory without an API if the pagination limit were raised on Full View or the aforementioned cues were present in the compact view somehow.)
If nothing else, it'd be much more comfortable to reduce, by a factor of 5, the amount of time I spend blocked on the various latencies involved in requesting the next page of results. (Acquiring the button with the mouse and clicking it, waiting for the network round-trip, waiting for my sluggish browser to re-render the base portions of the site template, and re-acquiring the position of the entries so I can begin to do my skim-scroll reading again.)
That's why bookmarkable infinite scrolling also occurred to me as a desirable concept. Regardless of how perfect the solution is, the base problem is the mental drain of scrolling past literally hundreds of top-rated stories I've already read to find the handful sprinkled among them which I haven't yet read. Anything which can reduce that is welcome and will be appreciated.
A reasonable concern and one I don't mind. My biggest reason for hating Twitter for doing that doesn't really apply here.
(I don't use Twitter beyond responding to others specifically because there's no RSS feed and I don't want my e-mail inbox to devolve into "RSS without the organization". Fimfic's update notifications are perfectly fine.)
This is relevant to my interests. I must research it further.
4597572
The problem is the data doesn't exist to do that filtering and it's kind of difficult for it ever to exist based on how Elastic works here. It's not impossible but it's prohibitively difficult to make it a "let's make this one day" kind of project. There would have to be some kind of cached data for stories you've fully read in elastic but then whenever a new chapter is added, every single person who's ever read a chapter on that story needs updating and it just becomes a nightmare.
In the Quick Start Guide leads the endpoint link to a 404. might want to fix that
4597588
I can understand and accept that with regards to actually filtering by that data, but what about some kind of view that combines the increased pagination limit of List View chapter-level read/unread flags or bookshelf membership data that's already displayed in Full View?
I can already craft a terms-compliant userscript if I'm willing to accept "all results hidden" as a risk and that would be far more comfortable if I didn't have to click "next" so many times.
(And it doesn't strictly have to be read/unread status either. Whether my script is hiding based on the absence of unchecked read/unread checkboxes or the status of certain bookshelf toggle buttons, either would be a great help when I'm looking for something new to read. In fact, being able to hide based on bookshelves would work better, since it'd be more flexible, based on how I file away "I'm interested, but not right now" stories.)
4597645
That's a half solution though. If I send down 100 results based on the fast that only 10 will show that's extremely wasteful on resources. I'm absolutely not doing that. The reason I can show many more cards is because they don't select chapters. When stories can have up to 1,000 chapters, searching by longest on full view is already slowish. Its just not going to happen.
Squee!
4597652
What about bookshelf membership? That's story-scoped, so it shouldn't have to select chapters, and I can still use that to build a userscript which hides "stuff I've read" based on how I assign it to bookshelves.
It'd even be more useful, because I could toggle between interpretations like "hide stuff I've shown interest in" (present in any shelf) and "hide only the stuff I've fully read or plan to be caught up on soon" (by personal policy, stuff in my tracking list).
4597666
If you want to read stuff you haven't read in bookshelves...use the unread chapters functionality?
4597671
You misunderstand.
I'm saying that the base functionality I desire is "Hide results that I've put in any bookshelf" and it'd be a nice secondary feature to some times be able to say "On second though, I don't mind if stuff I bookmarked long ago shows up in this 'filter for tagX and sort by rating' query as long as it's my "Read it Sooner" or "Read it Later" shelves. It'll save me having to make two or three queries."
EDIT: I clarified the "bonus feature" part with the underlined text.
*sees new blog from knighty*
thumbs.dreamstime.com/z/school-bullies-lonely-kid-5926115.jpg
Hmm. API...API...Armor Piercing Incendiary?
4597652
Except that if you don't implement the suggestion you are sending those 100 responses ANYWAY! So, it is a no-change situation. In one, 100 results are displayed in which the user ignores 90, and the other only 10 ten are displayed, but in both cases all 100 responses are sent (but one is much more user friendly).
Also, as is the case for ANY situation in which you choose to implement a feature for which no data currently exists (i.e., a flag of some kind), it doesn't matter. The information -- going forward -- will slowly accumulate. I can live with not being able to sort/filter old stories knowing that going forward I will be able to do so, even if it is limited to only NEW stories or to stories I read only after that implementation. Hey, if nothing else, I can quickly reload an old story, thus setting its relevant flag as needed.
Meanwhile, you're doing a great job with the site. Thanks for all your hard work. I really have no complaints about any of it current features.
Neato!
4597671
I know this is a bit off-topic but I think this site needs a really good app for smartphones and tablets. It's kinda a pain in the ass the way it is right now.
4597960
Not likely to happen. I have worked on an app twice and decided it's not worth it currently. I think the mobile site is in a good spot where it is right now and I say that as someone who mainly uses Fimfic nowadays on a phone. I have a long term game plan there.
4597970
I agree, once you get Use to it reading on the phone via a browser is really good. I read almost everything this was and find the phone version quite pleasant.
4597543
If the worry is that users skipping the site would deprive FimFiction of revenue to operate, have you considered using per-call payments? Something like 21.co's Bitcoin Payable API, where each HTTP request has to have an open bitcoin payment channel and pay a certain number of Satoshis per API request?
Or to require app developers to have their users bring their own per-user API keys...and only grant per-user API keys to people who fund you on Patreon, to compensate for the lost ad revenue?
Having been tinkering off-and-on for the past year on an app that syncs my bookshelves with corresponding Calibre metadata, I am thrilled to see this finally become a thing. I'll be happily converting all my site-scraping code to proper API usage ASAP.
Looks overcomplicated, honestly. But I don't really care for that sort of paranoid, micromanagey stuff like invite only API keys, so take that for what you will.
4598351
Not sure how any of this is over complicated... The invite only is just while it's still work on progress
4598193
Would you like access? Like me on discord if you do
Would you be upset if I scraped the entire site periodically, saving any new or altered stories and chapters as an archive?
4598443
Kind of, yes. Considering somebody already does that, meaning it'd just be double the load for zero benefit.
4598494
Are you talking about the user Fimfarchive? If so, would you happen to know the methods Fimfarchive uses? Seeing that s/he probably does not use the API yet, the archives are probably collected in a very roundabout way.
That being said, I have a personal suggestion that will probably be disapproved of, which is that I feel that deletion of anything should be disallowed. Instead, there should be a system somewhat like Github with version control, where previous versions of chapters can be found by going into earlier "commits." Or maybe something like the Deviantart stash, where stories the author wishes to hide can be placed in and accessed by those who really want to, but cannot be truly deleted. This altogether should negate the need for some users to put a huge load on the site by archiving everything.
4598367
I guess I'm just complaining about OAuth and all that token stuff I mean. It's standardized, so I suppose you can mostly ignore it behind a client API, but I think it requires every client application developer to also maintain an active web server. I'd just use a digital signature to verify logins and/or clients, so instead of like...
client -> user | client id plus scopes, plus URL for my server that I totally have
user -> fim | all that stuff
user <- fim | request for approval
user -> fim | okay
user <- fim | go to this URL
user -> somewhere | a token to get a token, sent to aforementioned URL
somewhere -> client | here's the token, which I totally didn't steal
and from then on:
client -> fim | that token-getting token plus client secret that can't be secret unless client is closed source
client <- fim | the real session token, logged in as that user with allowed scopes
I'd have something like...
client -> user | client public key plus scopes, and no URL
user -> fim | all that stuff
user <- fim | request for approval
user -> fim | okay
and from then on:
client -> fim | signature of client key
client <- fim | session token logged in as that user with allowed scopes
Since public keys don't have to be secret, It avoids needing that secret token-getting token and a dedicated, secure web server. It also doesn't (technically) force all clients to be closed source.
The rest of the API seems pretty chill. Though (after complaining about overcomplication) I'd ask for support for sending actual patches, as in... the client remembers the previous chapter contents, and only sends a diff from that, instead of the whole new 40K chapter. That actually gets annoying when you just uploaded your chapter, then you see a one-word typo and you gotta wait 20s to upload the whole chapter again. It's a minor complaint though.
4598684
Frankly I have no idea what system you just described there. There's a reason OAuth is well used, because it's battle tested and secure. Adhoc solutions wouldn't add anything other than potential concerns. OAuth 2 is ludicrously simple to set up, and if you don't want to do anything with users, you just use client credentials which does not require a secure server, just ability to make SSL requests.
4598692
Yes, I'm aware that if you just want to read stories and not change anything, you don't need to authenticate.
Anyway feel free to ignore me. I just won't be able to write clients for it any time soon. (Trust me that's not a big loss.)
4598494
Are you talking about fimfetch, or somebody else?
Thanks for this post. I might want to check it out later this year or next year when I have a better understanding of app development and Xcode. Seriously, FimFiction needs its own iOS and Android app and I plan to make one.
4598372
Thanks. I'll look into downloading Discord and setting up an account in the next couple days. Not sure how often I'll be able to participate (thin walls and awkward working hours make voice chat a dubious prospect), but I'll give it a go.
Yay! I'm excited to try this out!
4599150
It's just text chat
maybe this could mean the fimfiction app can soon become a reality? sure a browser on the phone is good, but a dedicated app is usually better :)
4598609
For each release I currently traverse all story IDs using the old API. I do this over the course of an entire month. When a new or updated story is detected its contents is fetched from the story downloader. The stories are then converted to EPUB and passed through Calibre to ensure they work on most e-readers.
The new API will in time make it possible to include more meta data. Also, to be kinder to the server.
4602242
Is there a way to customize the epubs on download? I find that in my epub readers (I've tried several) there are no gaps between paragraphs or indentations, thus it is difficult to differentiate where a paragraph starts and ends.
Sorry if I don't know what I am talking about, but doesn't the website already support this by adding the '&unread=1' command to the end of the URL any time you perform a search? I know it's only directly added by the site itself in your bookshelves (as signified by the banner at the top stating "You are viewing unread chapters only. Click here to view all stories" or something like that), but I distinctly remember being able to add it at the end of any search URL manually if wanted, I.E. https://www.fimfiction.net/stories?q=&order=updated&unread=1 would provide a list of all stories with unread chapters.
My actual point in all of this is that the functionality seems to already exist, so adding it to the API would be, to my mind, possible but pointless, since a simple script could add the necessary URL addition as needed.
A better question might be why can't we just hit a checkbox to add this to the search instead of doing so manually?
4603402
There are some third party tools for that. There might also be some setting in your e-reader for changing how paragraphs are separated. If not, you could try a tool like Calibre yourself. If all else fails (and you know how) you could even try editing the style sheet of the book, assuming your e-reader uses EPUB.
How does this file converted by Calibre (Memory Pending by Kiroberos) look on it?
You might also want to try djazz's Chrome extension.
4602242
I wonder, whether this makes it senseless, that I spent a day to write a character tagger for the stories I pulled from your archive? (e.g. https://ponystories.de/?id=13477) Well, at least, I got richer in experience XD
And I wanted to take the opportunity to thank you again for your work :)
4606380
That's pretty awesome! That's is the first user-friendly interface I've seen for the archive. It would be very interesting to see how your tagging compares to the tags that authors set themselves. Yours are probably a bit more objective.
I strongly doubt I'll be able to finish moving to the new API before the next release unless I delay it. Besides, nothing is ever a waste if you learn something from it.