Fimfarchive 20150601 released!

Eighth release with 112180 stories.

---

The complete archive.
Name: fimfarchive-20150601.zip
Info hash: 82557E5E1DEA61B470D9A65E94FFBE53CECD9F43
Size: 3.3 GB
Torrent: Torcache
Magnet: magnet:?xt=urn:btih:82557e5e1dea61b470d9a65e94ffbe53cecd9f43

The xdelta3 patch.
Name: fimfarchive-20150307-to-20150601.xdelta3
Info hash: A21B5051EC22ED64FB180D3ABB5B4A4CB9ECE3DE
Size: 0.4 GB
Torrent: Torcache
Magnet: magnet:?xt=urn:btih:a21b5051ec22ed64fb180d3abb5b4a4cb9ece3de

---

Please make sure to read section 3 about formatting in the README file if you are planning to read stories from Fimfarchive in an e-reader. If you do not, then you may end up missing large parts of the stories.

Comments

[Adin Terim]

Is there a reason why you don't use FanFicFare to download the stories with formating and authors notes intact? Or would that be simply too slow to do?

[Cold Spike]

I have two questions. What do the patches do? Are they newer stories for those that have already downloaded the original archive?

Also, can you tell programs like Calibre to mass add all stories? Is this possible without making things a mess?

[Fimfarchive]

3115134

I have two questions. What do the patches do? Are they newer stories for those that have already downloaded the original archive?

The patches are for upgrading from an older version of Fimfarchive, in this case from release 2015-03-07 to release 2015-06-01. I wouldn't worry about using the patches unless you already have the previous version and really need to save bandwidth. Since I haven't documented how to use the patches, they require that you learn how to use xdelta3 yourself.

Also, can you tell programs like Calibre to mass add all stories? Is this possible without making things a mess?

They can, I just tried using the "Add books --> Add multiple books from archive --> Multiple books per directory in the archive" option in Calibre. It will however make a bit of a mess since it creates a lot of files and directories in the library (at least 350 000). It would take a very long time to import all the books that way, but it certainly is possible.

[Fimfarchive]

3115104

Is there a reason why you don't use FanFicFare to download the stories with formating and authors notes intact? Or would that be simply too slow to do?

I created Fimfarchive long before I heard about FanFictionDownLoader (or FanFicFare, as it is called now). When I first started with Fimfarchive there really wasn't any story content that wasn't included in the EPUB download. So, when someone actually told me about FanFictionDownLoader I didn't see much point in switching over to that.

I also saw that FanFictionDownLoader works by scraping information from Fimfiction's story viewer instead of using Fimficition's API or story downloader. This puts a far higher load on Fimficiton's servers, and is something that a Fimfiction staff member has asked me not to do.

The stories in Fimfarchive do contain formatting, but sadly no author notes.

[Cold Spike]

3115422 How much of a mess? If I were to have a dedicated machine or server just for Fimfiction, would it matter if it made that many files?

[Fimfarchive]

3115908

I'd say it wouldn't be that much of a problem on your regular computer since Calibre puts everything in its library directory. If it causes too many issues you can probably just remove it. Calibre structures its library in the following way:

/author_name/story_title/{cover.png, story_title.epub}

The main issue with this would be that there's a lot of author names (39218, to be exact). I don't think your computer would have much of an issue with that, but it can cause a few problems. For example, it may take a while to open the library directory.

If you get any issues really depend on what operating system and file system you're using.

[Adin Terim]

3115501
That makes sense.

[Bad Horse]

I think I preferred indexing by story number rather than by story name. Particularly because some stories don't start with letters in A-Z.

3115104 What is FanFicFare?

[Bad Horse]

3115501 What do you mean by "Fimficition's story downloader"?

[Fimfarchive]

3326253

For Fimfarchive?

I figured it would be more user friendly to organize the archive by author and then title, especially for people who don't want to bother with the index file. It has been that way since the first release two years ago. The index file is organized by story ID though.

What Calibre does I have no control over.

3326256

Ah, sorry. I usually refer to that as the "story exporter". It's just the regular HTML and EPUB download links on Fimfiction.

[Adin Terim]

3326253
FanFicFare is a calibre plugin that downloads fanfiction from a wide verity of sites (100+).

I find the epub files it generates to have the best quality formatting, preferring it over the builtin epub generator. As Fimfarchive said it scrapes the story data rather than using the built in epub generator so it has access to the author's note box and cover image.

[Bad Horse]

3326661 I suppose it's more convenient for people browsing the archive, but it's harder for my automated scripts to use it now, because the names used are not the actual names on fimfiction. I wonder whether mapping uppercase into lowercase and spaces into underscores causes collisions, and what happens to unicode characters. knighty restricted the characters allowed in usernames about a year ago, so that's less of a problem now.

[Fimfarchive]

3326930

True, you can find the path of each story in the index though.

>>> import json
>>> with open('index.json') as fobj:
... meta = json.load(fobj)
...
>>> meta['9']['path']
'epub/s/sethisto-18/the_greatest_equine_who_has_ever_lived-9.epub'

[Bad Horse]

There are some files that have entries in the big json file for June 1 that didn't exist on Feb. 28 2015, and didn't exist yesterday, because the stories had been deleted. Story 327, for instance. How did they end up in the json file? Are those older entries that weren't replaced? I want to know because number of views for different stories isn't comparable if retrieved on different dates.

[Bad Horse]

3328967 Yes, those are old data that you must be reloading from your database. The api returns
{"error":"Invalid story id"}
when a story has been deleted. Could you include a file that gives the date of retrieval for each story?

[Fimfarchive]

3328967

I looked into the EPUB meta data and saw that the story had been fetched sometime around 2014-04-05, so it was definitely in the archive before February 2015. I confirmed this by looking at the index for fimfarchive-20140418.

It would definitely be a good idea to add dates for when a story was downloaded and for when its meta data was updated. I'm not sure when I'd actually get around to doing that though since it would take quite a bit of effort going through all the raw data from previous releases to get accurate dates.

All meta data is updated for all currently available stories with each Fimfarchive release. That cannot be done for unavailable stories, but it is not possible for people to vote on those stories regardless. Do note that even when the meta data is updated there will still be a three week gap between the first and the last story in the archive. I cannot do anything about that since fetching the information too quickly would put an unreasonable load on the Fimfiction servers.

[shelvacu]

Hey I'm trying to download all the files (or rather, all the xdelta files and files needed to get all the zips) and got all of them except this delta file, fimfarchive-20150307-to-20150601.xdelta3

I tried the magnet link and it has yet to fetch the metadata, and torcache.net seems to have been replaced by spam.

Having similar issues trying to download fimfarchive-20150601.zip

Hopefully I just need to wait a little longer, but in case that won't work, could you post an alternate torrent link please?

Nevermind, just had to have patience :)

However you should probably still remove the torcache links

[Fimfarchive]

4766130

Sorry, I completely missed your message.

While too late, you can find previous releases at http://jocketf.se/files/fimfarchive