• Member Since 15th Jun, 2013
  • offline last seen October 11th

Fimfarchive


Ensuring that history is preserved.

Fimfarchive

Fimfarchive is a project which aims to release all stories available on Fimfiction, together with their metadata, as a single archive. The archive is organized by author and could be used for backup, offline reading, or data mining.

A new version will be released each season via BitTorrent, approximately once every three months. When suitable, an xdelta3 patch will also be provided for users who do not wish to redownload unchanged stories.

Note that the archive contains a large number of files. Unzipping it to your file system may not be necessary if the archive is to be used together with some application. If you are a developer, reading directly from the ZIP-file may be preferable.

See the README for more information.

Sep
7th
2017

Fimfarchive 20170907 released! · 7:53pm September 7th

Seventeenth release with 156848 stories.


The complete archive.
Name: fimfarchive-20170907.zip
Size: 4.9 GB
Torrent: Takiri
Direct: Sweden
Magnet: magnet:?xt=urn:btih:c3101a71b5dc64c38a78f1b46a128cd165191226
MD5: 09029ca06a3ad550aba68db00eda4c7f


Excuse me while I go collapse somewhere!

Report Fimfarchive · 253 views ·
Comments ( 149 )
  • Viewing 130 - 149 of 149
Comment posted by darksymphony777 deleted September 13th

2496780
Don't worry about that. Not that it matters anyways ^^

Well, at least you actually do something. After coming home from work, I don't really have the motivation to implement the features for my searcher. ;( But it's good to know, that it's planned. Just would have fit, since I have some vacation at the end of September and could therefore take the time to migrate. ^^

I really wanted to do that, too. Going to 80%, I mean. My boss would not let me though, always telling me, I could come and go when I want anyways and don't have to adhere to strict working times. Yeah, sure, that definitely let's me have more freetime *eyeroll*

" So, unless life happens I'llĀ hopefullyhave plenty of energy left for this project." ==> As a computer scientist, I find this statement really alarming. :twilightsheepish:

if old: ... if new and old: ... if new: ... if not old and new: ...
That's some use of variables :D

2496675

Yeah, I definitely got those release counts all messed up! :facehoof:

The plan is to base the December release on APIv2. There just wasn't enough time for me to do anything other than some basic planning before this (September) release. After all, I do have to start the updater for APIv1 (and for the first APIv2 release) one month before the actual release in order not to flood the site with requests. This will allow APIv2 to mature a bit as well.

I realize I haven't been spending enough time on this project to keep everything running smoothly. I've had to be very careful not to promise anything I know I wouldn't have the energy to implement. For that reason (and a few others as well) I'm currently in the process of stepped down my day job to 80%. So, unless life happens I'll hopefully have plenty of energy left for this project. :pinkiecrazy:

Also, the fetcher rewrite was "finished" "a little while" ago. I have yet to port over all the other tools and utilities I use to make a release. However, this will still make migrating to APIv2 way easier than it was before.

2496369
I guess after having two 16th releases, it is obvious to jump to the 18th one ;P

Anyways, thanks for the info and thank you for your work. Will the new release be done with the new API?

The eighteeth release will be here about one week later than usual. :derpytongue2:

2482936

The sixteenth release contains 153 547 stories.

2408606 how many stories do you have?

So much thanks for this!

I've always got the files pinned on my seedbox! <3

2355019

Great! You're welcome! :twilightsmile:

Thanks to your older work, I've been able to uncover the original version of a story, before the lyrics ban was implemented. Thank you!

2352300
For now I have the basic solution that works with ePub and make searches very quick (DocFetcher). I found three stories I was looking for with it, and I can recommend it to others for similar purposes. So having opportunity to use more powerful search tools would be nice, but it's very low priority.

2351733

Other than writing a crude tool myself, I don't really know of any that could do that. I usually convert things with Calibre, but that really is not quick.

However, if you'd like I can try converting all of them to FB2.

Hello.First of all let me thank you for your work.

I was trying to use fimfarchive to find stories that I've read but can't remember the name by looking for some combination of keywords (or strings) that I can remember.

I ended up using DocFetcher. It spends a few hours to build it's own database about the archive's size but after that it searches almost instantly. For now I found stories I was looking for.
Does anyone have a better solution for full text search of the archive?
Alternatively - to work with the plaintext I can name a few much better tools than DocFetcher. If space on the disk is not the issue, can you recommend some batch converter which would be able to convert entire archive to some plaintext format (presumably conversion from ePub yo HTML will be fastest, but I may be mistaken)? Preferably in a few hours, and not in a few days.

2337861

Very unlikely unless it exactly matches the description of a story. :unsuresweetie:

If I give you a description would you be able to find the story?

2316933 that should be very helpful. And it will help me go in the right direction with my plans. Thank you.

2316887

Chinchillax usually creates spreadsheets for people to use. I haven't seen any for this release though, so I dusted off an old TSV exporter script. Below you'll find an ODS file which you should be able to open in any modern spreadsheet application. It may not contain all the data, but I hope it helps a bit. :twilightsmile:

http://jocketf.se/files/fimfarchive/fimfarchive-20160901-index.ods

2316877 okay, that site helped immensely in finding the name and author of a removed fic I was searching for. Thank you. I wtill wanna make a searchable index that is user friendly

  • Viewing 130 - 149 of 149
Login or register to comment