Perfect FIMFiction EPUBs: Calibre and the Fanfiction Download plugin · 6:11am Apr 19th, 2014
Note:This post has been supplanted; see here.
Recently, I wrote a blog post comparing various methods of getting FIMFiction.net stories onto an Apple product/iBooks app, for portable reading. I had concluded that the Fanfiction Downloader plugin for the free eBook organizing program Calibre was the closest of these methods to perfect, except that it couldn’t handle [quote] tags correctly.
Well, I got together with JimmXinu, the developer of that plugin, and he was able to fix the problem, so now I can definitely proclaim that this solution is the best one, at least for those of you using EPUB e-readers. The solution may very well work for other e-reader formats, but I don’t have the hardware to back that claim. As a bonus for using this solution, you now have a library program for organizing all of your downloaded fanfiction, including being able to quickly download new chapters for unfinished stories that you are following. Oh, and did I mention that the plugin inserts this neat summary page at the beginning of each story it downloads that includes all of the descriptions and tags from the main story page on FIMFiction.net? (Feel free to use this method to download my unpublished “Tag Test” story to check that it does everything the site supports.)
So, here is what you need to do if you’d like to use this solution. Yes, I agree that it is a long list. I am a perfectionist, and I wanted something that looked as close to what I see in a browser as possible.
1. Download and install Calibre. There’s versions for Windows, OS X and Linux.
2. Launch the program and click the “Preferences” button in the toolbar at the top. In the dialog that appears, click the “Plugins” button under “Advanced”. Click the “Get new plugins” button in the bottom-left corner. Find “Generate Cover” in the list that’s presented (either type the name in the “Filter by name” field, or click the Plugin Name header to sort the list alphabetically). Install the Generate Cover plugin—this will do the work of getting the cover illustration of each fanfic you convert.
3. Next install the “FanFictionDownLoader” plugin (all one word). I’m going to call this “FFDL” from now on, to save typing on my part.
4. Now when I did this, I ended up with so many buttons on my toolbar that sometimes the “FanFiction DL” button wasn’t visible, so I had to go into Preferences - Toolbar, and change the order of the buttons, for both “the main toolbar” and “the main toolbar when a device is connected”. Personally, I made it button #2 in both cases, after “Add books”. You don’t really need to have the “Generate Cover” button visible, as it’s called behind the scenes by FFDL anyway.
5. One last change to the behavior of Calibre itself that I think is necessary before getting into customizing the FFDL plugin: adding a column to the story list so you can tell which stories are complete or not. If you’re going to use Calibre to download updates to stories, and checking each story for updates takes 10 seconds or so, it helps to have a way to only select incomplete stories to check. Click “Preferences”, then “Add your own columns”. Click the “Add custom column” button at the bottom of the dialog box that appears. Set the fields for this new column as follows: Lookup name of “completed” (all lowercase). Column heading of “Completed”. Column type of “Column built from other columns, behaves like tags”. Template of “{tags:contains(Completed, Complete, Incomplete)}”. Search column of “Text”. Do not Show in tags browser. Do not Show as HTML in book details. At this point, you might need to remove some unnecessary columns so that you can see the new one—uncheck the ones you don’t want (the Series, for example, is never set from FIMFiction.net, and if FIMFiction.net is your only source of fanfiction, then Publisher becomes redundant). Click Apply (upper left corner) to make your changes stick.
6. If you click on the little down arrow next to the “FanFiction DL” button, you get a little menu. Select “Configure Plugin”.
7. On the first tab (Basic) of the configuration dialog, check the following options: Default Update EPUB Cover when Updating EPUB [Default Options on Download section], Update Calibre Cover when Updating Metadata [Updating Calibre Options section], Keep ‘Add New from URL(s)’ dialog on top [GUI Options] and Include images in EPUBs [Misc Options section]. The “New from URL” option makes it easier to copy and paste fanfic URLs into the appropriate dialog, but it could potentially get in your way, so setting that is a personal preference. The other changes I had you make control adding the cover and other images to the EPUB.
8. The second tab (personal.ini) is where the most extensive changes are going to be made. The tab is one big text document for you to customize. Sections of the document are broken out with titles in square brackets: [defaults] for the Defaults section, [epub] for the EPUB section, and so on.
8a. In the [defaults] section, copy and paste in the following. It doesn’t matter where, just so long as it’s between [defaults] and [epub]. This change will prevent the plugin from stripping out parts of the story formatting that we’ll need to make the EPUB look like the website.
## If set true, any style attributes on tags in the story HTML will be
## kept. Useful for keeping extra colors & formatting from original.
keep_style_attr: true
8b. In the [epub] section, copy and paste all of the following. The output_css part only added the following compared to the default: blockquote (to make it look like FIMFiction.net’s [quote] tag), .authors-note (to make it look like FIMFiction.net’s authors’ notes), and .indented (to add a small indent in front of paragraphs that started with a tab character in BBCode, just like FIMFiction.net). The last addition, nook_image_fix, apparently breaks the story if exported to a Barnes & Noble Nook e-reader, but the default setting forces inline images to be forced onto their own line. Setting it to false fixes this.
## output background color--only used by html and epub (and ignored in
## epub by many readers). Included below in output_css--will be
## ignored if not in output_css.
background_color: ffffff## Allow customization of CSS. Make sure to keep at least one space
## at the start of each line and to escape % to %%. Also need
## background_color to be in the same section, if included in CSS.
## 'adobe-hyphenate: none;' prevents hyphenation on newer Nooks
## STR(wG) (1.2.1+ for sure)
output_css:
body { background-color: #%(background_color)s;
text-align: justify;
margin: 2%%;
adobe-hyphenate: none; }
pre { font-size: x-small; }
sml { font-size: small; }
h1 { text-align: center; }
h2 { text-align: center; }
h3 { text-align: center; }
h4 { text-align: center; }
h5 { text-align: center; }
h6 { text-align: center; }
.CI {
text-align:center;
margin-top:0px;
margin-bottom:0px;
padding:0px;
}
blockquote {
background-color:#f5f5f5;
padding:10px;
border: 1px solid rgba(0,0,0,0.2);
margin: 10px 0px;
}
.authors-note {
background-color:#eeeeee;
color:#555555;
font-family:arial,sans-serif;
line-height:1.7em;
padding:16px;
border: 1px solid #dddddd;
}
.center {text-align: center;}
.cover {text-align: center;}
.full {width: 100%%; }
.quarter {width: 25%%; }
.indented {text-indent: 1em; }
.smcap {font-variant: small-caps;}
.u {text-decoration: underline;}
.bold {font-weight: bold;}## if the <img> tag doesn't have a div or a p around it, nook gets
## confused and displays it on every page after that under the text
## for the rest of the chapter. I doubt adding a div around the img
## will break any other readers, but in case it does, the fix can be
## turned off. This setting is not used if replace_br_with_p is
## true--replace_br_with_p also fixes the problem.
nook_img_fix:false
Note: It looks like FIMFiction is stripping out some initial spaces that the Personal.ini file needs to work. In the part where it's saying that every line needs to start with at least one space, put in one space. So a space before “body”, a space before “adobe-hyphenate”, a space before “pre”, and so on until you reach “.u” and “.bold”.
Finally, these are the steps to get a perfect EPUB onto your iPhone/iPad/iWhatever:
1. Plug your device into your computer, which should launch iTunes (launch iTunes if it doesn’t).
2. Launch Calibre.
3. Click the “Connect/share” button in Calibre’s toolbar. In the menu that appears, select “Connect to iTunes”. In the list of stories, a new column called “On Device” will appear, as well as some new buttons in the toolbar.
4. On the FIMFiction.net website, copy the URLs of one or more stories that you want to download, one URL per line. Personally, I keep a little text document open when I’m going through the site and copy and paste every interesting story URL in there as I go, so that I’ll have the list all together when I’m ready to download. Note that the download process takes a while, so I wouldn’t copy more than ten URLs at a time.
5. Click the arrow next to the “FanfictionDL” button on the toolbar, and select “Add new from URL(s)”. If you click the button instead of the arrow, it will try to look for updates on whatever fic you’ve got selected in the list.
6. A little dialog will appear asking for URLs—your copied URLs should already be here. Add any ones you may have missed, and click OK.
7. At the bottom right of the screen, a counter next to the word “Jobs:” will set to the number of stories you wanted, and a little widget will spin. Seconds to minutes later, you’ll get a dialog telling you how many of your stories it managed to download. (Every once in a while, the plugin chokes on a story—that’s when you’ve got to bug politely solicit JimmXinu to update the plugin to handle it.) Click OK on the dialog, and the stories will be added to the library. You can double-click a story from the library if you want to see a preview of it.
8. In the list of stories, select the ones you want to send to your device. Then click the “Send to Device” button to send them to iTunes.
9. Switch to iTunes. Click the iPhone button on the upper right (or iPad button or iWhatever button). In the list of items on the top, click the Books button. You will get a list of all of the books you sent from Calibre. Check the boxes next to each story you actually want on your device, then click the Sync button at the bottom to send them.
10. To remove stories, go to Calibre, click on the stories on the list/library, and press the Delete key. You will be given the choice to remove the story from the device, from your library, or from both. (Obviously, you have to use the “Connect/share” button first if you just launched Calibre and you want to remove them from your device.) Go back into iTunes and into the Books part of your device, and the stories should no longer be there. Click Sync to, well, sync up your device.
11. Finally, to prevent iTunes from locking up, perform the following steps in order when you want to disconnect your device: Either shut down Calibre, or click the arrow next to the “Device” button and select “Eject this device”. If you're browsing inside the device in iTunes, click the Done button in the upper right. Either shut down iTunes or click the eject button that’s inside the iDevice button. Then unplug the device.
I think that should cover it. Hopefully I didn’t just unleash a deluge of questions because of my unclear instructions.
Great tips! Thanks for all the info, I'll be sure to give this a try!
Nice. Note for any non-iDevice-user, Calibre can usually sync directly to almost any other eReader without going through iTunes, or whatever other software you are using. Many ebook reader users use Calibre as their main library organizer/syncer instead of the software that came with their device, since it often works better.
Thanks for this. I had been mutilating html files with regex to get them on onto my kindle, but this is so much easier.
I'm using a kobo and I get occasional issues with epubs from here, usually missing entire chapters, but this fixed that and added some really nice stuff.
Awesome ^^ Now I just need a way of passing all of the URLs in my Read Later list into the plugin. :applejack unsure:
EDIT: I wrote a small parser in Java to get the URLs from the page source. If it's of any help: http://tinyurl.com/pk6hvpm
It worked very well with kindle – had the best results changing the default output format to mobi.
Thanks for the guide! This is perfect for long airplane trips or other situations where you do not have access to the internet.
im having trouble. antime i try step 5-7 of getting stories no numbers apear next to the jobs. something goes wrong. i looked at the log and this is what showed
Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
File contains parsing errors: <???>
[line 73]: .CI {
[line 79]: blockquote {
[line 85]: .authors-note {
http://www.fimfiction.net/story/185827/
nvm fixed it
Holy shit, I can't thank you enough for this post! Everything's so easy now, it's amazing!
Found this a couple days ago and need to actually implement this into my Calibre, but a couple questions first: A) If I follow all these steps, including the optional 8a, will it download in-line pics with the chapters? (For example AestheticB's The Immortal Game) And B) If I do step 8a, will it also download various text colors set by the author like in Shortskirtsandexplosion's Background Pony. I specifically linked chapter 9 as it was the best one to give an example of without having to dig too far through the chapters.
3068292
You might have a little bit of trouble--the plugin was renamed from "FanfictionDownloader" to "FanFicFare" quite recently, so search for the plugin under the new name instead of the old one.
I'll eventually put up a blog post on how the new version differs from the old version. So far what I've found is that the "Include images in EPUBs" option from Step 7 has moved into Step 8b: Where you see "#include_images:true" after the "output_css" part in personal.ini, remove the "#" character.
Now with that said, I downloaded the two stories you linked to just to be sure that it still worked, and this is what I found:
* "The Immortal Game": I went through the first chapter, and saw the Celestia and Luna cutie marks in their proper places.
* "Background Pony", Chapter 9: The greens, the magentas, and the cornflower blues all come through. (This part had better have worked--I wrote my own fanfic that was color-dependent. That's why I went to all the trouble of developing this method in the first place!)
Oh, and one final note: Step 8a isn't optional. All of the steps labelled "8" are logically the same step, but the instructions are so long I split them up into a general Step 8 and specific Steps 8a and 8b.
3068376 thankfully I decided to read the comments before asking for help. You should really update that in the post, mostly so people I plan on sending here don't have the same issue on finding the fanfic plugin
What would be most useful right now is having a function that can find and add the columns for Age Rating, Warnings, and when it was last updated.
I can go into the xhtml parts itself, but I don't know the python language or how to make a script that takes that information and puts it into their own columns.
3599088 There's probably a plugin for that...
it was working fine but now i keep running into this
Status
Title
Author
Comment
URL
Bad
Unknown
Unknown
HTTP Error 404:
http://www.fimfiction.net/story/183040/
3988753
Hmm...I was able to add it to Calibre just fine. Just to be sure we have the most-current versions of everything, my Calibre is 2.57.1, and my FanFicFare plugin is 2.3.3. Also, I decided to download the URL http://www.fimfiction.net/story/183040/besides-the-will-of-evil instead of http://www.fimfiction.net/story/183040/ -- not sure if that will make any difference.
I skimmed through the story looking for things that might cause your download to fail, and saw YouTube videos embedded at the start of Chapters 43 & 48--I wonder if that's causing the problem?
3991505 okay my fanfickfare is out of date. where do i find the updated version and when i update it do i have ti mess with the configuration again?
3991505 i tried it with the new plugin and it worked. thank you so much, i have to go on a very long road trip soon and thought i wouldnt be able to read any of my favorite stories
A wonderful guide and most effective in the initial setup, however, according to the information I can find, Calibre doesn't play well with any iOS device running iOS 9. (Specifically the go-to one, Marvin). Any suggestions?
4107116
I'm afraid not--the only Apple device I own is my iPhone. Maybe try it under a Windows emulator?
4107235 Sorry, I meant specifically for an iPhone or an iPad. Do you have this system operational for your iPhone running iOS 9?
Oh. Yes, I have iOS 9.3.3 on my iPhone, and I've never had any trouble with Calibre. But it's an iPhone 5S, not a 6--maybe that makes a difference.
Then key issue with Calibre with me is that you have to run everything in the right order, or one or more programs will hang:
1. Connect the phone to your PC to get iTunes to launch.
2. Wait for synchronization to start before launching Calibre.
3. Connect to iTunes from Calibre.
4. Move any eBooks you want on or off of iTunes using Calibre.
5. Exit Calibre.
6. Go into the Books section of iTunes to re-select the books you just transferred over, and transfer them from iTunes to your iPhone.
7. Exit iTunes.
If you're being told that Calibre and iTunes 9 don't get along, maybe it's only when you're using Calibre without the FanFicFare plugin?
Do you happen to have an updated post about this? I'm trying to re-integrate Calibre back into how I read FIMFiction and I can't find any of the plugins on Calibre's plugin store. Any help would be appreciated.
The plugin's called "FanFicFare" instead of "FanFictionDownloader" now. And you shouldn't have to make any of those modifications I listed...I think.
4586037
Mmm... Is there a way to have the tags and the "completed status" from Fanficfare while downloading th the epub from fimfic?
4798389
), so what I do is download the epub directly, open it in my editor of choice (Sigil) and add my own front page with all the data that FanficFare usually adds. This is also when I fix any other problems I find from the download.
There have been cases when FanficFare utterly fails to download a fic (they usually fix the bug with the next release, but I want to read that story now....
This is a tremendous pain. But I am detail oriented to an obsessive degree, so I actually get a kick from it.
Your individual mileage my vary.
4800757
Oh. I'll try sigil then, I didn't knew it hehe.
Thanks!