FIMFiction and Calibre

I’ve written a few posts over the years regarding the use of the free program Calibre to copy stories from this site to portable devices supporting the EPUB format; this post represents the current state of this research.

With a few tweaks, it’s possible to support practically every tag and feature FIMFiction supports using Calibre and its FanFicFare plugin (fka the Fanfiction Downloader plugin). Here are the only exceptions:

• Tab indents at the start of each paragraph: Oh that’s right, the site doesn’t do that anymore either.
• [icon]: How many people are actually writing fiction that incorporates Font Awesome icons?
• [math] and [mathblock]: Sorry, just too complicated.
• [embed]: Youtube shows a still image, SoundCloud shows a link, and Fimfiction shows the text unformatted. I could include all the code in the output_css to handle the last one, but I doubt anyone is going to stick this tag inside a story as opposed to a blog entry.

Here is a list of steps to get this solution running:

1. Download and install Calibre. There’s versions for Windows, OS X and Linux.
2. Launch the program and click the “Preferences” button in the toolbar at the top. In the dialog that appears, click the “Plugins” button under “Advanced”. Click the “Get new plugins” button in the bottom-left corner. Find “Generate Cover” in the list that’s presented (either type the name in the “Filter by name” field, or click the Plugin Name header to sort the list alphabetically). Install the Generate Cover plugin—this will do the work of getting the cover illustration of each fanfic you convert.
3. Next install the “FanFicFare” plugin (all one word). I’m going to call this “FFF” from now on, to save typing on my part.
4. Now when I did this, I ended up with so many buttons on my toolbar that sometimes the “FanFicFare” button wasn’t visible, so I had to go into Preferences - Toolbars & menus, and change the order of the buttons, for both “the main toolbar” and “the main toolbar when a device is connected”. Personally, I made it button #2 in both cases, after “Add books”. You don’t really need to have the “Generate Cover” button visible, as it’s called behind the scenes by FFF anyway.
5. One last change to the behavior of Calibre itself that I think is necessary before getting into customizing the FFF plugin: adding a column to the story list so you can tell which stories are complete or not. If you’re going to use Calibre to download updates to stories, and checking each story for updates takes 10 seconds or so, it helps to have a way to only select incomplete stories to check. Click “Preferences”, then “Add your own columns”. Click the “Add custom column” button at the bottom of the dialog box that appears. Set the fields for this new column as follows: Lookup name of “completed” (all lowercase). Column heading of “Completed”. Column type of “Column built from other columns, behaves like tags”. Template of “{tags:contains(Completed, Complete, Incomplete)}”. Search column of “Text”. Do not Show in tags browser. Do not Show as HTML in book details. At this point, you might need to remove some unnecessary columns so that you can see the new one—uncheck the ones you don’t want (the Series, for example, is never set from FIMFiction.net, and if FIMFiction.net is your only source of fanfiction, then Publisher becomes redundant). Click Apply (upper left corner) to make your changes stick.
6. If you click on the little down arrow next to the “FanFicFare” button, you get a little menu. Select “Configure FanFicFare”.
7. You shouldn’t have to change anything on the first (Basic) tab of the configuration dialog. The GUI Option “Keep ‘Add New from URL(s)’ dialog on top” makes it easier to copy and paste fanfic URLs into the appropriate dialog; if that gets annoying, turn it off.

8. The second tab (personal.ini) is where the most extensive changes are going to be made. The tab is one big text document for you to customize. Sections of the document are broken out with titles in square brackets: [defaults] for the Defaults section, [epub] for the EPUB section, [fimfiction.net] for the code specific to this site, and so on.

   1. In the [epub] section, find the output-css code and replace it with the following. Changes are in orange, additions are in red.

      ## Allow customization of CSS.  Make sure to keep at least one space
      ## at the start of each line and to escape % to %%.  Also need
      ## background_color to be in the same section, if included in CSS.
      ## 'adobe-hyphenate: none;' prevents hyphenation on newer Nooks
      ## STR(wG) (1.2.1+ for sure)
      output_css:
       body { background-color: #%(background_color)s;
              text-align: justify;
              margin: 2%%;
      	adobe-hyphenate: none; }
       pre { font-size: 100%%; }
       h1 { text-align: left; }
       h2 { text-align: left; }
       h3 { text-align: left; }
       h4 { text-align: left; }
       h5 { text-align: left; }
       h6 { text-align: left; }
       .CI {
           text-align:center;
           margin-top:0px;
           margin-bottom:0px;
           padding:0px;
           }
       blockquote {
          padding:0 0 0 2.0em;
          margin-left:auto;
          margin-right:auto;
          border-left:5px solid currentColor;
          }
       code     {background-color: #fdf6e3;}
       pre code {
          background-color:#fdf6e3;
          padding:1rem;
          tab-size:4;
          color:#657b83;
          display:block;
          position:relative;
          word-wrap:break-word;
          word-break:break-word;
          white-space:pre-wrap
          }
       pre code:before{
          font-weight:bold;
          display:block;
          position:absolute;
          top:-12px;
          right:0;
          font-size:.75rem;
          background:#fdf6e3;
          padding:0.5em 1.0em;
          line-height:1;
          content:"Code"
          }
       .authors-note {
          background-color:#eee;
          color:#555;
          font-family:arial,sans-serif;
          line-height:1.7em;
          padding:16px;
          border:1px solid #ddd;
          }
       .bbcode-figure-left {
          max-width:40%%;
          float:left;
          margin-right:1.5rem;
          }
       .bbcode-figure-right {
          max-width:40%%;
          float:right;
          margin-left:1.5rem;
          }
       .indented {text-indent: 2ex;}
       .bbcode-indent-1 {text-indent: 2ex;}
       .bbcode-indent-2 {text-indent: 4ex;}
       .bbcode-indent-3 {text-indent: 6ex;}
       .bbcode-indent-4 {text-indent: 8ex;}
       .spoiler  {background-color: #333;}
       .center   {text-align: center;}
       .cover    {text-align: center;}
       .full     {width: 100%%; }
       .quarter  {width: 25%%; }
       .smcap    {font-variant: small-caps;}
       .u        {text-decoration: underline;}
       .bold     {font-weight: bold;}
       .big { font-size: larger; }
       .small { font-size: smaller; }

      Note: If you prefer the way quotes used to be formatted, with gray outlined boxes, replace the blockquote bit with the below:

       blockquote {
          background-color:#f5f5f5;
          padding:10px;
          border:1px solid rgba(0,0,0,0.2);
          margin:10px 0px;
          }
      

   2. Find “nook_img_fix” inside the [epub] section, and change the value from true to false. If this causes images to repeat indefinitely on your reader, then turn it back to true.
   3. In the [fimfiction.net] section, add the line “include_author_notes:true”. This will add author’s notes to stories.

---

Procedure to get a FIMFic e-book onto your device

(For practically anything other than an iPhone/iPad.)

1. Plug your device into your computer, and close any dialogs that pop up.
2. Launch Calibre. It should automatically connect to the device.
3. On the FIMFiction.net website, copy the URLs of one or more stories that you want to download, one URL per line. Personally, I keep a little text document open when I’m going through the site and copy and paste every interesting story URL in there as I go, so that I’ll have the list all together when I’m ready to download. Note that the download process takes a while, so I wouldn’t copy more than ten URLs at a time.
4. Click the arrow next to the “FanFicFare” button on the toolbar, and select “Download from URLs”. If you click the button instead of the arrow, it will try to look for updates on whatever fic you’ve got selected in the list.
5. A little dialog will appear asking for URLs—your copied URLs should already be here. Add any ones you may have missed, and click OK.
6. At the bottom right of the screen, a counter next to the word “Jobs:” will set to the number of stories you wanted, and a little widget will spin. Seconds to minutes later, you’ll get a dialog telling you how many of your stories it managed to download. (If this step fails, it means that either Fimfiction.net implemented a new feature that FFF doesn’t know about yet, or a new feature was added to FFF that accidentally broke downloading stories from Fimfiction.net. In either case, you should first check that you have the latest version of the FFF plugin and if you do, you can go to the official FFF forum thread and politely ask for a fix.) Click OK on the dialog, and the stories will be added to the library. You can double-click a story from the library if you want to see a preview of it.
7. In the list of stories, select the ones you want to send to your device. Then click the “Send to Device” button.
8. To remove stories, go to Calibre, click on the stories on the list/library, and press the Delete key. You will be given the choice to remove the story from the device, from your library, or from both.

The Procedure for iPhones and iPads is unfortunately much more convoluted, at least when using a PC:

1. Before you start, find out what folder your e-books are being stored in on your computer. Launch iTunes and from the Edit menu, select Preferences. Click the Advanced icon to get the iTunes Media folder location. For example, my location is C:\Users\McPoodle\Music\iTunes\iTunes Media\. On Windows 10, this works out to This PC\Music\iTunes\iTunes Media\. Inside that folder should be a Books\ subdirectory—this is the folder you want.
2. If you have any stories already on your iDevice that you want to remove, go to the My Books app, tap on Select, select each book, then tap on Delete. (You can’t remove books from iDevices using either iTunes or Calibre anymore.)
3. If iTunes is not currently running, plug in your iDevice to launch it.
4. Wait until iTunes begins synching with your iDevice and gets to Step 2 of 6 or later. Note: If iTunes decides to update either itself or the OS on your iDevice, you’ll have to wait for this to finish before continuing.
5. Launch Calibre (finally).
6. Click on the Connect/share button, pick “Connect to folder”, and navigate to the location you found in Step #1.
7. Follow steps 3 - 8 in the “non-iDevice” instructions. The last step is optional, as the iTunes sync should have removed any unwanted stories from the Step #1 location.
8. Close Calibre.
9. In iTunes, open your iDevice under “Devices” in the sidebar. Click on “Books”.
10. Open Explorer to the directory from Step 1. You will see one subdirectory for each author of a story you “sent to a device”. Navigate into each of these subdirectories, select the .epub file, and drag it into iTunes, specifically the list of books. Doing this will copy the e-book onto your iDevice.

Comments

[KMCA]

I use calibre too, great service.
I've sent several PM's over the years as various things on the fimfic download crapped out and always got back rather dissatisfying responses so the change became very required.

I also still use the cover art generator by sheer momentum.

[slide_potentiometer]

That's a lot more in-depth than my download epub -> calibre -> mobi -> kindle workflow.

[iisaw]

4872142
Is there an advantage to downloading the epub version? I read on a kindle, too, but I do the html -> calibre -> mobi -> kindle thing. 99% of stories come out fine for me... but no images.

[Wireball]

Suggestion in regards to getting it running, Step 5:

On the custom completed column, I noticed it was tagging all my other (warhammer) books as incomplete, probably because Python uses the last option if in doubt.

Rearranging the Template field like this defaults to Complete if not otherwise specified:

“{tags:contains(Incomplete, Completed, Complete)}”

Update: As McPoodle noted in the response, this would erroneously tags hiatus/etc stories as complete when they aren't.
-

I also ran into a minor issue regarding getting it running, Step 8a:

In sections to modify and add, personal.ini doesn't accept a single '%' sign - it has to be '%%'.
Otherwise it complains "InterpolationSyntaxError: '%' must be followed by '%'..."
It worked once I used double percent signs:

# Modified for fimfiction as per McPoodle directions. Default commented above.
pre { font-size: 100%%; }
h1 { text-align: left; }
h2 { text-align: left; }
h3 { text-align: left; }
h4 { text-align: left; }
h5 { text-align: left; }
h6 { text-align: left; }
# End modification

And in the added section (relevant excerpts shown):

# snip
.bbcode-figure-left {
max-width:40%%;
float:left;
margin-right:1.5rem;
}
.bbcode-figure-right {
max-width:40%%;
float:right;
margin-left:1.5rem;
}
# snip

(Observed on FanFicFare 3.3.0 / Calibre 3.37)

Thank you so much, by the way! I was wondering how to tell the incompletes from the completed ones.

[McPoodle]

4991935

Regarding the Incomplete tag, you're free to use your version, but the reason I had mine is because FimFiction has more than two statuses, and I wanted stuff like "On Hiatus" to be labelled as "Incomplete".

You're absolutely right about the percent symbols, so I fixed the article accordingly.

[Wireball]

4992412
Good point! I was looking at it and thinking, "Surely this many stories couldn't have been completed".
Your settings work better. I bulk-updated the metadata in my other published ebooks to add the tag "completed, "

On a side note, to show content-rating, e.g. Everyone, Teen, etc in the tags:
I had to add a line to FanFicFare's personal.ini fimfiction section:

add_to_include_subject_tags:,rating ##leading comma required

Oddly, this only seemed to be mentioned on page 182 of a 322-page mobileread.mobi "[GUI Plugin] FanFicFare" thread.

Is it normal for the complete/incomplete status, content-rating, and warnings to be hidden from Calibre tags by default? I'm beginning to wonder if maybe something is messed up with my years-old installation.

And where do you usually go to find this information, or are you really good at interpreting Python? :)

[McPoodle]

4996217

I think I searched through the same thread as you did to work out that complete thing...then went to a Python reference page when it didn't work.

[Lumos]

I made a program that downloads multiple stories from bookshelfs, folders and other lists of stories on fimfiction.net. You can choose one of three file formats that are available on this site. You can download it here: https://github.com/GameBiit/fimfiction-stories-downloader