Epubs and statistics · 10:04am Aug 11th, 2015
Doesn't look like anyone commented on my previous blog, about the offline versions of The Daily Unlife. Ah well. There wasn't too much in there to discuss.
Anyway. Making those epubs purely by looking at the internals of some other generated epubs was quite a learning experience, and since I knew fimfic has an API for downloading story information, and have done an interesting project that involved scraping web pages as final project when graduating, I decided to combine all that knowledge (and a nifty web browser emulating class from that aforementioned project) to make my very own Fimfiction Epub creator!
Now, I've since been informed that the Calibre program has a perfectly good plugin that already does that, and in fact one of the epubs I studied (made by volrathxp) was probably made by that plugin. But, whatever! It was a fun project, and I added some extras that might be fun to play around with. You can edit the story's description, chapter contents and author's notes (in raw HTML, mind you), allowing you to clean up irrelevant things like "New chapter released!" notes; you can change the cover image url, if you happen to know of better art for a fic; and you can even add and remove whole chapters, to cut out the kind of irrelevant chapters that have since become against the rules, and to add relevant side-writings (like the Night Eight chapter for TDU) into the main fic's Epub file.
One nifty little extra I added in that aspect is the ability to import chapters by ID. Sure, you'll need to open the program a second time and load that other fic in it to get that chapter ID anyway (it can be copied as text from the "Review and edit chapters" screen), but this way you just need to copy an ID rather than the chapter's title, text and author's notes.
Ah yes. Chapter IDs.
One of the interesting things about the Fimfic API is that it gives you the internal IDs of the chapters. Some of you people, especially authors, have probably noticed that unpublished chapters are opened with a "fimfiction.net/chapter/*chapterid*" url instead of the normal "fimfiction.net/story/*storyid*/*chapterindex*" one. Now, since I made an index page on Google Docs for The Daily Unlife, in which I put the actual release dates of the chapters (whereas fimfic itself switched from "last updated" to "created" dates at some point, making the dates on older chapters meaningless), I decided to do a little experiment.
So, for no reason whatsoever except my personal curiosity, here's a graph of how many chapters were added on fimfic over the time span it took to create all of The Daily Unlife. The zero-point for the chapters is put on the opening date of Fimfiction itself, in July 2011.
I can't help but wonder what happened between the 17th and the 25th of April in 2013. That's between me publishing Day Eight and Night Eight, by the way. For some bizarre reason, in the middle of the drought between seasons, nearly 90k chapters were added there in barely more than a week, increasing the amount of used chapter IDs on the site by 50% of everything it had up to that point.
...maybe it was a spam attack? 
Anyway, so, yeah, that's what I've been up to the past week. If anyone can get some use out of that epub converter thingy, be my guest.
And if anyone feels compelled to look at the C# source code (I got some nifty functions and classes in there for progress bars, JSON parsing, ini file reading, numeric up/down controls and html cleaning), it's on the site too, in the "dev" folder.
Maybe fimfic became the the place to go to that week, resulting in a lot of google docs getting imported over. Or maybe that was the week the importer went online.
3311584
Doesn't look like it. I found a blog from February 2012 complaining about the GDocs import, so it seems that thing has been there pretty much from the start.
How does it compare to the site's built-in converter, though? I know some formatting gets lost with the native one, and it also doesn't include embedded images, and sometimes the cover art isn't exported either, I think? How good is this one with that? Also, how large are files? The built-in seems to convert to about 2kb/1K words.
Note: I read more mlp fiction as .epub files on my phone than I do on the site itself. I have something like a gigabyte of fiction on there.
3311996
The size depends a lot on how many images are inside it. The Daily Unlife is about 4 to 6 mb, depending on whether I enable image optimizing (meaning the small version of the cover is used, and any non-emoticon embedded images are resaved as jpeg to keep the file size down), but TDU has a cover image for every single chapter.
In pure text, without any images, the original Fallout: Equestria, including chapter 20.5, is about 1.4 megabytes. This is about the same for my exporter and fimfic's own.
As for how my exporter compares to fimfic's in other areas... there's really no comparison. Fimfic's own epubs are stripped of all formatting, all images, all author's notes, and don't even contain the title picture or synopsis. On mine, you can customize the title picture if you would really want that, and you can enable options to generate synopsis and TOC pages, and, if the fic itself doesn't put the chapter titles in its story text, you can fix that by checking the option to add title headers to all chapters.
The epub conversion screen:
nyerguds.arsaneus-design.com/project_stuff/2015/FimFic2Epub/fim2epub_2_epub_opts.png
Oh. That reminds me. I don't think I've built in support for blockquotes yet
3312041
Yeah, this looks very nice.
Think you could turn this into a browser plugin?
3312070
That's... not my area of expertise, at all. As I said, though, there's a plugin for calibre, named "FanFicFare", that does similar things, though with a lot less options.
I'm beginning to really like using epubs + Calibre as reading method. It remembers where you were when you closed the program, and this feature is even portable across devices (a usb flash drive in my case), since Calibre saves its bookmarks inside the epub file
I just wish I'd realized sooner you can launch epubs with Calibre directly in the reader itself (ebook-viewer.exe) without bothering with the library; I despise systems that want to sort my stuff for me.
3312140
I'm using Moon+ Reader on an Android phone myself. Convenient since it has a nice, customizable reading interface, a night-mode for a darker screen, and low battery consumption.
3312664
Most of my reading time is the 2+ hours per day I spend commuting by train. Since I work on a laptop at work anyway, I decided to simply start taking it home with me every day, so I can use it on the train to read. (And write. And program.)
E-Readers are probably not my thing anyway; I want to be able to note down my reactions while reading, to post them in the story comments afterwards. I always got my chapter + notepad open
(on a related note, I just fixed a bunch of bugs in the HTML cleanup of the converter, and fixed a crash that happened when a story contained 0 images. Yay, v1.3!
)
3311996
Hm. I experimented a bit with jpeg compression, and with the jpeg quality set to 30% (which is about the minimum I can stand to look at) I can actually compress TDU to just below one megabyte (compared to 7.2 mb when pure uncompressed), so that's pretty nifty
I added that quality as customizable setting in the tool (v1.5) now.
3326157
Neat.