Text Processing is Magic

Since even Bad Horse says I’ve out-nerded him, I might as well explain how I did it in great, excruciating detail. Because I’ve finally published the results as fit for human/pony consumption.

It’s a programmer’s approach to making your writing comfortable. As such, it’s probably not for everyone, but if you’re the kind of person who spends their life near wikis, code, mathematics or scientific writing, it’s very much worth trying, and then there’s no reason not to apply it to horse words too.

1. I’m not sure exactly, but I think it’s been a decade. Pretty much since the first PHP implementation turned up, I’ve been using it for my Wordpress blog… Ah, those were the days.

For the longest time¹ I’ve been using Markdown as the primary source format for everything written – blog posts, documentation, papers, letters, fiction, etc, etc. At the heart, it’s merely a convention derived from how people used to write email back before HTML email became available, and remains easy to remember and easy to read.

You mark up text as _italic_, **bold**, `monospace`
using symbols in easy finger reach. Paragraphs are separated
by empty lines.

You can also do stuff like lists:

  * apples
  * oranges[^1]
  * pears

[^1]: This is a numbered footnote.

That sort of thing. Look it up and RTFM. The advantages of using plain text as the source are pretty numerous:

• There are tools to produce just about any kind of output you want from Markdown source easily, and usually, you can use the same source to produce many different kinds of output.

• 2. I have one! Muahahaha. No, I’m not using it.
  3. These days, you can edit Markdown in your browser using any of a number of Chrome applications and keep it in Google Drive, so even Chromebooks are not an exception.

  Not only it remains human-readable, it remains editable on any platform, from mobile to antique VT-52 terminals.² Every platform under the sun has a plain text editor, those that don’t aren’t really computers.³ In all likelihood, the files will continue to be readable when they outlive you.

• It plays nicely with source control, and I like source control. In particular, using source control makes it easy for other people to see what you changed. I.e. it’s very useful for an editor or a proofreader.

When Pandoc came about, the task of writing complex documents became even easier, because with that swiss army knife of text-based markup formats, you can go from a plain text manuscript to a print-ready PDF in ten seconds flat if you know what you’re doing, and in a few hours if you don’t.

When I started writing horse words, the first thing I noticed was Fimfiction’s quirky and bizarre bbcode syntax, so naturally, I went fishing around for a tool to save me from this horror – I use a lot of italics. Eventually, I found a pandoc output formatter script, originally authored by HeirOfNorton.

4. Since Fimfiction does not let me do proper link anchors, I elected to stick the footnote bodies below the paragraph they are referred to in for readability. Unless I find an XSS vulnerability, that’s the way it stays. But if I do, I’m probably not using it, since that would surely not be received well.

Pandoc is extensible – it parses your source document into an abstract syntax tree, and then permits you to feed this tree through your own script, which will render the elements into whatever format you require. HeirOfNorton’s script has not been maintained for a few years, and both FimFiction’s bbcode syntax and Pandoc itself have changed during that time, but have restored the thing and expanded it significantly, mostly in the attempt to produce nice-looking, readable footnotes⁴ – in particular, it now uses unicode characters to imitate superscript.

You can get the results here: https://github.com/methanoliver/pandoc-fimfic

Inside you will find documentation detailing the use, quirks, as well as a complete reference for Fimfiction bbcode, because I’ve had to extensively investigate how Fimfiction’s bbcode rendering works to get the results I wanted.

I use it for everything here that is longer than a quick oneliner, from stories to blog posts, and often, even comments. This blog post too, of course. Nothing beats a footnote. :)

Addendum for super-nerds

The recommended way (recommended by me, at least) to use it is to write a Makefile, keep your entire collection of horse-writing in a git repository, and acquire a decent plaintext editor with support for Markdown, capable of calling a build system – I use Atom these days, but back on Windows I used Notepad++.

This lets you compile all the source files using the same configuration with just a keypress, or even, on every save, and gives your writing a unified, easy to change look. Kind of like so:

FIMDIR = fimfiction

MD_FILES := $(wildcard *.md)
BBCODE_FILES := $(addprefix $(FIMDIR)/,$(notdir $(MD_FILES:.md=.bbcode)))

FIMFIC_CONFIG = data/fimfiction-meta.yaml

PANDOC_STRAIGHT=pandoc -f markdown-raw_html+autolink_bare_uris
PANDOC=$(PANDOC_STRAIGHT) -S
FIMFIC=-t data/fimfic.lua $(FIMFIC_CONFIG)

$(FIMDIR)/%.bbcode : %.md $(FIMFIC_CONFIG) | $(FIMDIR)
    $(PANDOC) $(FIMFIC) -o $@ $^

$(FIMDIR):
    @mkdir -p $@

all: $(BBCODE_FILES)

You can also use the same makefile to produce PDF or EPUB versions. GNU Make might not be much of an option when you’re using Windows, but .bat files are perfectly adequate when you’re not dealing with a hundred-chapters-expected monstrosity like I am.

Comments

[GhostOfHeraclitus]

Your nerdiness is a thing of legend. I tried writing this same thing (pre-Pandoc) for the same reason you did, but got bogged down and gave up on it eventually due to lack of time. I am so very happy you wrote this since I can finally use Markdown to write pony things.

My hat's off to you, sir.

Also: I don't know if you know about it, but I suspect you'd really like Penflip

Also, for your amusement, I give you what I found as my test document of Penflip. I completely forgot I wrote this, why I wrote it, or what exotic blend of recreational chemical I was on to create it. It is presented here without comment and without adequate apology.

The Pangram Poser

The quick brown fox jumps over the lazy dog.

This does not please him.

He leaps up into the air, all fury, teeth and claws. He tries a snarl and a blood-curdling scream all at once, and it comes out as a sort of stridulating yowl, but he doesn't care. He's too angry to care.

"What the hell is your problem, Fox? Huh? Can't you let a soul rest for a pawful of seconds," he screams, treating each word as a separate wound to be torn from the air. Oh, if only the little brown bastad would just say something. Anything. One bit of lip. C'mon. Dog had been waiting for a moment like this.

"Resting? You? When have you ever done something? Mm? I mean your first name is Lazy, for heaven's sake."

The dog takes a swing. He has been rehearsing this moment for ages. Fox would do something, something bastardly as he always does, and he'd say something cool and menacing—okay, "what's your problem" is a bit weak but he's sure he was looking cool when he said it—and then the little bastard would give him lip and he'd smack 'im right across that smug muzzle of his. Right across. He'd stop there, of course. He's no thug. Just once punch, enough to make his teeth rattle a bit. Just to show who's the boss. Who's got seniority here.

Except things aren't going like they are supposed to. Punch's not connecting. The damned fox ducks, weaves, slips and the dog can't even see him anymore. He stands stock still, one paw held straight out, with the hurt and vacant expression of someone utterly betrayed by an inner fantasy. From behind he can hear a voice—its usual measure of smugness doubled and then doubled again—drawl out familiar words.

"And my first name's Quick. Kinda makes you think, don't it?"

The dog flings himself back and slams the fox—whom his parents had wisely named 'Quick' and not 'Observant'—into a filing cabinet. The fox rolls away, grabs a stack of files and throws them straight at the dog's face. The dog shrugs them off, and leaps again, going right for the throat this time.

"What in the name of Serif and all his angels are you two doing?"

The dog manages of look of cherubic innoncence as he sails through the air and holds the look right up to the point when he crashes into the fox at balistic velocitiy. They both scramble to their feet struggling with all their might to project the look of someone who was just doing some extra-vigorous calisthenics. The fox spoke first.

"Captain Shrdlu! What a—ah—what an unexpected pleasure! We were just, just—uh—"

"—I was teachin' the kid, boss" the dog intejects, "Showing him this killer hand-to-hand move, right."

Captain Shrdlu looks... well. Not unimpressed. He's normally unimpressed. What he was doing now was several notches below 'unimpressed' bordering on 'underwhelmed.'

"I didn't know that you were a hand-to-hand expert, Detective," he says.

"Well, sir, you know, you, uh, you pick stuff up on the streets."

"No, I don't know. That's what a someone dealing with litter does¹. You two want to tackle that problem next?"

﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏﹏
1. Captain Etaoin Shrdlu was one of those people with enormous pride in the sense of humor they do not, in fact, possess.

"Nossir."

"Nossir."

"Glad to hear it. Now quit with the foreplay on city time and get your sorry asses to your old haunt at Pangram Station. Somebody fobbed the qursh pyx."

[Oliver]

3867559

Also: I don't know if you know about it, but I suspect you'd really like Penflip

I didn't. It looks very promising indeed.

The Pangram Poser

(spends a few minutes laughing)

This reminded me of a joke wherein Adam Jensen (of Deus Ex Human Revolution fame) walks into the lobby of Sarif Industries and asks the receptionist:

"Is the boss in?"
"No, mister Sarif has left for the conference."
"Would it be correct to say the building is currently sans Sarif?..."

[GhostOfHeraclitus]

3867579
Am glad you like the little tidbit. There's a surprising number of typography and language jokes in that little snippet. My personal favorite is the Captain being named Etaoin Shrdlu. :)

Oh, that reminds me! In the chats in that game, Sarif's handle is Franticpony (I think) and I made a joke about it: I had a reporter be named Sans Serif and had Spinning Top describe him as 'a frenetic pony.'

Nobody got it.

Still worth it.

Oh, and I'm about to put up a blog post. I apologize for its contents in advance.

[Oliver]

3867586

My personal favorite is the Captain being named Etaoin Shrdlu. :)

I'll admit I only understood this one because of SHRDLU. :)

Nobody got it.

Well, now I get it... :)

Oh, and I'm about to put up a blog post. I apologize for its contents in advance.

(looks for a bomb shelter)

[GhostOfHeraclitus]

3867593
Bomb shelters will not save you from this.

[Oliver]

3867664

(wraps himself in a white sheet and crawls off towards the cemetery)

[PoweredByTea]

Markdown for fimfiction? Ohmygoshohmygoshohmygosh. If I ever stopped being lazy and actually wrote some fimfiction, this would be the best thing ever! Thank you for this guide.

3867579
3867586

On Penflip, I was briefly very excited about it, until I lost work with it. Apparently, between commits, your work is only being persisted in browser local storage, which isn't as persistent as the penflip guy seemed to think. Also, you can't share work between computers. Now, it could be fixed by now. I did send a very angry email and got a response. But I wouldn't trust anything to penflip until you can verify if it has been fixed.

[SIGAWESOME]

Whoah

That is pretty neat.

Two Three things come to mind:

1. When the dilly-dallying fk did [ code] tags become available?!
2. What is FontAwesome?
3.

The recommended way (recommended by me, at least) to use it is to write a Makefile, keep your entire collection of horse-writing in a git repository,
<...>
This lets you compile all the source files using the same configuration with just a keypress

If I see mention of Bison/Lex + FimFic I will ask Ghost to stage in intervention..

[Oliver]

3871498

1. No idea, just tried it one day and it worked.
2. An icon kit for websites.
3. Nah, but I might mention a Pandoc filter, too... :)

[SIGAWESOME]

3871579

I think my brain had an abrupt stutter-step jolt over the idea of "patching" and "compiling" horse words...

jessie@xyzzy> ~ $ patch numbered < editor_edits.patch jessie@xyzzy> ~ $ ./configure jessie@xyzzy> ~ $ make jessie@xyzzy> ~ $ make fimfic

(maybe "make install" fires up cURL and publishes your story to FimFic?)

[Oliver]

3873439

I think my brain had an abrupt stutter-step jolt over the idea of "patching" and "compiling" horse words...

It's not uncommon to use those words when you deal with LaTeX and friends. Or static website generators, or PDF, or other processes where the resulting artifact is a "document" or "book" but doesn't look much like what you have to start with... :)

(maybe "make install" fires up cURL and publishes your story to FimFic?)

I wish. That would save me some clumsy copy-pasting, but alas, FimFiction has no published API. I'm not really up to picking it apart deep enough to do that, only to have knighty change everything on me a month later.

[Cursed Quill]

*bookmarks for later use*
>>aporia/7728225

emacs

<blink><marquee>Boo, hiss, assertions of vi superiority, holywar, etc. </blink></marquee> </joke>
I really should start automating, with how much I comment…been so long since I used a makefile…

[Oliver]

4305904

<blink><marquee>Boo, hiss, assertions of vi superiority, holywar, etc. </blink></marquee> </joke>

Atom? :) I found it comparatively harder to program than Emacs, though, but there’s a similar edit-server solution out for Atom too.

Oh, and there’s also Spacemacs if you’re into vi superiority, and there should be something that does the same for Atom too by now…