• Member Since 11th Apr, 2012
  • offline last seen 11 hours ago

Bad Horse


Beneath the microscope, you contain galaxies.

More Blog Posts758

Jan
21st
2019

We are the dark matter · 5:40am Jan 21st, 2019

"The amount of creativity hidden in something like fan-fiction is enormous. This is like the dark matter of play."

- Simon DeDeo, "The Data Science of Play":

I just wanted to post that quote.  But in case you're wondering whether to watch that video, I'll tell you a bit about it:

DeDeo describes art and science as play, and measures large datasets of publications to learn things about creative communities.  Like his claim that poems are becoming less innovative:

He briefly mentions fan-fiction in that talk.  In a poster, he gives more-detailed results of a study of fan-fiction, which he says shows that fans are conservative and penalize novelty:


I don't dispute his data; but if it's supposed to mean that fan-fiction readers or communities are unusually intolerant of novelty, then it's wrong in several ways.

DeDeo measured "novelty" by counting the number of times each story uses each word, then computing the Kullback-Leibler (KL) divergence of its word frequencies from the average across all stories.  This KL-divergence score measures the "surprise" present in a story as the number of bits of information that it would take to specify its distribution of words, given the average distribution of words. Some problems with that:

  • KL-divergence counts misspellings and bad writing as "novel". There are more ways to write badly than to write well.  The problem of distinguishing novelty from noise is very difficult, but similar to the problem of distinguishing complexity from randomness, so perhaps approaches using complexity measures could be useful.

    • In DeDeo's defense, modernist literary critics have given disordered, ungrammatical, and even nonsensical writing credit for being novel, from Gertrude Stein in the 1920s to José Saramago today.  So "bad", or at least "random", is part of what they mean by "novel".
  • In my own studies of fimfiction, I've found that most ratings are made by people who aren't members of the community (they neither follow, nor are followed by, anyone who is part of the community). Their ratings therefore say nothing about popularity within the community.
  • The influence of community members can be estimated as the number of blogs they've posted, times the number of followers they have. I found that half of the influence belongs to the 0.12% of users with more than 805 followers, whom we might call the "elite". Their opinions dictate what community members perceive as the opinion of "the community".
  • I also found that fimfiction isn't a single community, but a set of about 10 mostly non-interacting communities, organized around different tastes in fiction which correspond closely to genres.  "My Little Pony" is not a genre; MLP fan-fiction is a collection of MLP stories of all different genres.  The stories of a small community or genre would appear more novel by this measure, and would also get fewer likes, than those of a large community or genre.  That alone would account for his results.

    • This might not apply as completely on ao3.org, which he studied, since they have no good mechanisms for forming communities or interacting with other users. But it will still be the case that stories in more-popular genres, such as romance and porn, will get more likes than stories in less-popular but more-esteemed genres.
  • You'd get the same results if you studied print fiction instead of fan-fiction. The most-popular genre, romance, is the least-novel; the most-novel genre (at least by a bag-of-words measure), literary fiction, is the least popular.
  • This popularity problem is due to the fact that DeDeo measured how many users gave a story kudos, while the question about the relationship between kudos and novelty is only interesting if we count what fraction of users who read the story gave it kudos. This is easily done by dividing kudos by views.
Report Bad Horse · 852 views · #data #fanfiction
Comments ( 33 )

Also, I call foul on categorizing "Marvel" and "The Avengers" as distinct fandoms. Similarwise (but less certain) for "Homestuck" and "MS Paint Adventures"; technically those are different, but I kinda doubt large numbers of people are writing Problem Sleuth fanfiction.

In my own studies of fimfiction, I've found that most ratings are made by people who aren't members of the community (they follow 0 or 1 people).

I don't doubt this, but would you be able to re-link any discussions you've had on this point?

KL-divergence on word distributions is an... odd way to measure novelty. For one, it's going to favour shorter stories (assign higher originality scores to them), since word distributions are going to approach some kind of equilibrium as the word count increases that's likely close to the average distribution, in addition to the misspelling things. It's also going to favour people who use a wide and archaic/esoteric vocabulary (for one example, someone who replaces every instance of "said" with a synonym is going to shoot up in divergence), which, coincidentally, tends to turn people off a story if someone is using very purple language.

There's other issues with using KL-divergence like this, assuming that they're assuming an i.i.d. distribution, which is a whole other can of worms that make this incredibly suspect.

5000266

It seems like it would favor writers indulging in dialect humor or modernist word games like Joyce. Finnegan's Wake would score way the hell up the chart, wouldn't it?

If you go to a dinner party and someone says, "Okay, the topic of conversation is X and we're going to solve this by the end of the dinner party," it's a disastrous dinner party.

Once a week I hold a dinner party for a half-dozen of my closest friends, and on many of those occasions, there will indeed be a set goal. It's often something like, "There's a centaur warlord holding the king's daughter hostage, and you guys need to infiltrate his stronghold to rescue her." or "The evil witch queen is spreading endless winter across the world, and you guys need to free her mother who can stop her." or even "You guys are all mice, and the nest is being attacked by an owl." I consider these to be very successful dinner parties.

This is, of course, all about play, and he even mentions D&D a few paragraphs later. But it seems like almost every social situation in my life is framed by some purpose or driving situation. "We're here to learn how to dance," "We're here to watch old sci-fi together," "We're here to play board games," etc. Almost none of them are formless.

They write short stories.

:rainbowlaugh:

5000266 5000270
I agree that the measure of novelty he uses seems pretty strange, and largely orthogonal to actual novelty of the story. This seems like a statistician's fallacy, in which they feel that every property must be exactly measurable, and having found a method of doing that they don't question whether it's really appropriate.

It's true that most fanfiction doesn't favour linguistic originality, since it explicitly borrows names, settings, personalities and modes of speech from the source material. And there's a very successful seam of stories that don't aim to innovate explicitly. A touching slice of life story may not contain any unusual words at all, and may not seem novel to a reader, but that doesn't mean the author has in any sense chickened out of innovation, nor does it mean the story isn't novel on a deeper level.

I have to wonder how the results would change if you took Sturgeon's Law into account, and only sampled the top 10% or 5% of fanfictions (assuming you could define a measure of that) rather than the whole set. When comparing to classic poetry, that filter is mostly pre-applied, since so much of the doggerel and drivel of past centuries hasn't survived to today to be analysed.

Measuring word divergence and noting that higher divergence from the average correlates generally to lower approval by readers is exactly what I'd expect to find, but also not an observation that I think is particularly useful. A raw measure of word divergence is an objective measure, ie., it's a measure of style, because style is objectively observable, but reader enjoyment of a story is determined by their interpretation of the substance of a story, that is, it's largely subjective.

I'm not sure why he's under the impression that you can measure what's most important about a subjective aspect of fanfics with objective quantification as if it was a direct causal relationship.

I mean, the correlation is there, sure, but it strikes me as incredibly superficial, on the order of noting that increased ice cream sales correlate to higher murder rates, and presenting superficial correlations in a context that implies "this is legit science" (you know, like a publication purporting to analyze large datasets) feels deceptive and likely to end up giving naive readers dangerous suggestions (whether intentionally or not). The incredibly depressing conclusion I'd draw here, if I was a noob writer who just wanted a lot of views and likes because I've been drawn into the Skinner-box system that is author-reader interaction on many sites, is that I should strive to do my best at aping already popular authors in order to "hit the checkboxes" of style that the objective measurement here is correlating to maximized reader approval.

This does not help me write better stories or grow as an author. It teaches me to be machine churning out output with just enough variation of style to be considered "new" stories, and to be highly conservative of output patterns that have worked in the past.

I also found that fimfiction isn't a single community, but a set of about 10 mostly non-interacting communities, organized around different tastes in fiction. The stories of a small community would appear more novel with his measure, and would get fewer likes, than those by a large community. That alone would account for his results.

This certainly sounds right, but I wouldn't mind learning more. Do you have a blog post going into this?

I agree with you, mostly because this is short enough for me to read.

Wanderer D
Moderator

At some point I wonder if you'll talk about those 10 mostly non-interacting communities! LEarning more about this fandom is something I love XD

PresentPerfect
Author Interviewer

I would say the lack of kudos is due to fandoms declining over time. I mean, just look at any of Pascoite's analyses of view counts garnered from EQD posts over the years. You get into something, everyone gets into it, most of them move on when the next big thing comes around.

For DeDeo's conclusion that poetry is becoming less innovative over time, does he correct for the fact that the "past" part of this "similarity to past" metric increases over time? For example, if you were to randomize word usage across all poems, would you see the same trend of decreasing innovation? After all, it's easier to craft something new and novel if the existing corpus is smaller.

While I agree with you and other people pointing out that this is a wonky metric for novelty, I also think that measuring the popularity of novelty is a misleading metric for how creative a group is, even if you had a really great metric for novelty. I've said before, experimental writing is an experiment, and experiments don't always confirm the hypothesis of "this would make a good story." That doesn't mean that fandom doesn't try them at rates higher than other creative communities, but the higher try/fail rate (as compaired more traditional stories) means that even when they're very successful they're not going to be as widly read or liked. I don't think this counts as penalizing them, or if it does I think any community high on artistic risk taking is going to either show similar behavior or it has to be blowing smoke up one another's tails.

His methods remind me of the Cooper-Harper Rating Scale:

upload.wikimedia.org/wikipedia/commons/thumb/2/28/Cooperharper_full.pdf/page1-800px-Cooperharper_full.pdf.jpg

It's supposed to be all objective and stuff ("But look guys--FLOWCHART!") but look at those decision blocks: "satisfactory," "adequate," even "controllable" might not mean the same things to a top-tier test pilot and some guy who flies for DHL.

5000334

I've said before, experimental writing is an experiment, and experiments don't always confirm the hypothesis of "this would make a good story."

vignette.wikia.nocookie.net/alvinfanon/images/7/7d/Clyde_Crashcup_Movie.jpg/revision/latest?cb=20120121111521
"Perhaps if I added a SECOND red-and-black alicorn and made them kiss..."

5000270

Or The Water of the Wondrous Isles ("Whilome, as tells the tale, was a walled cheaping town hight Utterhay.")

A wise pony once said, "There's lies, damned lies, and statistics."

While I would tend to agree with nearly all of the objections to the article, I can't necessarily disagree with the conclusion. There is a lot of good writing on the site; but there is a whole lot more cookie-cutter, formulaic, cliche'd writing; and that is what tends to get the majority of upvotes.

It's understandable to a degree, since humans are hard-wired to prefer the familiar to the novel, as an evolutionary survival mechanism, but there's also a good deal of downvoting of some of the more interesting work, simply because it challenges the reader. One of the most popular stories on the site consists of the same short story written over and over and over with only minor variations, with a rudimentary overarching framing plot that gets only token development over the course of the first volume, and wasn't even remotely tied into the subplots.


5000291

Well, that's the big problem here; the overwhelming majority of authors on the site aren't writing because they have something to say, because they have some story deep inside them that they want to get out; they're writing because they want the validation, they want the rush of getting kudos, they want to be acknowledged and appreciated. In other words, they're writing primarily to be popular. Hence the endless stream of lazy crossovers, aping of popular authors, slapping pony paint jobs on stories plagiarized from anime and video games, and writing to conventions with built-in appreciative audiences (eg. Displaced). The type that spend more effort trying to game the system in order to hit the Featured box, than they do writing good stories. And it's a fairly effective practice, given how much of the Featured box tends to be clop or brainless crossovers at any particular moment (my own observations are that the Featured box averages about 50% clop). It doesn't help that the majority of authors on the site are typically teens or early adults, and are very heavily emotionally invested in peer approval.

I actually had one such kid try to give me advice on how to be more popular, right down to dictating how I should change my author page and avatar, because their current incarnation might put readers off. No matter how many times I tried to explain that I wasn't interested in that sort of popularity, and was focused on the writing instead, it just didn't sink in (it probably didn't help that I used a lot of multisyllabic words that he likely didn't understand). But that's just an artifact of the larger focus on popularity and peer validation.

wait, conservative compared to what? Peer-review papers? the theoretical Wundt-Berlyne Curve?
is the binary "kudos" system on AO3 really that useful as data? you can't even downvote on the site, and everyone doesn't read everything.

I kinda wonder if the most popular fanfics within each fandom then become the ones everyone else imitates, pushing them into the "average" fiction mold, closer to the 0 novelty mark than they initially would've been. With all the trends that go on, it sure feels like it sometimes.

5000241 I don't think I've posted anything about this. Here's a graph:
lh3.googleusercontent.com/9yLNR4r0dIu7ACHWeOKAO8Qqzg2i1dJC8oOdNX3AsXoFc5QMwg4pJdQa8PZm_-IFFYvEnaQ1hhxCYqUQcTN9A2235z4UluMPbNzklOtKnIGnCKaXqf-05LaSdRhVmJHLi9AHNMpk
The solid line shows the number of users (written on the line) who follow F people, where F is the label on the x-axis. The distance along the x-axis is the logarithm of the label on the x-axis, because otherwise the graph would be really big, and all the detail would be scrunched up along the left edge.

The first point on that line, 106422, is the number of fimfiction's users who follow zero people.  The second, 18309, is the number who follow 1 person. I collected this data in, I think, early 2016; the total # users at the time was about 200,000.  So most users didn't follow anyone. This is not apparent on the site, because most of these users are nearly invisible: they generally don't leave comments, don't follow anyone, and aren't followed by anyone. I was only able to get their total number because I exhaustively found all the followers of all users; the number of users missing from the total given on the stats page was thus the number of users who were completely invisible, in addition to the users who followed no one whom I'd found thru other means. At the time, stories had a stat saying how many times they'd been favorited, so I was able to get the number of favorites made by these invisible users in the same way.

The dashed line is the number of favorites made by people who followed that many users. I didn't print numbers on that line, but you can see from its shape that the same thing happens: about half of the favorites were made by users who followed zero other users.

5000325 He might have made that chart with the same method he used for studying scientific articles, which was to compare each year's texts to those of the previous year. But maybe not! I only saw the talk; there's probably a paper.

5000574

wait, conservative compared to what? Peer-review papers? the theoretical Wundt-Berlyne Curve?

Yes, and also to poems from a poetry journal.

5000309 A blog post about the communities, or about the reasoning in the bolded sentence? If the first, see my reply 5000651

5000334 You never told me if you liked your Hearths-Warming present! :fluttercry:

5000683
You are a bad horse, Bad Horse. Only you could find an evil Hearth's Warming present. :ajsmug:

5000627
Wow, that's a much larger effect than I would have originally assumed. Was this data collected post-bookshelves? I wonder if there's any significant effect from people renaming their Favorites bookshelf, which I imagine would be more likely to happen for dedicated readers (though indeed, the proportion is probably negligible).

Also, given that the median time offline is so large, do you think you'd observe the same effect if you only looked at stories published in, say, 2018? I think it's possible this trend could have been an artifact of Fimfiction in the earlier days/during the fandom's height.

5000836 Good question. I'm pretty sure it was pre-bookshelves, because I wouldn't have been able to attribute missing favorites to invisible users. Do you know when bookshelfs began?

Renaming Favorites, and users with multiple favorites-type shelves, is a big problem for studying the site now.

I think it's possible this trend could have been an artifact of Fimfiction in the earlier days/during the fandom's height.

The graph shows that the number of people a user follows has no effect on how many favorites they mark, because the graph of # of favorites made in a bin matches the graph of how many users are in that bin quite closely, I can't think of any reason why that would change with the size of the fandom.

I do find it strange that following more people doesn't correlate with reading more stories. I wish the site still reported words read per user. I think we can assume that most people who don't follow anybody just pick stories off the front page.

5000865

I think we can assume that most people who don't follow anybody just pick stories off the front page.

I'm not sure, but I seem to remember back in the day there being a number of people who would only read things posted to EqD, and just came here from the links. I also wonder about groups, I think there were some people in the AppleDash group who didn't follow people but watched stories posted to the group instead. Other genre groups might have similar watchers.

5000865
It looks like bookshelves were existing at least by December 2014 (possibly earlier, sometime as early as September 2014--also I can't believe it's been that long! We've had bookshelves for longer than we haven't.

I think I described it poorly. What I meant was is that I imagine that as the site's active user population (including the silent readers) has gone down, the proportion of favorites coming from "the community" would increase. Maybe that's completely off base, though?

Perhaps the people who tend to follow large numbers of users are more likely to be engaged in groups and blog posts, cutting into reading time?

5000911 Looking at my databases, I see I have databases from September 2012 - October 2015. I'll have to reread my notes to make sure I didn't mess up the results, but I think reports on the number of favorites a story has went away at about the time bookshelves appeared.

What I meant was is that I imagine that as the site's active user population (including the silent readers) has gone down, the proportion of favorites coming from "the community" would increase. Maybe that's completely off base, though?

That seems likely, yes, and thanks for pointing that out. The silent readers have a mean time offsite of something like a year, and I was thinking they were people who rarely visited the site--but then they couldn't have made that many favorites. I ought to redraw that graph looking at a smaller time interval, but I can't--there's no way to find out when a favorite was made.

5000911 Okay, those numbers come from a database thru Oct 12, 2015, but it counts only favorites found on user accounts, as there was no way to find total number of favorites on fimfiction by then. So it significantly underestimates the number of favorites made by users who follow no one and are followed by no one, because they could be found only if they'd made a favorite which was picked up by an earlier version of the scraper, before stories stopped listing the users who had favorited them. Any idea when that was?

This is from a paper I'm writing:

51.8% of the favorites were made by the 12.4% of users who had been online within the past day when my web-crawler loaded their user page. But 80.3% were made by people who watch zero to three users. 64.0% were made by people who watch no users at all—and this is an under-estimate, as my scraper often can’t find the favorites of users who watch no one. So the favorites are made mostly by people who are online frequently (note the datapoints in Figure 3 are just medians), yet also mostly by people who don’t follow other users. This means that most of the favorites, which are used to decide which stories will be featured on the front page, are made by people Abercrombie & Longhurst would call “fans,” passive participants with no social connection to fandom.

11,115 of the 29,734 writers in my DB—just over 51%—watch less than four people, and 25.3% watch none. They received 25.4% and 11.9% of the story views, respectively. FF has a large number of writers who do not engage with the community outside of their own stories. (Unlike most other fandoms, there are no popular large-scale meeting places on other websites.)

5001361
I can confirm that they existed at least until December 2012 (though it says it only listed the first 100? As a reader I almost never used this feature when it existed, so I can't remember the details of it). After that they switched to whatever that new system was and I can't tell from cached pages. I think it was still working by July 2013 (the button looks the same). The Site Statistics page used to show the Top 10 most favorited stories, so that might be a proxy for how long that functionality lasted; it was working until one of the December updates--I vaguely remember it actually being broken, and that the missing section isn't a problem with the archive.org cache. In any case, that section ended completely with the Library/Bookshelf Update.
So sometime between December 2012 and October 2014, with a few possible transition times listed.

That paper looks really interesting! I'm curious exactly how that your scraper worked prior to the bookshelf update: was it just finding users via who watched who (and that's why it couldn't find the Favorites of users who don't want anyone)? I would think that you could at the time still find the Favorites of such users if you just went up the user index numbers, though I don't know how feasible or resource intensive that is.

5001625 I'd have to check the code to be sure, but my recollection is that it found users through references. Mostly by exhaustively recording who watched whom, but also by getting names from favorites (back when that was possible) and from the names of story authors. Stepping thru all the numbers seems the obvious thing to do, and either I was very stupid, or there was some reason for not doing it. I suspect it was impractical because the way that I requested pages would time out if there was no such page, and IIRC there are about 100,000 deleted user accounts that you'd have to step thru, meaning about 10 days of waiting for timeouts. But even that isn't much, spread out over a year. So... I guess I was just stupid.

It's probably because it was legacy code. Originally, I just wanted to collect information about stories, not about users or the watch-network, so the scraper logic was organized around stepping thru stories.

Login or register to comment