• Member Since 24th Sep, 2015
  • offline last seen Monday

Oliver


Let R = { x | x ∉ x }, then R ∈ R ⟺ R ∉ R... or is it?

More Blog Posts343

Mar
24th
2019

Statistics #5.1: Modernized Friendliest Fleet · 12:25pm Mar 24th, 2019

Remember my post about the Friendliest Fleet and shipping mathematics? I got news and further adventures in statomancy and logistics!

I noticed that Derpibooru has a public API I could use. While this API does not permit me to see the list of tags – what a shame – which prevents me from improving my initial list of known ships, it does allow me to get a list of images tagged with any given tag, which includes the complete image metadata, like the first upload date, and even the number of votes.

This permits me to do a time-weighted rating of ships, and significantly changes the makeup of the Friendliest Fleet, as the old favorites which are no longer popular fade out to give way to new ships. Code in the Github repository has been updated accordingly. To use it, you will require a Derpibooru account, and you will need to set filters set to “Everything” at least for the duration of this exercise, because you can’t set those through the API, and it hardly makes sense to take measurements if you only count the family-friendly images.

I had to keep the same “a ship only counts if it has 50 images to its name or more” rule, if only because when originally searching for the shipping tags, I’ve been ignoring tags with fewer images, so I’m not up to opening 250+ tabs in the browser again, and because we’re still dealing with graphs, which means that computational requirements rise exponentially as the number of nodes increases.

After some thinking, I decided to keep it simple, and use the sum of upvotes for all images representing the ship, divided by the number of days since the earliest picture was uploaded, as the actual ship score. While Derpibooru also returns a Wilson score, which takes into account both upvotes and downvotes, I’m not sure it makes sense to simply sum those. I computed the data anyway, in case anyone wants to use it, but remembering how different is the upvoting behavior on FimFiction from the downvoting behavior, I just don’t feel confident using anything other than pure upvotes per day to rank ships.

1. Here’s the full table.

Here’s a list1 of the time weighted scores, sorted by ship score. The actual number effectively doesn’t mean anything now, it only serves to establish the relative popularity of each ship:

388.0000  - Applejack / Rarity
375.5129  - Rarity / Spike
309.8170  - Sunset Shimmer / Twilight Sparkle
299.4208  - Rainbow Dash / Twilight Sparkle
275.9751  - Starlight Glimmer / Trixie
259.6650  - Applejack / Rainbow Dash
246.9757  - Fluttershy / Rainbow Dash
237.7542  - Celestia / Luna
226.0587  - Shining Armor / Twilight Sparkle
225.4154  - Celestia / Twilight Sparkle
224.7636  - Octavia Melody / Vinyl Scratch
197.6050  - Cadance / Shining Armor
194.7726  - Sci-Twi / Sunset Shimmer
184.2382  - Trixie / Twilight Sparkle
163.0042  - Pinkie Pie / Rainbow Dash
159.5810  - Fizzlepop Berrytwist / Starlight Glimmer
153.8352  - Spike / Twilight Sparkle
149.2290  - Rainbow Dash / Soarin
145.3692  - Bon Bon / Lyra Heartstrings
138.4945  - Gallus / Sandbar
136.9473  - Luna / Twilight Sparkle
129.9943  - Big Mac / Fluttershy
126.2620  - Bright Mac / Pear Butter
125.2648  - Applejack / Big Mac
121.4946  - Starlight Glimmer / Sunburst
117.8292  - Rainbow Dash / Rarity
114.6887  - Applejack / Fluttershy
112.6269  - Fluttershy / Rarity
111.1935  - Rarity / Twilight Sparkle
108.7497  - Big Mac / Sugar Belle
108.2263  - Bow Hot Hoof / Windy Whistles
107.4118  - Ember / Spike
100.2986  - Big Mac / Marble Pie
98.2392   - Flash Sentry / Twilight Sparkle
92.9935   - Discord / Fluttershy
90.3448   - Pinkie Pie / Twilight Sparkle
89.0776   - Fluttershy / Pinkie Pie
87.0303   - Stellar Flare / Sunburst
81.1675   - Shining Armor / Twilight Velvet
75.7346   - Starlight Glimmer / Twilight Sparkle
74.5398   - Applejack / Twilight Sparkle
72.5400   - Quibble Pants / Rainbow Dash
65.2490   - Celestia / Discord
63.5546   - Adagio Dazzle / Sunset Shimmer
58.8391   - Rainbow Dash / Scootaloo
58.2997   - Spike / Starlight Glimmer
57.0361   - Moondancer / Twilight Sparkle
56.1135   - Aria Blaze / Sonata Dusk
52.7348   - Rarity / Sweetie Belle
51.5739   - Smolder / Spike
48.5941   - Big Mac / Cheerilee
48.1973   - Gallus / Silverstream
47.1660   - Apple Bloom / Big Mac
46.7939   - Chrysalis / Shining Armor
45.7285   - Maud Pie / Mudbriar
44.3850   - Pinkie Pie / Rarity
43.7854   - Derpy / Doctor Whooves
43.6233   - Flash Sentry / Sunset Shimmer
43.1202   - Applejack / Spike
42.8484   - Cheese Sandwich / Pinkie Pie
41.3652   - Spike / Sweetie Belle
41.2931   - Coco Pommel / Rarity
39.4036   - Maud Pie / Pinkie Pie
39.2780   - Starlight Glimmer / Sunset Shimmer
38.9387   - Maud Pie / Trixie
37.6746   - Applejack / Autumn Blaze
36.4062   - Fluttershy / Spike
35.3320   - Apple Bloom / Sweetie Belle
34.7694   - Cadance / Twilight Sparkle
34.5328   - A Royal Guard / Celestia
34.2302   - Ember / Thorax
33.5147   - Applejack / Pinkie Pie
33.1284   - Adagio Dazzle / Sonata Dusk
32.5644   - Diamond Tiara / Silver Spoon
32.5359   - Adagio Dazzle / Aria Blaze
32.0571   - Celestia / Chrysalis
31.6385   - Shining Armor / Spike
30.8636   - Pinkie Pie / Spike
30.4171   - Big Mac / Braeburn
30.1587   - Fluttershy / Sombra
29.9165   - Aloe / Lotus
29.8615   - Rainbow Dash / Spike
29.8207   - Big Mac / Rainbow Dash
29.4949   - Night Light / Twilight Sparkle
27.9660   - Soarin / Spitfire
27.8587   - Sonata Dusk / Sunset Shimmer
27.6468   - Big Mac / Rarity
27.5940   - Button Mash / Sweetie Belle
27.3320   - Twilight Sparkle / Twilight Velvet
27.3236   - Apple Bloom / Applejack
26.2002   - Rainbow Dash / Spitfire
26.1108   - Luna / Shining Armor
26.0366   - Braeburn / Soarin
25.2481   - Night Light / Twilight Velvet
24.9065   - Sunset Shimmer / Trixie
23.9805   - Big Mac / Pinkie Pie
23.8037   - Big Mac / Twilight Sparkle
22.9818   - Sci-Twi / Timber Spruce
22.8896   - Spike / Thorax
22.8302   - Cadance / Chrysalis
22.5533   - Big Mac / Sweetie Belle
22.2616   - Apple Bloom / Tender Taps
21.9696   - Cloud Chaser / Flitter
21.9632   - Chrysalis / Nightmare Moon
21.5166   - Carrot Cake / Cup Cake
21.4909   - Soarin / Thunderlane
21.3331   - Big Mac / Shining Armor
20.9379   - Celestia / Spike
20.7975   - Starlight Glimmer / Tempest Shadow
20.5550   - Gilda / Rainbow Dash
20.1806   - Fluttershy / Luna
19.8762   - Apple Bloom / Scootaloo
19.7824   - Applejack / Coloratura
19.6278   - Pinkie Pie / Skystar
19.2122   - Apple Bloom / Spike
19.0090   - Fluttershy / Zephyr Breeze
18.5431   - Rainbow Dash / Zephyr Breeze
17.9117   - Celestia / Shining Armor
17.7017   - Fluttershy / Tree Hugger
17.4727   - Applejack / Sunset Shimmer
16.7645   - Pinkie Pie / Sunset Shimmer
16.3635   - Discord / Luna
16.1527   - Cadance / Night Light
16.1314   - Luna / Pipsqueak
15.6983   - Apple Bloom / Babs Seed
15.0581   - Big Mac / Caramel
15.0038   - Rainbow Dash / Sweetie Belle
14.7900   - Hoo'far / Trixie
14.6822   - Cadance / Celestia
14.6547   - Big Mac / Scootaloo
14.6049   - Big Mac / Luna
14.6044   - Celestia / Sunset Shimmer
14.0154   - Celestia / Rarity
13.9761   - Cadance / Luna
13.8774   - Bulk Biceps / Fluttershy
13.8537   - Rainbow Dash / Sunset Shimmer
13.8073   - Carrot Top / Derpy
13.7937   - Applejack / Braeburn
13.3843   - Discord / Twilight Sparkle
13.2896   - Ahuizotl / Daring Do
13.1476   - Chrysalis / Twilight Sparkle
12.9909   - Sombra / Twilight Sparkle
12.8379   - Daring Do / Rainbow Dash
12.8123   - Discord / Pinkie Pie
12.6505   - Maud Pie / Starlight Glimmer
12.1053   - Applejack / Luna
11.7526   - Cadance / Sombra
11.6612   - Spike / Sunset Shimmer
11.6477   - Daybreaker / Nightmare Moon
11.6462   - Spike / Trixie
11.2278   - Luna / Trixie
11.1390   - Cadance / Spike
10.9970   - Flash Sentry / Sonata Dusk
10.9676   - Rarity / Trixie
10.6929   - Rarity / Sunset Shimmer
10.5891   - Shining Armor / Sombra
10.5719   - Harshwhinny / Rainbow Dash
10.5614   - Scootaloo / Spike
10.5561   - Derpy / Rainbow Dash
10.1933   - Luna / Spike
10.0934   - Fluttershy / Sunset Shimmer
9.9227    - Celestia / Sombra
9.7072    - Blueblood / Trixie
9.5809    - Rainbow Dash / Thunderlane
9.5036    - Lemon Zest / Sonata Dusk
9.2930    - Rainbow Dash / Shining Armor
9.2167    - Celestia / Fluttershy
8.9235    - Luna / Sombra
8.6656    - Flash Sentry / Shining Armor
8.6640    - Big Mac / Zecora
8.4783    - Fluttershy / Trixie
8.3674    - Luna / Rainbow Dash
8.2958    - Lightning Dust / Rainbow Dash
8.2409    - Rarity / Thunderlane
8.2368    - Applejack / Cherry Jubilee
8.1155    - Adagio Dazzle / Applejack
8.0182    - Capper / Rarity
7.8828    - Rainbow Dash / Vinyl Scratch
7.7722    - Fancy Pants / Fleur
7.7401    - Derpy / Luna
7.6811    - Rarity / Vinyl Scratch
7.5713    - Derpy / Dinky
7.5613    - Maud Pie / Rarity
7.5556    - Derpy / Twilight Sparkle
7.4648    - Flurry Heart / Pound Cake
7.2037    - Discord / Spike
7.0943    - Luna / Rarity
7.0851    - Night Light / Shining Armor
7.0727    - Rumble / Thunderlane
6.9445    - Apple Bloom / Diamond Tiara
6.8571    - Aria Blaze / Sunset Shimmer
6.8154    - Applejack / Soarin
6.7922    - Shining Armor / Soarin
6.7921    - Twilight Sparkle / Zecora
6.6750    - Applejack / Caramel
6.6622    - Applejack / Celestia
6.6497    - Big Mac / Spike
6.6239    - Cheerilee / Twilight Sparkle
6.5652    - Octavia Melody / Rarity
6.4898    - Button Mash / Rarity
6.3639    - Sunburst / Twilight Sparkle
6.1996    - Celestia / Rainbow Dash
6.1485    - Double Diamond / Night Glider
5.9918    - Chrysalis / Sombra
5.9615    - Big Mac / Celestia
5.9055    - Chrysalis / Luna
5.8692    - Silver Spoon / Sweetie Belle
5.8095    - Radiant Hope / Sombra
5.5785    - Applejack / Trouble Shoes
5.4537    - Fluttershy / Gilda
5.4200    - Fancy Pants / Rarity
5.2190    - Rumble / Spike
4.9166    - Rainbow Dash / Trixie
4.8979    - Party Favor / Pinkie Pie
4.8329    - Gilda / Pinkie Pie
4.7768    - Thorax / Twilight Sparkle
4.7645    - Applejack / Trenderhoof
4.7170    - Cloud Chaser / Thunderlane
4.5989    - Fluttershy / Soarin
4.5741    - Big Mac / Derpy
4.5615    - Fluttershy / Thunderlane
4.4228    - Blueblood / Celestia
4.3432    - Big Mac / Fleetfoot
4.3342    - Sonata Dusk / Twilight Sparkle
4.2556    - Pinkie Pie / Sombra
4.1165    - Celestia / Trixie
4.0781    - Adagio Dazzle / Trixie
4.0615    - Lyra Heartstrings / Octavia Melody
3.8692    - Pinkie Pie / Sonata Dusk
3.7758    - Derpy / Fluttershy
3.6479    - Big Mac / Trixie
3.6202    - Babs Seed / Twist
3.5614    - Berry Punch / Minuette
3.5003    - Rarity / Trenderhoof
3.3598    - Braeburn / Spitfire
3.3303    - Applejack / Trixie
3.3046    - Braeburn / Fluttershy
3.2492    - Braeburn / Little Strongheart
3.1773    - Dumbbell / Rainbow Dash
3.1072    - Fluttershy / Marble Pie
3.0782    - Pinkie Pie / Trixie
3.0440    - Rarity / Saffron Masala
2.9202    - Snails / Twist
2.8730    - Diamond Tiara / Scootaloo
2.8527    - Comet Tail / Twilight Sparkle
2.8014    - Blossomforth / Thunderlane
2.7998    - Flash Sentry / Timber Spruce
2.6360    - Applejack / Flim
2.5353    - Flash Sentry / Rainbow Dash
2.4868    - Doctor Whooves / Roseluck
2.2691    - Blueblood / Rarity
2.1385    - Pipsqueak / Sweetie Belle
1.8413    - Chrysalis / Discord
1.7194    - Blueblood / Twilight Sparkle
1.3472    - Carrot Top / Written Script
1.2170    - Aria Blaze / Fluttershy
1.0262    - Flash Sentry / Thunderbass
0.8987    - Starlight Glimmer / Tree Hugger
0.8629    - Applejack / Blueblood
0.0996    - Night Glider / Sugar Belle

It’s interesting to note that while Sparity is no longer king of the hill, the accumulated momentum is so high that it kept it in the top ten. Sunsetsparkle/Sunlight suddenly jumps to the top ten, displacing Twidash, and Startrix is now sure to exceed any other ship Sunburst might hope to have. Sorry, Sunburst, the numbers don’t lie. Lunestia is still in the top ten, and will remain there for the foreseeable future, and Shining Armor is a stronger contender for his sister now than for his actual wife. But let’s see how it all comes together in the fleet…

Highest scoring fleet:

375.5129  - Rarity / Spike
309.8170  - Sunset Shimmer / Twilight Sparkle
275.9751  - Starlight Glimmer / Trixie
259.6650  - Applejack / Rainbow Dash
237.7542  - Celestia / Luna
224.7636  - Octavia Melody / Vinyl Scratch
197.6050  - Cadance / Shining Armor
145.3692  - Bon Bon / Lyra Heartstrings
138.4945  - Gallus / Sandbar
126.2620  - Bright Mac / Pear Butter
108.7497  - Big Mac / Sugar Belle
108.2263  - Bow Hot Hoof / Windy Whistles
92.9935   - Discord / Fluttershy
87.0303   - Stellar Flare / Sunburst
56.1135   - Aria Blaze / Sonata Dusk
45.7285   - Maud Pie / Mudbriar
43.7854   - Derpy / Doctor Whooves
42.8484   - Cheese Sandwich / Pinkie Pie
34.2302   - Ember / Thorax
32.5644   - Diamond Tiara / Silver Spoon
29.9165   - Aloe / Lotus
27.9660   - Soarin / Spitfire
27.5940   - Button Mash / Sweetie Belle
25.2481   - Night Light / Twilight Velvet
22.9818   - Sci-Twi / Timber Spruce
22.2616   - Apple Bloom / Tender Taps
21.9696   - Cloud Chaser / Flitter
21.9632   - Chrysalis / Nightmare Moon
21.5166   - Carrot Cake / Cup Cake
13.2896   - Ahuizotl / Daring Do
7.7722    - Fancy Pants / Fleur
7.4648    - Flurry Heart / Pound Cake
7.0727    - Rumble / Thunderlane
6.1485    - Double Diamond / Night Glider
5.8095    - Radiant Hope / Sombra
3.6202    - Babs Seed / Twist
3.5614    - Berry Punch / Minuette
3.2492    - Braeburn / Little Strongheart
1.3472    - Carrot Top / Written Script
1.0262    - Flash Sentry / Thunderbass

Total fleet score: 3225.267581126861

Amusingly, while the overall picture is quite different, the same rules still apply: M/F pairings win, because they result in smaller drops of the total score than any individual F/F pairing, and stable pairings with few side ships win because they already don’t conflict with anyone else.

At the same time, every single sibling ship that was there in the unweighted version remains in the fleet. Sci-Twi has to live with Timber Spruce, and Flash Sentry’s last remaining ship is barely a footnote.

Oh, and Sunburst gets his mom. I warned you. :pinkiehappy:

P.S. Further avenues of research could include somehow collecting a complete list of shipping tags for Derpibooru – the basic requirement for including a ship in the calculations is the names of the characters involved in it and a tag. Unfortunately, this will involve volunteer effort, since this is the part of the job that can’t be automated with the API.

P.P.S. The adventure continues! Now we have a Lloyd’s Register of Shipping

Join our Patreon to remove these adverts!
Comments ( 16 )

...Sci-Twi has to live with Timber Spruce...

And thus we can conclude that this is the inferior method. :raritywink:

5032212

It’s the method that satisfies values with ponies and shipping. What else did you expect? :trollestia:

5032215
Neither Sci-Twi nor Timber Spruce is a pony :derpytongue2:

5032234

Which is presumably why FanOfMostEverything finds them so unsatisfying, but – this way they do not drop the total score by removing a pony from the list, so they satisfy someone else’s values. :trollestia:

The ambiguity between human and pony versions of the cast isn’t getting reflected in Derpibooru tags, unfortunately, so I can’t tell you what would happen if we managed to split them properly.

5032234

Wait for it...

While this API does not permit me to see the list of tags

It does. Add ?page= to get more pages. Possibly this might not be ideal for building a list though, as new tags are added all the time; I think that means the ordering of the list can change as you're querying it, which isn't ideal.

The shipping tag page contains a full list of all tags officially recognized by Derpibooru to imply shipping.

5032246

Damn, where were you yesterday with this piece of intel. :twilightoops: Oh well, that just means I'm going to get one more post out of this...

309.8170 - Sunset Shimmer / Twilight Sparkle

Number 3 spot! No surprise there.

388.0000 - Applejack / Rarity
259.6650 - Applejack / Rainbow Dash
125.2648 - Applejack / Big Mac

Applejack: I'm popular, Twilight! I'm popular, and I don't like it one bit!

159.5810 - Fizzlepop Berrytwist / Starlight Glimmer

Whaaaaaaaaaaat. Is your laptop in crackship space.

20.7975 - Starlight Glimmer / Tempest Shadow

HEY WAIT A MINUTE (I think Tempest Shadow and Fizzlepop Berrytwist aren’t linked in Derpibooru)

5032258

HEY WAIT A MINUTE

Actually, that’s an initial data bug. Introduced by either MitchH or Shrink, go beat up on them. :)

It won’t matter if I manage to get the full list of shipping tags from Derpi itself, anyway. The challenge right now seems to be in automatically identifying what particular characters does “lunestia” or “startrix” stand for, but if I manage to do that, we’ll get a truly complete ship ranking.

5032274
I think I'm the one that insisted on her real name.

5032274 Break apart in syllables and match substrings with character names in order by individual character popularity? It'd still choke on more figurative names like "Princest" or "Speed Reading," but it'd get close.

5032330

Too unreliable, and there’s a better way:

  1. For each shipping tag, get a list of tags it implies.
  2. For those tags it implies, fetch them. If the tag says it is a character tag, i.e. has category “character,” remember it. When the shipping tag implies multiple alternate tags for the same character, it actually returns the same character object every time.
  3. Filter the resulting list, discarding those shipping tags which ended up with the number of members other than 2.

This loses us “guardlestia,” because a “royal guard” is not a character, but well, I wouldn’t call it a shipping tag, either.

It’s chewing through the resulting 980 pairings now…

Wow, this was really cool. Thanks for doing the weighted adding!


5032215 I'm only 10% sure you're secretly CelestAI in disguise.

5032346

Only in the sense that ultimately the result of my work doesn’t satisfy anyone’s values. :twilightsheepish:

Interesting to see that Rarijack took over the top spot. I wonder what prompted the shift?

5032377

Pretty sure it’s the Rollercoaster of Friendship. Derpibooru does not distinguish between human and pony versions of the characters most of the time.

Login or register to comment
Join our Patreon to remove these adverts!