/r/osuplace Behind The Scenes: Technology

The /r/osuplace community had a couple of technical projects running behind the scenes for making things work like the overlay on the /r/place canvas and the verification website. In this article I will talk about these projects and explain many of the components that were made and used. These tools are also made available for you to use, study and to potentially improve upon. I will apologise in advance for abundant usage of technical jargon in this article. I go pretty deep into the details for certain things we have done. I hope you will be able to get a rough idea of what we did. Firstly, I will talk about the verification site that was made to automatically verify users based on their Reddit account age. Secondly, I will talk about the overlay that was made for users so they know where to place the pixels on the canvas. Lastly, I will cover some of the Python scripts that were used to make the overlay work correctly. At the end I have also included some statistics and links to many of the people that worked on the projects.

Verification Site

A couple of days before the start of r/place oralekin approached me about using my tournament verification system (TVS) for the r/osuplace Discord server. I would help him out with the project and host it for them for free for the duration of the event. The TVS verifies a user by having them login with their osu! account and their Discord account on a website. After the user has done both, they will be checked against the criteria set in the backend. Then the Discord bot that sits in the same backend joins them to the Discord server and adds a role signifying that the user is verified now. The main advantage for this application is that staff members on medium to large servers do not have to manually verify the details of a user anymore. The application can also accommodate for edge cases where users do not meet the requirements, and they will be shown an error page or directed to the staff members of the respective discord server.

I modified this system recently for other people to be able to easily adapt and use in their own environments. So, I directed oralekin to this new version and he set it up with an additional path for the verification of a user's Reddit account and different verification criteria. I also assisted him with certain regressions that he came across when I changed the backend to be more universal. These changes will be added to the upstream repository as well. When it was finished we did a couple of test runs on the domain name and declared it ready for the event.

oralekin managed the DNS for his subdomain (o.ralek.in) and I gave him the IP of the proxy that this verification site was hosted behind. Due to the expectation of a high traffic flow and the potential of malicious actors trying to do something to it, we did the following things:

• Secured the backend using built-in security features
• Implemented a rate limiter to prevent the server from being flooded
• Used Cloudflare as a proxy to hide the server's IP and to also make use of their other security features like DDoS protection
• Used HTTPS end-to-end from the origin to the user using a combination of Let's Encrypt and Cloudflare. In Cloudflare the security was set to Full (strict).

We did not use HSTS, because if it is configured wrongly it could leave the domain unreachable for a long time.

It successfully verified more than 10.000 people during the event and it did not crash at all. We saw only two types of actions fail. From time to time Reddit's authentication would fail, because it could not obtain the access token for a user. This is likely because we were hitting rate limits on the Reddit API which caused the failures. The second action we saw fail was writing of cookies for certain users. This could be due to the fact that they have cookies or javascript blocked by default on their browser for privacy reasons, because both are required and we do not write the cookie without consent.

The final code base for this verification server can be found on GitHub. We are proud that this project was successful and useful to the community.

/r/place pixel overlay

After publishing the verification project oralekin told me they were working on an overlay to show users where to place their pixels on the canvas of /r/place for the osu! logo. I joined their voice chat and they were initially having some problems finding the right spot to insert the overlay into. Their original plan was to hook into the canvas itself and track the movement and zoom level. However, I quickly found out they were overthinking the operation by inserting an image above the canvas inside the same container.

By appending an element to that container it was possible to overlay an image that just worked without any extra effort needed. The first proper version of that element was the following:

<img style="position: absolute;left: 677px;top: 677px;image-rendering: crisp-edges;width: 101px;height: 101px;" src="https://cdn.discordapp.com/attachments/959453204163199056/959454670479298590/dotted_template_v3.png">

The first version of the Tampermonkey script that came out of this was:

// ==UserScript==
// @name         osu! Logo template
// @namespace    http://tampermonkey.net/
// @version      0.1
// @description  try to take over the canvas!
// @author       oralekin
// @match        https://hot-potato.reddit.com/embed*
// @icon         https://www.google.com/s2/favicons?sz=64&domain=reddit.com
// @grant        none
// ==/UserScript==
if (window.top !==window.self) {
    window.addEventListener('load', () => {
            document.getElementsByTagName("mona-lisa-embed")[0].shadowRoot.children[0].getElementsByTagName("mona-lisa-canvas")[0].shadowRoot.children[0].appendChild(
        (function () {
            const i = document.createElement("img");
            i.src = "https://cdn.mirai.gg/tmp/dotted-place-template.png";
            i.style = "position: absolute;left: 0;top: 0;image-rendering: pixelated;width: 1000px;height: 1000px;";
            console.log();
            return;
        })())
    }, false);
}

https://hot-potato.reddit.com/embed was embedded as an iframe on the Subreddit, so we had to address it directly to be able to access the child ShadowRoots of that embed. Reddit also used the Custom Elements API for the elements of r/place which they named <mona-lisa-embed/> ,<mona-lisa-canvas/> etc. This made it easy to traverse the DOM using JavaScript to get to the correct location for the element. After that many other contributors like LittleEndu, Wieku, ekgame and DeadRote added more features to this script. The very first versions can be found on oralekin's GitHub Gist. As you may have noticed I did not get co-credited for creating the script, but I do not mind it. Later, oralekin's gist got replaced with this script made by Wieku, Deadrote, and 101arrowz, hosted on Wieku's GitHub Gist. This takes a different approach using the normal version of the canvas. It masks and dots it in the browser without the need for separate python scripts. A user interface is also included for some level of control on how the overlay gets displayed. The screenshots showcasing the userscript have been taken after the wipe of /r/place.

Over the course of the event I heard that the script got used by many communities and even got expanded on by them with their own versions and image templates, which is awesome. However, there were also certain communities that used the code, but also took the effort to remove the original authors' names from the script.

The image template that /r/osuplace used was hosted on my S3 space with CDN. So, while I was not credited, it was very obvious that I was involved with this script.

The following two files are hosted and served to hundreds of thousands of users across the world:

• https://cdn.mirai.gg/tmp/dotted-place-template.1-1.png (full canvas with all the allies and friends of /r/osuplace)
• https://cdn.mirai.gg/tmp/dotted-place-template.png (full canvas as dotted template suited as overlay)

The images are hosted on DigitalOcean Spaces but with Cloudflare functioning as the proxy for a secure CDN connection between the two. Initially, this caused a number of issues due to double caching happening. There was caching on the edge at Digitalocean, but Cloudflare also cached it. The solution was to create a page rule for the CDN subdomain and to set that to bypass all of Cloudflare's caching. This way we could keep using Cloudflare's proxy to keep a secure connection between it, DigitalOcean and the end user, as well as prevent any of the caching problems. We also set the edge caching TTL on DigitalOcean at 10 minutes and if quicker refreshes were needed we could manually purge the cache for the specific files as well.

The files were generated by LittleEndu using a couple of Python scripts. I have received his permission to document the scripts, so these will be listed below. Where applicable, instructions on how to run them are also given, with example output. I also added his name and the original filename at the top of every file.

# dotter.py by LittleEndu (2022)
import sys
from pathlib import Path

from PIL import Image
import numpy as np

target_filepath = Path(sys.argv[1])
img = Image.open(target_filepath).convert('RGBA')  # make sure image is RGBA
img_3x = img.resize((img.width * 3, img.height * 3), Image.NEAREST)
img_3x_arr = np.array(img_3x)

for i in range(img_3x_arr.shape[0]):
    if i % 3 == 0 or (i - 2) % 3 == 0:
        img_3x_arr[i, :, 3] = 0
for i in range(img_3x_arr.shape[1]):
    if i % 3 == 0 or (i - 2) % 3 == 0:
        img_3x_arr[:, i, 3] = 0

img_3x_back = Image.fromarray(img_3x_arr)
img_3x_back.save(f'!dotted_{target_filepath.name}')

This first script resizes the image to 3 times the original size in both axes with nearest neighbour scaling. After that makes every first and 3rd duplicate pixel blank to create a dotted template of pixel art.

Run instructions:

> pip install pillow numpy
> python ./dotter.py file-to-dot.png

LittleEndu also made a script that reverts the dotted template back to the original image:

# reverse_dotter.py by LittleEndu (2022)
import sys
from pathlib import Path

from PIL import Image
import numpy as np


def i_dot(target_filepath):
    target_filepath = Path(target_filepath)
    img = Image.open(target_filepath)
    img.paste(img, (-1, -1), img)
    img_array = np.array(img)
    small_img_array = img_array[::3, ::3, :]
    small_img = Image.fromarray(small_img_array)
    small_img.save(target_filepath.with_suffix('.1-1.png'))


if __name__ == '__main__':
    i_dot(Path(sys.argv[1]))

Run instructions:

# pip install pillow numpy
# python ./reverse_dotter.py dotted-template.png

If you use the dotted template as input, the following output will be produced:

He also created a script that placed pixel art on the canvas at specific x and y coordinates. This was useful for setting the location of artwork on the canvas for the template.

# convert_to_2k.py by LittleEndu (2022)
import sys
from pathlib import Path

from PIL import Image

target_filepath = Path(sys.argv[1])
img = Image.open(target_filepath).convert('RGBA')

x = int(input('x coord: '))
y = int(input('y coord: '))
w, h = img.size

new_img = Image.new('RGBA', (x + w, y + h), (255, 255, 255, 0))
new_img.paste(img, (x, y))

new_new_img = Image.new('RGBA', (2000, 2000), (255, 255, 255, 0))
new_new_img.paste(new_img, (0, 0))
new_new_img.save(f"2k_{target_filepath.name}")

How to run (example with the first version of the /r/osuplace logo):

# pip install pillow
# python ./convert_to_1k.py template_v3.png
x coord: 727
y coord: 727

Lastly, he wrote a simple script to rename the files for the CDN and to generate a reverse dotted version.

# rename_for_cdn.py by LittleEndu (2022)
import shutil
import sys

shutil.copy(sys.argv[1], "dotted-place-template.png")

import reverse_dotter

reverse_dotter.i_dot("dotted-place-template.png")

The outputs of this script were the final files that were uploaded to the CDN for use by everyone using the Tampermonkey script.

Closing words

It was honestly so awesome to see the /r/osuplace community work together, not just by themselves, but together with many other communities as well. Over the duration of the event over 436.000 requests were made to the cdn, with more than 2/3rd actually hitting the png content-type. About 100 GB of bandwidth was transferred between my servers and end-users (verification server data included too). Cloudflare also reports "Web traffic requests by country" over the past 7 days. Here are the top 5 countries:

1. United States: more than 150.000 requests
2. Canada: more than 30.000 requests
3. Germany: ~30.000 requests
4. United Kingdom: more than 20.000 requests
5. Philippines: more than 15.000 requests

Thank you for reading through this blog post. I hope that you have learned something new from this, or maybe this motivated you to use one of our projects for yourself. Either way, I think the efforts of this community were super cool and I look forward to next time.

Links

oralekin's GitHub: https://github.com/oralekin
LittleEndu's GitHub: https://github.com/LittleEndu
Wieku's GitHub: https://github.com/Wieku
Deadrote's Twitch (has no GitHub): https://twitch.tv/deadrote
ekgame's GitHub: https://github.com/ekgame
exdeejay's GitHub: https://github.com/exdeejay
101arrowz's GitHub: https://github.com/101arrowz
My GitHub: https://github.com/MiraiSubject

Let me know if I forgot to credit anybody!

/r/osuplace verification stack: https://github.com/oralekin/oth-verification/tree/place
osu! tournament hub verification stack: https://github.com/MiraiSubject/oth-verification

First versions of the userscript: https://gist.github.com/oralekin/240d536d13d0a87ecf2474658115621b
Most recent versions of the userscript: https://gist.github.com/Wieku/7d5a88c0a34d5bdf0b6bd9959f7d5e1d/
/r/httyd place userscript (source for masking/dithering by 101arrowz):
https://github.com/Anticept/httyd-place

Mention me on Twitter or Discord if you have something to say about the article! https://twitter.com/MiraiSubject

Apologies if the English is lacking here and there, if you find any mistakes, let me know and I will change it.

Update 1 (2022-04-08 13:06 UTC): Added 101arrowz's contribution to the blog, courtesy of Anticept on Twitter.