Occasionally I would write a note on there that would fit well into the blog format, or was long and more story based. Some of my posts on this blog worked much better as wiki pages than as blog posts. I have wanted to unify these two sites for a long time now. Last week I finally got around to it!

The Ochrs site generator that I wrote to build the wiki can now generate chronological feeds of posts and RSS feeds. Feeds are generated based on tags, where each tag has its own RSS feed. A note can be added to a chronological feed by adding a publish date (date field in the frontmatter), which will then be used as the publish date in all feeds that the note is a part of. This approach is great since it allows each note to exist anywhere in the wiki hierarchy while also being a part of many feeds. The RSS feed will be at notes.zachmanson.com/<tagname>.xml, and the list of notes in a chronological feed can be shown on a page using the ochrs:chrono:<tagname> build function (see Ochrs Syntax for more on how this works), which will results in all the posts being displayed in blog-style HTML.

To succeed zachmanson.com/blog/ I have created the posts tag. The feeds can be seen at notes.zachmanson.com/posts and using RSS. I have moved the existing posts here over to the this new feed.

If you are reading this using RSS then I would like to apologize that you will have to subscribe to a new feed. I would have like to automatically redirect your feed reader, but GitHub Pages doesn't let you create 301 redirects. The redirect-from plugin is capable of meta tag redirects, but those are soft and not followed by feed readers. This is very frustrating.

I've marked a bunch of longer notes I've made as posts, so the new feed already has 7 new posts. Hopefully more will come soon!

The Error

The type error was being triggered by a chord component, where the key was undefined. The chords are pulled from Ultimate Guitar, so I inspected the JSON payload from the original source.

Why the hell is there a [ch]H[/ch] chord in there? That's definitely the cause as I never accounted for non-existent keys. Looking at Ultimate Guitar, it renders as a B chord.

Huh?

One Google Search Later

Germany and the Netherlands ha(d/ve) their own musical key notation that include(d/s) a H chord. This site claims it ended in "1994/1995", though I've seen other sources claim its still taught this way. This comes as an artifact of the bizarre history Western music notation, which is a problem I seem to keep running into recently in my attempts to learn more about music theory.

I learned in music school that the chord is called "B" (like in the rest of the world), but the note is called "H" (eg. the C Major scale would be C, D, E, F, G, A, H, C).

TobTobXX, who reported the bug

Western Music's Stupid Origins

Western music is based on ecclesiastic modes used in church in the early Middle Ages, which only used the diatonic notes of the C scale (natural notes). The musical notation systems of the time reflected this, not accounting for notes outside of the C scale. When sharps and flats later came into more common use, the existing notation systems needed a way to distinguish them from the natural notes they sat between.

This problem first arose with B natural and B flat, according to the Encyclopedia Britannica. The first method of distinguishing B from B flat was using two different forms of the lowercase "b" character:

Niecks, Frederick. “The Flat, Sharp, and Natural. A Historical Sketch.” Proceedings of the Musical Association, vol. 16, 1889, pp. 79–100.

Somewhere along the line in Germany, monks transcribing these square and round "b" characters confused the squared "b" for "h", and this was later assumed to be intentional. "H" became a convention for writing B natural, while the "b" character remained convention for writing B flat. "B" and "b" became amalgamated, both coming to represent modern B flat.

In the rest of the world, this "H" note didn't catch on.

In time, notation for sharp and flat notes other than B flat was needed. This use of square and round "b" to denote B and B flat eventually evolved into our modern notation for indicating natural and non-natural, accidentals. ♭ comes from the round "b", while ♯ and ♮ come from the square "b".

Somehow, Germany still hasn't fully corrected this mistake, continuing to use "H" to represent B natural and many places. Including Ultimate Guitar. Ugh.

Resolution

I don't love dealing with problems caused by the whims of millenia dead monks, but this was an interesting rabbit hole to fall into. The issue has since been patched, and I look forward to my mistakes ruining someone's day in 3023.

If you look into the source of a LinkedIn profile page (at your own personal risk) you'll find a text element like this

will have HTML like this

<div class="display-flex flex-row justify-space-between">
    <div class="
          display-flex flex-column full-width">

        <div class="display-flex flex-wrap align-items-center full-height">
            <span class="mr1 t-bold">
                <span aria-hidden="true"><!---->Product manager, Creative coder<!----></span>
                <span class="visually-hidden"><!---->Product manager, Creative coder<!----></span>
            </span>
            <!----><!----><!---->
        </div>
        <span class="t-14 t-normal">
            <span aria-hidden="true"><!---->Freelancer/Consultant<!----></span>
            <span class="visually-hidden"><!---->Freelancer/Consultant<!----></span>
        </span>
        <span class="t-14 t-normal t-black--light">
            <span aria-hidden="true"><!---->2009 - 2018 · 9 yrs<!----></span>
            <span class="visually-hidden"><!---->2009 - 2018 · 9 yrs<!----></span>
        </span>
        <span class="t-14 t-normal t-black--light">
            <span aria-hidden="true"><!---->Europe<!----></span>
            <span class="visually-hidden"><!---->Europe<!----></span>
        </span>

    </div>

    <!---->
    <div class="pvs-entity__action-container">
        <!---->
    </div>
</div>

Gross. Why the hidden duplicated text? Why the empty comments? Why seperate aria versions of text? Is this related to using Ember? LinkedIn has so many little unpleasantries like this.

You click it and you are taken direct to the paper. It really streamlines the process for a lot of sites.

It hilariously appears on Wikipedia articles, as if the University of Western Australia has been incredibly gracious by allowing me to see Wikipedia. The reasoning here is that access to (at least one of) the citations is being provided through LibKey Nomad, but this makes the banner no less obnoxious.

The banner doesn't even have a function like it does on other journals, clicking it just takes you to the references list.

Wikipedia is one of the most impressive things humans have created. Keep your banners to yourself, Third Iron.

There's no way of knowing that your last good day is Your Last Good Day. At the time, it is just another good day.

The story presents this in the context of Augustus' worsening condition and eventual death.

So far, I've been lucky enough to be free of facing slow health declines in myself and those close to me, but the concept of the Last Good Day has sat quietly in the back of my mind since first encountering it. Whenever I have some kind of impending loss, no matter how small, my thoughts tend to orbit around it.

With my bachelor's degree coming to a close soon, lots of my friends are moving away for careers or further study in other countries. The groups that have formed won't continue to exist in the way the have for the last 3 years. Not everyone is leaving of course and no one person is single load-bearing pillar of the group, but buildings tend not to handle losing half their supports.

I've been aware that this will happen for months, and at every group event in my head I'm constantly thinking Is this going to be the Last Good Day? Is this Good enough? Can I make this better? This is definitely not healthy and definitely doesn't help me enjoy the limited time left with my friends. This line of thinking is something I hesitate to even talk about with my friends in fear of inspiring the same kind of thought spirals in their minds.

Time has gone on and the erosion has begun. As I write this, another will be flying to Sydney tomorrow morning. Enough friends have left now that the Last Good Day has certainly passed, and yet I'm still not 100% sure which day it was. There's some strong contenders though.

That's life, I know I'll make more friends and there's no need to be melodramatic. Regardless, I'm going to miss this period and the people I've spent it with.

Cheers and auld lang syne and so forth.

Usability

537c928: Made the emoji summary more vague, so a correct guess shows up as 9 green squares no matter what, rather than giving away the shape of the final solution.

Minecraftle 24/05/2022

⬜⬜⬜
⬜⬜⬜
⬜⬜⬜

🟩🟩🟨
⬜🟩⬜
⬜🟩⬜

🟩🟩⬜
🟩🟩⬜
⬜🟩⬜

Minecraftle 25/05/2022 3/10
⬜🟩⬜
⬜⬜⬜
⬜⬜⬜

⬜⬜⬜
🟩🟩⬜
🟨⬜⬜

🟩🟩🟩
🟩🟩🟩
🟩🟩🟩

23dc225: Added check so that only valid Minecraft recipes could be submitted as guesses.

bc40b98: Added proper support for horizontal reflections of recipes.

f3295d7 & 36386df: Made end-of-game popup display different text for Random games.

fef189d: Changed the given items for a more interesting set of possible recipes.

30a3829: Made end-of-game popup togglable.

f43b062: Added loading indicator to initial pageload.

8a842d9: Added slot highlighting to the ingredients panel to help keep track of which items have been used before.

Optimisation

c502eb2: Rewrote the entire script that turned the 600+ Minecraft crafting recipe JSON files into single file with a usable schema. Previous implementation was very messy, future modifications will be much easier.

172b3f2: Optimised recipes validation, still relatively inefficient though.

cd0c61e: Made game submission API a seperate endpoint to user stats endpoint, rather than being a hacky overload of the stats endpoint.

007865b: Significantly reduced size of recipes.json to only include recipes that are actually possible to create with the given ingredients. This reduces initial pageload time and reduces load on my server.

007865b: Set init.js to store recipes.json and items.json in local storage, and only request them if they don't exist there. This makes pageloading much faster, and reduces load on my server.

Revenge

Since initial release 2500+ people have played the game. From what I can tell this primary originated from popularity at a local primary school and the now-defunct LikeWordle.com. Most of these players haven't become regular users though, only playing one or two games. Really its just me and my ex competing for the number one spot on the leaderboard. After months of hard work she overtook me as number 1 and I have been clawing my way back ever since. Obviously I could cheat since I'm the admin but where's the fun in that? Despite that, I'm not opposed to adding features to help me take back the crown.

309a557: Added indicators to the stats page to show users how far behind and ahead they are of their neighbours on the leaderboard.

5dca9d2: Added an account recovery button since I accidentally deleted my account the day after taking back the crown (I swear). This does expose a prexisting security flaw, but does enable it to be used in a positive way. For reference, clearing your browser data will make your device forget which account is it associated with.

Bugfixes

23dc225: Fixed bug caused by the use of item tags in Minecraft source files. Item tags represent groups of items that are effectively interchangable in crafting recipes, such as coal and charcoal. Minecraftle conversion scripts have been updated to expect this, and will replace these with the preferred variant e.g. coal/charcoal group will be replaced with coal.

d98e83a: Forced game to be set to Perth timezone. Deployment VPS is in North America, so the server-side calculation of the current recipes wasn't changing at midnight in Perth, where most of the player base is.

c0ce8c2: Fixed stick recipe being overwritten by second stick recipes that uses bamboo.

It was some of the first JavaScript I ever wrote and it reflects that, much of the old code makes me want to cringe. With the benefit of 6 months professionally working on large TypeScript projects, Minecraftle seems quaint (and somewhat shocking that it even works).

Regardless, it holds a place in my heart as my first webdev project with any complexity and is likely responsible for getting my foot in the door at my current job. For this reason I am willing to continue tending the garden it occupies.

Over the last week I built Alculator using Next.js and hosted on Vercel. The site uses data lovingly freebooted from the Dan Murphy's public API to rank products by the ratio of price to standard drinks, and displaying this data in a card.

The images for these cards are hosted on a Dan Murphy's server (media.danmurphys.com.au), and I am simply using Next.js <Image> tags to point to them.

...
    <div className="card center">
      <div
        className="flex center-aligned"
        onClick={() => (window.location.href = `https://www.danmurphys.com.au/product/${item.stockcode}`)}
      >
        <Image
          alt="Image of drink"
          height="100"
          width="80"
          src={`https://media.danmurphys.com.au/dmo/product/${item.stockcode}-1.png`}
        />
        <div className="fill-width">
          <div className="flex space-between align-center">
            <h3>{item.name}</h3>
...

I'm using Vercel's free tier to serve the application which grants 100GB of bandwidth, which is magnitudes higher than any amount of users I expected to serve. I put the finishing touches on the site on December 30th, sent the link to a few people and went to the pub.

Over the next few hours I started receiving strongly worded alerts from Vercel:

Either I had gone viral or my code was a lot more inefficient than I thought. At 1AM I returned home, having reached the Ballmer Peak to diagnose the problem.

On the Vercel/Dashboard/Usage page, right at the bottom is the harmless looking graph titled "Image Optimisations" which turned out to be the culprit. By default, Vercel takes all images hosted in <Image> tags and caches them all at the Edge. Vercel had optimised 1400 of these images before I discovered this and at the free tier this is capped at 1000 images. Whoops.

The solution is simple:

next-config.js

/** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: true,
  images: {
    unoptimized: true,
  },
};

module.exports = nextConfig;

ssg.py

def blog_footer():
    return """
    <footer>
      <p>
        <a href="/blog/feed.xml">Feed</a> -
        <a
          href="https://github.com/pavo-etc/pavo-etc.github.io/tree/master/generator"
          >ironprof</a
        >
      </p>
    </footer>"""

page_template = jinja2.Template(open("template.html", "r").read())
with open("output.html"), "w") as f:
    f.write(page_template.render({
        "blog_footer":blog_footer
    }))

template.html

<!DOCTYPE html>
<html>
  <body>
    {{ blog_footer() }}
  </body>
</html>

output.html

<!DOCTYPE html>
<html>
  <body>
    <footer>
      <p>
        <a href="/blog/feed.xml">Feed</a> -
        <a
          href="https://github.com/pavo-etc/pavo-etc.github.io/tree/master/generator"
          >ironprof</a
        >
      </p>
    </footer>
  </body>
</html>

This follows normal Python rules, so inheritance is followed and parameters can be passed into functions in the template.

components.py

def nav(activated_link):
    links = ["projects", "blog"]
    link_elements = []
    for link in links:
        is_active = ' activated-link' if activated_link == link else ''
        link_elements.append(f'        <a href="/{link}/" class="discrete-link{is_active}">{link.capitalize()}</a>')

    right_links_str = '\n'.join(link_elements)

    is_root_active = ' activated-link' if activated_link == '.' else ''
    return f"""
    <nav>
        <div class="flex">
        <div class="flex-left">
            <a href="/" class="discrete-link{is_root_active}">Zach Manson</a>
        </div>
        <div class="flex-right">
{right_links_str}
        </div>
        </div>
    </nav>"""

def blog_footer():
    return """
    <footer>
      <p>
        <a href="/blog/feed.xml">Feed</a> -
        <a
          href="https://github.com/pavo-etc/pavo-etc.github.io/tree/master/generator"
          >ironprof</a
        >
      </p>
    </footer>"""

ssg.py

import components

page_template = jinja2.Template(open("template.html", "r").read())
with open("output.html"), "w") as f:
    f.write(page_template.render({
        "components":components
    }))

template.html

<!DOCTYPE html>
<html>
  <body>
    {{ components.nav("blog")}}

    <p>Lorem ipsum amirite</p>

    {{ components.blog_footer() }}
  </body>
</html>

output.html

<!DOCTYPE html>
<html>
  <body>
    <nav>
      <div class="flex">
        <div class="flex-left">
          <a href="/" class="discrete-link">Zach Manson</a>
        </div>
        <div class="flex-right">
          <a href="/projects/" class="discrete-link">Projects</a>
          <a href="/blog/" class="discrete-link activated-link">Blog</a>
        </div>
      </div>
    </nav>

    <p>Lorem ipsum amirite</p>

    <footer>
      <p>
        <a href="/blog/feed.xml">Feed</a> -
        <a
          href="https://github.com/pavo-etc/pavo-etc.github.io/tree/master/generator"
          >ironprof</a
        >
      </p>
    </footer>
  </body>
</html>

I have the standard complement of social media platforms for someone my age, but I wouldn't say I use any of them they way I would like to. For me, social media ideally is a tool I use to occasionally and asynchronously keep up with what's going on in my friends' lives. My actual use of social media platforms is one of the following:

• Barely use at all (Facebook)
• Be angry that barriers are put between me and my friends posts (Instagram)

Instagram once supported my ideal relationship with social media, but has incrementally moved further and further away. For years each update has made it harder to see my friends posts, cramming more things into my feed I have no interest in. The main feed has been so polluted that they recently added the "feature" to see a feed consisting exclusively of your friends posts. What was once the primary point has become a footnote in a changelog, hidden behind an awkwardly placed dropdown menu. Snapchat, while a very different beast, has suffered the same gentle decline in usability.

In contrast to this, the GitHub feed has been nothing but pure (though compsci slanted) embodiment of my ideal social media. Over the last 3 years of university, I have followed a total of 21 people followed people on GitHub, either friends or people I admire. This is tiny compared to my social graphs on other networks, but I found seeing updates from these 21 people more rewarding than hundreds of people on Facebook, and significantly less rage inducing than using Instagram.

Obviously these platforms are extremely different and selection bias plays a massive role in my experience using these feeds, but I seeing the projects that are being worked on by people I am interested in is always a lovely respite from the rest of the social internet.

Once you submit a prompt to DALL·E, you are shown more curated images and their corresponding prompts while your request is processed.

DALL·E is capable of producing some amazing images. It takes a bit of warming up, but after little practice I was able to create wonderful and bizarre generative art. Despite this, I felt I didn't know how to produce something as interesting or engaging as their curated images the presented to me when using DALL·E. I could logically understand that the tool had potential, but I felt seperated from it.

Using Midjourney is completely a different experience. Midjourney currently runs its generative art AI through a bot on their company Discord server. You generate art by typing in a command in certain channels, and all the art produced is visible to everyone on the server simultaneously. Before you can even type something you are assaulted by an array of incredible images, diffusing right before your eyes.

While both generators are impressive, using Midjourney was a visceral experience for me in a way that DALL·E was not. Where DALL·E is sipping a glass of water, Midjourney is a firehose blasting your face. I was immediately in awe.

The design of Midjourney makes it abundantly clear how big an impact this technology will have on the world. Watching hundreds of strangers generate incredible images in real-time is jaw dropping. The unfiltered, uncurated stream of regular people using the software is a lot more impressive than the sanitised, slick, and isolated interface of DALL·E. I was mesmerised, and within minutes saw images that moved me, despite being created by an AI without any intention behind them.

While I am sure both of these are the future, only one made me feel it.

Every time I begin to write for this site, I open Codium and create a new file in my repo. Then I realise I meant to create a new folder, call myself an idiot and start over. This is the first step of my process, which currently concludes with running my anemic static site generator. I hope to soon replace with this a bespoke and iron proficient(?) site generator, but until then I will continue using Pandoc to fulfill my static site generator requirements.

My static site generator requirements:

• Markdown input
• Code highlighting
• with theming?
• Simple templating
• Minimum (ideally 0) dependencies
• Automated?

Markdown Input

It can convert documents between myriad formats, most relevantly Markdown to HTML.

pandoc inputfile.md -o output.html --to=html5

Code Highlighting

There are plenty of tools that can do this, but Pandoc was the first I found that had code highlighting without having any depedencies like highlight.js, just saving the colour information in inline CSS

Any fenced Markdown code blocks with a language specified (like the following) will have the syntax automatically highlighted by Pandoc.

# Title

## Subtitle

Lorem ipsum amirite.  Hey look a fenced code block:

```Python
if x == 5:
    print("x is equal to 5")
```

Pandoc will turn this Markdown into the following HTML:

<h1 id="title">Title</h1>
<h2 id="subtitle">Subtitle</h2>
<p>Lorem ipsum amirite. Hey look a fenced code block:</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="cf">if</span> x <span class="op">==</span> <span class="dv">5</span>:</span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a>    <span class="bu">print</span>(<span class="st">"x is equal to 5"</span>)</span></code></pre></div>

The various HTML tag class spaghetti are used to highlight the code, though the CSS required to actually make this work is not included in the standard Pandoc command invocation. To actually generate the CSS that makes these classes work you need to use the --standalone flag, à la:

pandoc inputfile.md -o output.html --to=html5 --standalone

This will include some inline CSS, along the lines of:

code span.al { color: #ff0000; font-weight: bold; } /* Alert */
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code span.at { color: #7d9029; } /* Attribute */
code span.bn { color: #40a070; } /* BaseN */
code span.bu { } /* BuiltIn */
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code span.ch { color: #4070a0; } /* Char */
code span.cn { color: #880000; } /* Constant */
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
code span.dt { color: #902000; } /* DataType */
code span.dv { color: #40a070; } /* DecVal */
code span.er { color: #ff0000; font-weight: bold; } /* Error */
code span.ex { } /* Extension */
code span.fl { color: #40a070; } /* Float */
code span.fu { color: #06287e; } /* Function */
code span.im { } /* Import */
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
code span.op { color: #666666; } /* Operator */
code span.ot { color: #007020; } /* Other */
code span.pp { color: #bc7a00; } /* Preprocessor */
code span.sc { color: #4070a0; } /* SpecialChar */
code span.ss { color: #bb6688; } /* SpecialString */
code span.st { color: #4070a0; } /* String */
code span.va { color: #19177c; } /* Variable */
code span.vs { color: #4070a0; } /* VerbatimString */
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */

This CSS is great, but only actually needs to be generated once. I copied this CSS in a single file. After copying this text, the --standalone flag can be omitted.

with theming?

The colours used in the highlighting can be customised! As with everything else in my life at the moment, I use Dracula, which thankfully already has a theme for Pandoc.

Once the theme is installed, Pandoc can highlight using the theme with the --defaults flag:

pandoc inputfile.md -o output.html --to=html5 --standalone --defaults path/to/theme/dracula.yaml

The HTML classes will be the same names regardless of the theme, but the inline CSS will be dependent on the theme. Again, this only needs to be run once, and the inline CSS can be copied and saved to be used later.

Simple Templating

Pandoc makes it simple, as long as you only need simple template insertions! HTML templates can be created easily. A basic template could look like the following:

<!DOCTYPE html>
<head>
    <link rel="stylesheet" href="/styles/code.css">
    <title>$title$</title>
</head>

<body>
    <header>
        <h1>$title$</h1>
        <p>$subtitle$</p>
        <p>$date$</p>
    </header>

    <article>
        $body$
    </article>
</body>

Note the inclusion of the stylesheet. code.css includes the CSS for code highlighting.

The body variable is automatically assigned to the content of the Markdown file, and the rest of the variables can be assigned manually using a YAML header in the Markdown file.

---
title: A Titular Title
subtitle: A subtitular subtitle
date: 2022-06-12
---

## Chapter 1: Where it all began

Lorem ipsum amirite.  Hey look a fenced code block:

```Python
if x == 5:
    print("x is equal to 5")
```

To use the template, you use the --template flag in the Pandoc invocation:

pandoc inputfile.md -o outputfile.html --to=html5 --template=path/to/template.html

Dependencies

None! The outputted HTML has no external dependencies.

Automated?

It's a pain typing out that command for each post - so don't.

#!/usr/bin/env bash

nonposts=("styles/", "templates/")

for d in */; do
    echo "$d"
    if ! [[ ${nonposts[*]} =~ $d ]]; then
        echo "is post"
        echo "generating index.html"
        name=$(echo "$d" | sed 's/.$//')
        echo "pandoc ${d}${name}.md -o ${d}index.html --to=html5 --template=templates/template.html"
        pandoc ${d}${name}.md -o ${d}index.html --to=html5 --template=templates/template.html
    fi
done
echo "done"

This script is designed to work on the following directory structure:

blog
├── basic-seleniumbasic
│   ├── adding-reference.png
│   ├── basic-seleniumbasic.md
│   ├── dotnet-frameworks.png
│   ├── index.html
│   └── seleniumbasic-installer.png
├── covid-updates
│   ├── covid-updates.md
│   ├── index.html
│   └── media_release.png
├── crafting-recipes
│   ├── crafting-recipes.md
│   └── index.html
├── file-descriptors
│   ├── file-descriptors.md
│   └── index.html
├── generate_posts.sh
├── hep-001
│   ├── hep-001.md
│   └── index.html
├── index.html
├── poor-mans-site
│   ├── index.html
│   └── poor-mans-site.md
├── templates
│   └── template.html
└── usernames
    ├── index.html
    └── usernames.md

It opens every directory (except those in nonposts), runs pandoc on a Markdown file with the same name as the directory, and outputs index.html based on templates/template.html.

The top level index.html is not modified by the script at all. This page presumably would link to the subdirectories and must be manually modified. Ugh.

The assignment required the design and implementation of protocol to enable distributed compilation of files and remote command execution across multiple servers (r(emote) + (m)ake = rake). We needed to write a server program, and two client programs (one in C, the other in Python). Between Beej's guide and the coursework, we fumbled our way to working prototypes, able to pass files, messages, and any other arbitrary data between our clients and servers. The commands and files needing to be passed to servers were defined in a Rakefile, a strictly written parody of makefiles.

It was extremely satisfying to see the programs work, with files being sent from my partner's machine to servers at my house and back again in perfect-ish synchronicity. Just for our own gratification we tested it by compiling larger and larger C programs, which the servers handled by creating subprocesses to execute in parallel threads.

# Executes a given command
def run_command(cmd_str, execution_path):
    if cmd_str.startswith('remote-'):
        cmd_str = cmd_str[7:]

    if verbose: print("\tExecuting", cmd_str)
    proc = subprocess.Popen(cmd_str, cwd=execution_path, shell = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    global n_active_procs
    n_active_procs+=1
    return proc

(Obviously this is a security nightmare, we literally open the servers to execute arbitrary commands without any santising or checks. Please don't abuse. Or better yet, don't use.)

In our satisfaction-based testing, we noticed that the server would regularly crash when it received its 1010th to 1020th connection from the clients.

Got connection 1012 from ('192.168.0.28', 56193)
    <-- cost-query
    --> cost 2
Got connection 1013 from ('192.168.0.28', 56194)
    <-- 2 1 cc -c func3.c
    <-- filename: func3.c
    <-- file (size 39)
    Saved file /tmp/rs-5c1c832d-a18f-4189-b039-f6335af03aeb/func3.c
    Saved 1 files to dir: /tmp/rs-5c1c832d-a18f-4189-b039-f6335af03aeb
    Executing cc -c func3.c
Traceback (most recent call last):
  File "rakeserver.py", line 151, in <module>
    receive_command(received_data, connection)
  File "rakeserver.py", line 88, in receive_command
    proc = run_command(cmd_str, execution_path)
  File "rakeserver.py", line 41, in run_command
    proc = subprocess.Popen(cmd_str, cwd=execution_path, shell = True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1605, in _execute_child
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

1020 is suspiciously close to 1024, which immediately seemed like a red flag. Though select() wasn't being called in the server, I was aware of the existence of limitations around socket/file descriptors over 1024 in some functions.

As far as I could tell all sockets were being closed properly by the server and all files that were opened were closed. Adding some debug output revealed that the sockets were leaking in pairs‽

It was also noteworthy that many file descriptors were being reused by the server, so clearly some of them were being handled properly.

After hours of StackOverflow and dubious imitations of it, I resigned myself to hoping the markers never ran the server for long enough to get 1010 connections and notice the leak. The rest of the assignment was functional and we'd get a good mark regardless of this bug. The only thing at risk was my pride.

We fixed up the other remaining bugs, generally cleaned up formatting and completed our write-up, readied for submission the next day. On the morning it was due, I hoped fresh eyes would help me squash this tantalisingly simple bug.

Fresh eyes prevailed and I found the leak. To allow the stdout and stderr to be passed back to the client, we set the proc object to capture the command's outputs with the arguments stdout=subprocess.PIPE, stderr=subprocess.PIPE. These were retrieved later in the program:

msg2 = str(proc.stdout.read().decode("utf-8")) 
msg3 = str(proc.stderr.read().decode("utf-8"))

read() should have been the giveaway. Unbeknownst to me, subprocess.Popen() opens files to capture the outputs, which I was never closing.

msg2 = str(proc.stdout.read().decode("utf-8")) 
msg3 = str(proc.stderr.read().decode("utf-8"))

proc.stdout.close()
proc.stderr.close()

Leak plugged!

Update 2022-06-20:

Just learnt about popen() in C, now I feel really stupid.

For the first 2 years, this was sufficient for me, since for the most part WA was both physically isolated and had restrictions on who could enter the state. This led us to have very little community transmission, only having minor and shortlived clusters of cases every few months.

All good things must come to an end, and COVID-19 v5.0 Omicron debuted in WA on Christmas Eve 2021. Since then I've (somewhat unhealthily) been compulsively checking the listing of media releases every day to find the number of local cases.

Why not automate this?

Web scraping can't be that hard, right? It's not, but person writing the media releases clearly is trying to spite me. The scraping script I wrote first generates a list of every update since a given date by traversing the DOM with Beautiful Soup and parsing the article titles. This would be fine but on a whim the release titles will change format:

COVID-19 update 18 March 2020
COVID-19 update – 19 March 2020 (em dash)
COVID-19 update - 17 October 2020 (hyphen)
COVID-19 update – 1 October 2020 (em dash & no leading 0)
COVID-19 update – 01 January 2021 (em dash & leading 0)
COVID-19 update - 05 January 2021 (hyphen & leading 0)
COVID 19 update 2 December 2020 (COVID 19)
COVID19 update – 13 July 2020 (em dash & COVID19)
COVID-19 update –  4 October 2020 (em dash & 2-space gap)
COVID-19 update: 6 March 2022 (colon)
COVID-19 update  7 March 2022 (2-space gap)
COVID-19 update: 27th March 2022 (colon and ORDINAL)

Gross. The somewhat warty regex I resigned myself to:

match_text = "COVID.?19( )+(U|u)pdate.{0,6}[0-9]+(st|nd|rd|th)?( )+[a-zA-Z]+( )+[0-9]{4}"

I part of this I'm least proud of is .{0,6}, to just match up to 6 arbitrary characters. It's hacky but it does allow for the media releases to have more inexplicable spacing errors or random characters to use as seperators.

Ordinals only appeared in March 2022 and are what motivated my to whine about this on the internet. They are particularly annoying to my program since it only scrapes articles since the previous run. This is calculated by parsing the date in the media release title which is frustrating when they change the format of their dates on a whim. It's not a major change to the codebase, but it's just another spaghetto in my spaghetti.

Spaghetto 2: Actual Data

The titles are just the first issue. Once a list of COVID-19 updates has been assembled, each link is opened by Beautiful Soup, and the content of all <p> tags are concatenated. The case numbers are written in normal and again inconsistent sentences. The concatenated string is searched with more inelegant (and shockingly functional) regex to attempt to find the number of total new cases and locally transmitted cases.

# For local cases
match_text = "(total )?(of )?(([0-9]|,)+) new(?! local)( COVID-19)?( case(s?))?"
# For all cases
match_text = "(([0-9]|,)+)( are)?( new)? local( COVID-19)?( cases)?"

This took a surmountable and annoying amount of trial and error to find, and I'm sure will need updating in the future when the Government changes their phrasing on their next whim.

Ongoing Development...

For now this setup works without much any effort from me, barring the occasional regex update.

Unfortunately, this experience hasn't turned me off web scraping so I feel a niggling desire to add more functionality to the site. Daily hospitalisations, deaths, and other morbidities are all tempingly trivial to add, right? It's hard to call a project complete, even when you've already done everything you set out to do. I have a sneaking suspicious that the xkcd comic will be more prophetic than I'd like.

The meeting was very preliminary, essentially just getting to know one another and bounce some ideas off each other. One idea I had been mulling over was "Wordle but with for Minecraft crafting recipes", which requires no explanation for players of these two games (which among my peers includes everyone). My group immediately latched onto the idea, but acknowledged the scope of including all Minecraft recipes would be challenge, both in terms of game design and actually acquiring them.

Transcribing by hand was possible but likely impractical, as was automatically scraping the wiki. Luckily, the thousands of modders before us had poured through Minecraft's source file and through cursory googling we were able to get to .minecraft/versions/1.18.1.jar/data/minecraft/recipes/, where all crafting recipes in the game are stored as lovely little JSON files. Perfect!

Upon further inspection there are some mildly interesting quirks within these JSON files. The majority of recipes vaguely follow this format:

{
  "type": "minecraft:crafting_shaped",
  "pattern": ["###", "#X#", "#R#"],
  "key": {
    "R": {
      "item": "minecraft:redstone"
    },
    "#": {
      "item": "minecraft:cobblestone"
    },
    "X": {
      "item": "minecraft:bow"
    }
  },
  "result": {
    "item": "minecraft:dispenser"
  }
}

Seems like a straightforward and logical way of laying it all out. Presumably the "type" being "crafting_shaped" is to denote recipes with particular shapes (like a dispenser) as different from recipes where block arrangment doesn't matter (like mushroom stew).

Shapeless crafting recipes follow a simpler format:

{
  "type": "minecraft:crafting_shapeless",
  "ingredients": [
    {
      "item": "minecraft:brown_mushroom"
    },
    {
      "item": "minecraft:red_mushroom"
    },
    {
      "item": "minecraft:bowl"
    }
  ],
  "result": {
    "item": "minecraft:mushroom_stew"
  }
}

The recipes folder contains a lot more than two types though:

zach@grannysmith recipes % awk 'FNR == 2' *.json | sort | uniq -c
  24   "type": "minecraft:blasting",
   9   "type": "minecraft:campfire_cooking",
 543   "type": "minecraft:crafting_shaped",
 183   "type": "minecraft:crafting_shapeless",
   1   "type": "minecraft:crafting_special_armordye"
   1   "type": "minecraft:crafting_special_bannerduplicate"
   1   "type": "minecraft:crafting_special_bookcloning"
   1   "type": "minecraft:crafting_special_firework_rocket"
   1   "type": "minecraft:crafting_special_firework_star"
   1   "type": "minecraft:crafting_special_firework_star_fade"
   1   "type": "minecraft:crafting_special_mapcloning"
   1   "type": "minecraft:crafting_special_mapextending"
   1   "type": "minecraft:crafting_special_repairitem"
   1   "type": "minecraft:crafting_special_shielddecoration"
   1   "type": "minecraft:crafting_special_shulkerboxcoloring"
   1   "type": "minecraft:crafting_special_suspiciousstew"
   1   "type": "minecraft:crafting_special_tippedarrow"
  70   "type": "minecraft:smelting",
   9   "type": "minecraft:smithing",
   9   "type": "minecraft:smoking",
 198   "type": "minecraft:stonecutting",

Hold on, that's not just crafting recipes! All kinds "recipes" are stored in this directory, from normal smoking to smithing. This explains the repeated entries for items like cooked porkchop.

zach@grannysmith recipes % ls cooked*.json
cooked_beef.json                           cooked_mutton_from_smoking.json
cooked_beef_from_campfire_cooking.json     cooked_porkchop.json
cooked_beef_from_smoking.json              cooked_porkchop_from_campfire_cooking.json
cooked_chicken.json                        cooked_porkchop_from_smoking.json
cooked_chicken_from_campfire_cooking.json  cooked_rabbit.json
cooked_chicken_from_smoking.json           cooked_rabbit_from_campfire_cooking.json
cooked_cod.json                            cooked_rabbit_from_smoking.json
cooked_cod_from_campfire_cooking.json      cooked_salmon.json
cooked_cod_from_smoking.json               cooked_salmon_from_campfire_cooking.json
cooked_mutton.json                         cooked_salmon_from_smoking.json
cooked_mutton_from_campfire_cooking.json

Smelted items have an even simpler format:

{
  "type": "minecraft:smelting",
  "ingredient": {
    "item": "minecraft:porkchop"
  },
  "result": "minecraft:cooked_porkchop",
  "experience": 0.35,
  "cookingtime": 200
}

Puzzingly there is a value for cooking time for each item, though smelting time is dependent on the type of furnace used rather than the item being smelted.

For example, all items cooked in a blast furnace have the same value for cooking time:

zach@grannysmith recipes % cat *blast* | grep "cookingtime"
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100
  "cookingtime": 100

It could be to accommodate varying smelting times in the future, but for the moment this seems redundant.

All of the special crafting recipes aren't stored at all, which makes sense since these are more complex than what should be written in a single JSON file.

zach@grannysmith recipes % cat $(grep -lr . -e "special")
{
  "type": "minecraft:crafting_special_bookcloning"
}{
  "type": "minecraft:crafting_special_firework_star_fade"
}{
  "type": "minecraft:crafting_special_firework_rocket"
}{
  "type": "minecraft:crafting_special_shulkerboxcoloring"
}{
  "type": "minecraft:crafting_special_mapcloning"
}{
  "type": "minecraft:crafting_special_repairitem"
}{
  "type": "minecraft:crafting_special_bannerduplicate"
}{
  "type": "minecraft:crafting_special_armordye"
}{
  "type": "minecraft:crafting_special_tippedarrow"
}{
  "type": "minecraft:crafting_special_firework_star"
}{
  "type": "minecraft:crafting_special_mapextending"
}{
  "type": "minecraft:crafting_special_shielddecoration"
}{
  "type": "minecraft:crafting_special_suspiciousstew"
}

Tools smithed with netherite get a different and simple format as well:

> cat netherite_sword_smithing.json
{
  "type": "minecraft:smithing",
  "base": {
    "item": "minecraft:diamond_sword"
  },
  "addition": {
    "item": "minecraft:netherite_ingot"
  },
  "result": {
    "item": "minecraft:netherite_sword"
  }
}

Stonecutting unveiled an interesting quirk:

> cat chiseled_stone_bricks_stone_from_stonecutting.json
{
  "type": "minecraft:stonecutting",
  "ingredient": {
    "item": "minecraft:stone"
  },
  "result": "minecraft:chiseled_stone_bricks",
  "count": 1
}

The count value determine how many items are output by the recipe, and appears in over 579 total recipes. For the standard crafting recipes this value only appears for recipes that output more than 1 item, but curiously stonecutter recipes always include this value.

All this elicited a nose exhale and an "oh neat" from me as I poked around the files. Most of these details aren't important to the project, but the parts that are important will be a huge time save. Web scraping in my experience is a pain (though somehow I think the maintainers of the Minecraft wiki would be more consistent than the WA Government).

Now we can focus on the real trouble - every other aspect of puzzle game design and web development. I am particularly not looking forward to dealing with the interaction between Wordle-style position hints and different crafting orientation/positions.

Update 2022-06-05:

Minecraftle is playable and nearly fully finished. We submitted two weeks ago but there are a few things I want to implement for my own interest. The JSON files discussed in this post were used, but their format was inconvenient for querying so needed to be restructured. Yesterday I rewrote the script to restructure the recipe JSON files since the initial version was a mess, and finally implemented the functions to validate recipes on object placement. These changes revealed a few more quirks in the recipe JSON files.

Some recipes have multiple forms, such as torches which use coal or charcoal. I had assumed this would just result in two seperate recipe files, but:

> cat torch.json
{
  "type": "minecraft:crafting_shaped",
  "pattern": [
    "X",
    "#"
  ],
  "key": {
    "#": {
      "item": "minecraft:stick"
    },
    "X": [
      {
        "item": "minecraft:coal"
      },
      {
        "item": "minecraft:charcoal"
      }
    ]
  },
  "result": {
    "item": "minecraft:torch",
    "count": 4
  }
}

This actually caused a mild issue in Minecraftle that I currently have a hacky solution for. As far as I can tell this doesn't appear in many recipes (though I'm too lazy to write the regex to check). When we restructure the JSON files, we simplify the format so the pattern field contains the actual item names rather than character representations. This doesn't leave much room alternate forms, short of just repeating the whole recipe with the differing items. While totally possible, I just forced it to select the first item whenever there are multiple options. This means that you would be unable to craft a torch with charcoal.

As far as I can tell this poses no issues since in Minecraftle's current form you don't receive charcoal as a crafting ingredient, and I am yet to find another appearance of this format that uses the items in given_ingredients.json. I'm still uncomfortable with this though, and I hope to come back and fix it after exams.

Another oddity reared its head with stone tools:

> cat stone_axe.json
{
  "type": "minecraft:crafting_shaped",
  "pattern": [
    "XX",
    "X#",
    " #"
  ],
  "key": {
    "#": {
      "item": "minecraft:stick"
    },
    "X": {
      "tag": "minecraft:stone_tool_materials"
    }
  },
  "result": {
    "item": "minecraft:stone_axe"
  }
}

See it? tag replaces item in a number of cases, such as stone tools. Blackstone and cobbled deepslate have been added fairly recently to the game, and can be used for interchangably for cobblestone in a number of recipes. This kind of notation appears in a number of recipes:

> grep  "\"tag\":" *
acacia_planks.json:      "tag": "minecraft:acacia_logs"
barrel.json:      "tag": "minecraft:planks"
barrel.json:      "tag": "minecraft:wooden_slabs"
beehive.json:      "tag": "minecraft:planks"
birch_planks.json:      "tag": "minecraft:birch_logs"
black_bed.json:      "tag": "minecraft:planks"
blue_bed.json:      "tag": "minecraft:planks"
bookshelf.json:      "tag": "minecraft:planks"
bowl.json:      "tag": "minecraft:planks"
brewing_stand.json:      "tag": "minecraft:stone_crafting_materials"
brown_bed.json:      "tag": "minecraft:planks"
campfire.json:      "tag": "minecraft:logs"
campfire.json:      "tag": "minecraft:coals"
cartography_table.json:      "tag": "minecraft:planks"
chest.json:      "tag": "minecraft:planks"
composter.json:      "tag": "minecraft:wooden_slabs"
crafting_table.json:      "tag": "minecraft:planks"
crimson_planks.json:      "tag": "minecraft:crimson_stems"
cyan_bed.json:      "tag": "minecraft:planks"
dark_oak_planks.json:      "tag": "minecraft:dark_oak_logs"
daylight_detector.json:      "tag": "minecraft:wooden_slabs"
fletching_table.json:      "tag": "minecraft:planks"
furnace.json:      "tag": "minecraft:stone_crafting_materials"
gray_bed.json:      "tag": "minecraft:planks"
green_bed.json:      "tag": "minecraft:planks"
grindstone.json:      "tag": "minecraft:planks"
jukebox.json:      "tag": "minecraft:planks"
jungle_planks.json:      "tag": "minecraft:jungle_logs"
lectern.json:      "tag": "minecraft:wooden_slabs"
light_blue_bed.json:      "tag": "minecraft:planks"
light_gray_bed.json:      "tag": "minecraft:planks"
lime_bed.json:      "tag": "minecraft:planks"
loom.json:      "tag": "minecraft:planks"
magenta_bed.json:      "tag": "minecraft:planks"
note_block.json:      "tag": "minecraft:planks"
oak_planks.json:      "tag": "minecraft:oak_logs"
orange_bed.json:      "tag": "minecraft:planks"
painting.json:      "tag": "minecraft:wool"
pink_bed.json:      "tag": "minecraft:planks"
piston.json:      "tag": "minecraft:planks"
purple_bed.json:      "tag": "minecraft:planks"
red_bed.json:      "tag": "minecraft:planks"
shield.json:      "tag": "minecraft:planks"
smithing_table.json:      "tag": "minecraft:planks"
smoker.json:      "tag": "minecraft:logs"
soul_campfire.json:      "tag": "minecraft:logs"
soul_campfire.json:      "tag": "minecraft:soul_fire_base_blocks"
soul_torch.json:      "tag": "minecraft:soul_fire_base_blocks"
spruce_planks.json:      "tag": "minecraft:spruce_logs"
stick.json:      "tag": "minecraft:planks"
stone_axe.json:      "tag": "minecraft:stone_tool_materials"
stone_hoe.json:      "tag": "minecraft:stone_tool_materials"
stone_pickaxe.json:      "tag": "minecraft:stone_tool_materials"
stone_shovel.json:      "tag": "minecraft:stone_tool_materials"
stone_sword.json:      "tag": "minecraft:stone_tool_materials"
tripwire_hook.json:      "tag": "minecraft:planks"
warped_planks.json:      "tag": "minecraft:warped_stems"
white_bed.json:      "tag": "minecraft:planks"
wooden_axe.json:      "tag": "minecraft:planks"
wooden_hoe.json:      "tag": "minecraft:planks"
wooden_pickaxe.json:      "tag": "minecraft:planks"
wooden_shovel.json:      "tag": "minecraft:planks"
wooden_sword.json:      "tag": "minecraft:planks"
yellow_bed.json:      "tag": "minecraft:planks"

These tags are referring to .minecraft/versions/1.18.1.jar/data/minecraft/tags/items/, which holds yet more JSON files with lists of all items that belong to each tag. For example:

> cat stone_crafting_materials.json
{
  "replace": false,
  "values": [
    "minecraft:cobblestone",
    "minecraft:blackstone",
    "minecraft:cobbled_deepslate"
  ]
}

To deal with these I've currently just put edge cases in to handle these tags manually, though I would like to have a more robust solution in the future that can be determined by given_ingredients.json.

Both of these appear to be solving the same problem, and it's strange that coal doesn't use the tag field, especially since there is tag JSON file that seems made for this:

> cat coals.json
{
  "replace": false,
  "values": [
    "minecraft:coal",
    "minecraft:charcoal"
  ]
}

I theorize that the torch was older method of achieving this goal and just hasn't been updated to use the new system (though I am again too lazy and won't check old versions), unless there is some other difference I am not aware of between the two methods. Maybe I should send Mojang a pull request.

Recipe validation was the final feature I really felt was necessary for my own sense of completion. That said, we never added shapeless recipe support... or varying difficulties... or hardcore mode. Oh well.

Abstract

This hEP proposes a framework for adding first-class support for additional forms of h used in mathematical contexts, and that "𝐡" be added to the h programming language standard implementation. As of version 1.0, the only supported output characters are "h" and "'".

Motivation

The h Programming language is borne from the conlang h which was formalised in 2019 by Xe Iaso.

In their unveiling of the language they exhibited how the conlang's simple syntax could be used for complex conversation, all within the use of a single written character and phone:

<Cadey> h
<DoesntGetIt> h h h h
* Cadey facepalms

As stated in h's formal specification, it can be used to represent all possible communications within the confines of the letter h.

They later developed a syntax, compiler and runtime for the h Programming Language (hlang). This programming language was designed for the output of the h conlang, and expanded the character set in use to include both the Romanic form "h" and its Lojbanic equivalent "'".

A long-standing difficulty for the programming language is expressing mathematical notation in its output. While it is entirely possible to express any and all equations in the language, for this purpose it doesn't utiltise the "h" character to its maximum capacity and as such is unnecessarily verbose.

Goals

The primary goals of this proposal are to provide a specification for implementing additional forms of "h" in hlang to enable more terse output for use in mathematical contexts. The only additional form of "h" proposed in this hEP is "𝐡" (U+1D421).

The proposal would enable programs such as this:

$ cat new_equation.h
𝐡 𝐡 h '

AST this would generate during compilation:

H("𝐡 𝐡 h '")

Program output:

𝐡 𝐡 h '

This is an improvement over the existing implementation, which would require the much longer and more cumbersome hlang 1.0 source to produce the equivalent equation:

$ cat old_equation.h
h h h ' ' h ' h h h h h h h h h ' h

Translating this from h (conlang) to English:

a^2 + b^2 = c^2

Non-Goals

This hEP is not an attempt to sully hlang with characters outside of the formal grammar of the h conlang. The scope of the specification, while allowing for additional forms of "h" to be supported, is exclusively for forms of "h". It is not applicable to other characters, such as "g".

This hEP only explicitly proposes the addition of "𝐡" (U+1D421) though is designed to allow other forms of "h" to be supported by hlang in the future, such as "𝒉" (U+1D489), "𝓱" (U+1D4F1), or "𝕙" (U+1D559). While these forms could be implemented in a near identical manner to "𝐡", that is outside of the scope of this hEP. All of these forms of "h" are UTF-8 characters designed for use in mathematical equations.

The hEP is also explicitly not intending to enable support for "h" with diacritics, such as "ḧ" (U+1E27). Etymologically related forms of modern "h" are also outside the scope of this hEP.

Semantics

Under this proposal, "𝐡" would receive first class support by hlang. To output it would be the same as the standard "h" character:

$ cat math_h.h
𝐡

Produced AST:

H(𝐡)

Output:

𝐡

Represented with a node tree:

&peg.Node{
    Name: "H",
    Text: "𝐡",
    Kids: nil,
}

The node tree for the earlier Pythagorean example would be:

&peg.Node{
    Name: "H",
    Text: "𝐡 𝐡 h '",
    Kids: {
        &peg.Node{
            Name: "",
            Text: "𝐡",
            Kids: nil,
        },
        &peg.Node{
            Name: "",
            Text: "𝐡",
            Kids: nil,
        },
        &peg.Node{
            Name: "",
            Text: "h",
            Kids: nil,
        },
        &peg.Node{
            Name: "",
            Text: "'",
            Kids: nil,
        },
    },
}

The "𝐡" character should be usable in all contexts where "h" and "'" are currently usable. Grammatically it should follow all the same principles and restrictions as "h" and "'".

Dependencies

As with hlang v1.0, this proposal not require any dependencies, other than UTF-8 formatted source files.

Acknowledgements

Thanks to Xe Iaso for creating h and hlang, as well as all their other incredible projects.

And after the arguing has concluded, the fighting parties express that they respect the other.

These are rare. But it isn't rare that upon finding one of these I will glance at the usernames.

Today I glanced at the usernames of two people having a beautiful argument on a forum:

"euronforpresident"

Fine, nice little Game of Thrones reference.

And their opponent:

"Hitleresque"

Luckily there is a glimmer of hope. SeleniumBasic is a Selenium based framework for the VB family. Now you may look at it and think "this is unmaintained and hasn't been updated since 2016", but it was actually perfected in 2016 and no update could ever possibly improve it.

So it is only natural that these two exquisite pieces of software (that are in no way whatsoever out of date) work together. Unfortunately there's a few...underdocumented hiccups that arise from integrating these two. This post is written in the hopes of making the hiccups between VBA (in Excel) and SeleniumBasic slightly more documented.

Installation

• Excel (duh)
• SeleniumBasic
• A WebDriver
• .NET Framework 3.5

Given that the VBA interface in Excel hasn't been updated since...1832...I am going to assume Excel version will not have any major impacts, but for reference I used Version 2201.

Each of the major web browsers provides a WebDriver, which is a piece of software that allows a browser to be controlled and automated by external programs. The SeleniumBasic installer includes a number of WebDrivers which you can choose to include, but these all appear to be wildly out of date and will cause Excel to crash if you have a recent browser installed.

To fix this you will need to manually replace the WebDrivers you intend to use. Each of the major browser vendors distribute up to date versions of their own WebDrivers (Edge, Chrome, Firefox). Within each of these zip archive you'll find an executable, e.g. msedgedriver.exe for everyone's fifth favourite browser. To replace driver included with SeleniumBasic, you need to place the up to date executable in C:\Users\<username>\AppData\Local\SeleniumBasic. The up to date WebDriver will need to have the same name as the original outdated WebDriver, so msedgedriver.exe needs to be renamed edgedriver.exe. Irritatingly, it's probable that eventually automatic browser updates will render even these updated WebDrivers outdated, and these executables will need to be replaced again.

.NET Framework versions annoyingly don't show up as installed programs, so if you want to find out what versions you already have installed, you must use one of several annoying ways. The least annoying of these is to look in the C:\Windows\Microsoft.NET\Framework directory.

Actual Use

Now you can actually use it as you'd expect. Open the VBA IDE in Excel start using some of the gorgeous syntax and best in-class tools that Microsoft so kindly allows you to purchase.

To enable SeleniumBasic in Excel you'll need to navigate to Tools > References and check Selenium Type Library.

From here you can write fairly standard Selenium code in VBA.

Sub AutoSearch()

Dim driver As Selenium.EdgeDriver
Set driver = New Selenium.EdgeDriver

driver.Start
driver.Get ("https://old.reddit.com/")

Dim searchbox As WebElement
Set searchbox = driver.FindElementByName("q", -1, True)
searchbox.SendKeys ("selenium")

Dim submit As WebElement
Set submit = driver.FindElementById("search", -1, True)
submit.submit

driver.Quit

End Sub

Footnotes

1. Software versions used: Windows 10 10.0.19043, Excel 2201, SeleniumBasic 2.0.9.0, Edge 98.0.1108.50, EdgeDriver 98.0.1108.51
2. Official Selenium Documentation
3. A fantastic resource for VBA+Selenium is WiseOwlTutorials' series on this exact thing.