Daemonic Dispatches

My re:Invent asks

As an AWS Hero I get free admission to the AWS re:Invent conference; while it's rare that I'm interested in many talks — in previous years I've attended "Advanced" talks which didn't say anything which wasn't already in the published documentation — I do find that it provides a very good opportunity to talk to Amazonians.

While I'm sure many of the things I ask for get filed under "Colin is weird", I know sometimes Amazon does pay attention — at least, once I find the right person to talk to. Since I have quite a list this year, and I know some Amazonians (and maybe even non-Amazonians) may be interested, I figured I might as well post them here.

1. More Amazonian OSS developers at re:Invent. I'm looking forward to meeting some Valkey developers on Wednesday, but I was disappointed that none of the Firecracker developers are in attendance. Amazon has a policy of not having engineers attend re:Invent unless they're giving talks (and I'd love to see this policy changed in general) but it's absolutely essential for Open Source developers to go to conferences; that's how we meet potential contributors. If your open source team doesn't go to conferences, they're not really doing open source, no matter what license you put on the code.
2. Lower cross-AZ bandwidth pricing. I don't even particularly care about the cost; but being worried about avoiding cross-AZ bandwidth is making people design bad systems. One of the guidelines in Amazon's "well-architected framework" is to deploy the workload to multiple locations and Amazon specificially calls out using a single Availability Zone as a problem — but concerns about cross-AZ bandwidth (even if it turns out that the concerns are unwarranted!) are preventing people from following this guideline.
3. On-the-rack EBS storage. I don't know how Amazon datacenters are set up, but the latency of disk I/O to "SSD" EBS volumes strongly suggests that they are a significant distance away from EC2 instances which are accessing them. At the other end of the latency scale, some EC2 instance types have SSDs directly attached to the instance hardware, with dramatically better I/O performance — but have low durability (if the instance dies the data is gone) and no elasticity (each instance type has a certain amount of disk attached).
   Having EBS storage available on the same racks as EC2 nodes would provide an intermediate point, allowing lower latency than the speed of light allows for across-the-datacenter I/Os, while allowing some flexibility in the size of volumes. Users would have to accept that "provision me a volume on the same rack as this instance" might return "sorry all the disks on that rack are full"; but at least at instance launch time requests could be satisfied by searching for a rack with sufficient rack-local disk.
4. CHERI capable instances. This has been a long standing wishlist item for me and I know I'm not going to get it any time soon; but I know Amazon (and other clouds) have Morello boards for research purposes. CHERI has huge advantages for security and whichever cloud pursues this first will be miles ahead of the competition.
5. Marketplace support for "pending" or "scheduled" releases. When I add new FreeBSD releases to the AWS Marketplace, they first go through an approval process and then get copied out to all the EC2 regions; once that is done, the Marketplace updates the "product" listing with the new version and sends out emails to all the current users telling them about the new version. This often means that Amazon is sending emails announcing new FreeBSD releases a couple days before I send out the official FreeBSD release announcement.
   I don't want to wait and add new versions to the Marketplace later, because the timeline is unpredictable — usually a couple hours but sometimes a day or more — so I'd like to be able to tell the Marketplace about the upcoming FreeBSD release and have them get everything ready but not update the website or send out email until I'm ready to send out our announcement (we usually allow a few days for mirrors and clouds to sync).

In addition to these, I also had a couple requests which I can't write about due to the nature of what I was asking for. I did however have one more request I can write about — not for AWS, but for a re:Invent sponsor. I ran into some people from Zoom and mentioned that every time I join a call from my (FreeBSD) laptop, the Zoom website tries to get me to open a Zoom client, and only offers "open the meeting in your web browser" after failing repeatedly. Since my web browser's user-agent string includes "FreeBSD", they should be able to detect that there is no Zoom client and go straight to opening the meeting in my web browser. To their credit, they immediately understood what I was complaining about, and even offered to look into whether they could port their Linux client to FreeBSD.

I don't know if or when I'm likely to get any of these, but I like to think that I convinced people that what I was asking for was at least somewhat sensible. Maybe between them and other Amazonians who will no doubt read this, I'll get at least a few of the things on my wishlist.

For the sake of transparency: In addition to giving me (and other AWS Heroes) free admission and travel to re:Invent, Amazon is sponsoring my FreeBSD work. About half of what they're paying for is EC2-specific stuff; the other half is FreeBSD release engineering. Without their support, a number of important features would not have landed in FreeBSD 14.2-RELEASE; thank you Amazon.

Generalist AI doesn't scale

There has been a lot of talk about AI recently, and one particular point has received sigificant attention in the tech industry: The cost of training models. According to some insiders — and the market capitalization of NVIDIA — the computing power needed for AI training threatens to upend the entire semiconductor industry. This should not be a surprise: Generalist AI doesn't scale.

Reduced to its essentials, the task of training a size-N model is one of hill-climbing in N-dimensional space. You take O(N) inputs, run them through your model, and after each of them you nudge the model slightly uphill towards the desired responses. You need O(N) inputs because with any less than that the model will overfit — essentially memorizing the specific set of inputs rather than generalizing from them — and for each of these inputs you need to perform O(N) computation since you have N parameters in the model to tune. End result: O(N^2) computation.

Now, there are plenty of other problems in AI — one of the largest is generating enough training data (easy enough for Chess or Go where you can have the AI play games against itself, but for general knowledge you eventually run out of textbooks) — and you can push against scaling laws for a while simply by throwing more money at them; but in the end you can't defeat scaling. You'll end up boiling the oceans.

So what's the solution? Don't do Generalist AI. Instead, we need to switch to using a pool of expert AIs. Instead of a single size-N model, split the model into k parts, each trained on N/k inputs. On sub-model learns all about medicine; another learns all about modern art. You still have N inputs, but since each of them is only used to optimize a set of N/k parameters, your training cost is now O(N^2 / k).

And yes, you lose something by doing this — you probably won't get hallucinations of modern artwork depicting polypeptides. But, as with humans, most queries can be answered by the appropriate specialist; and it's better to have a collection of experts than a generalist which is too expensive to train effectively. (One could even have a "dispatcher" sub-model which knows enough to identify which of the specialists to refer a query to.) And by reducing the training cost, you become able to build a collection of models which is larger — and smarter — than a generalist model could ever be.

Specialization isn't just for insects. It's for AIs too.

Please test: FreeBSD 13.3-RC1

I just announced the availability of FreeBSD 13.3-RC1. This is the first release candidate of FreeBSD 13.3, and if no further issues are reported will be the only release candidate; I would like to start 13.3-RELEASE builds on Friday, with (allowing time for mirrors to update) the release announcement going out on the following Tuesday (March 5th).

This means there's a few days for people to do some last-minute testing and report any problems they find. If you have time to help out with testing, there are two things in particular which I'd like to see get attention:

1. Wifi, especially the iwlwifi driver. Bjoern Zeeb merged a significant number of changes to the wifi and linuxkpi (which is used by iwlwifi) code between BETA3 and RC1. While these changes were tested extensively, it's still a big chunk of code — more than I would normally have wanted merged so late, but it fixed serious stability issues with iwlwifi so I thought it was worth including anyway. But I'll feel much better about the release if I know people have been testing this code.
2. The installer. Most people who test FreeBSD BETAs do it by upgrading existing systems — fair enough, you test what you have. But this means that the installer doesn't get nearly as much testing as running FreeBSD systems get. So if you have a spare system laying around, please download an installer image and make sure that you can install FreeBSD 13.3-RC1! In particular, keep an eye out for any "missing" hardware or error messages about drivers being unable to reserve resources; we had a late fix to the way that ACPI devices reserve resources.

This is the first FreeBSD release I've managed since assuming the role of FreeBSD Release Engineering Lead, and the first time I've been "flying solo" (I managed FreeBSD 13.2, but Glen was looking over my shoulder for most of the release process); so it's entirely possible that I've gotten something horribly wrong. If you see anything which looks strange, please don't hesitate to get in touch — either directly (cperciva@FreeBSD.org) or by emailing the release engineering team (re@FreeBSD.org).

It's a privilege to manage the FreeBSD release process, but it also takes a significant amount of time. If you'd like to help me find time to work on FreeBSD, please consider contributing to my Patreon.

An APPR claim with Air Canada

Like most countries now, Canada has regulations requiring passengers to be compensated for flight delays under certain circumstances; Canada's regulations are called the Air Passenger Protection Regulations (commonly known as "APPR"). In May 2023, I flew to and from BSDCan on Air Canada; on the way home, one of my flights was cancelled and Air Canada rebooked me for the following day.

I filed a request for compensation with Air Canada, but received a response indicating that the cancellation was due to "unforeseen maintenance" which was "required for safety purposes" and thus exempt from the compensation requirements. There were two appeal mechanisms available at this point: First, via the Canadian Transportation Agency — which theoretically reaches decisions within 90 days but has recently been reported to have a case backlog of over a year — and second, via small claims court. I opted to take Air Canada to small claims court — which in BC, means the online Civil Resolution Tribunal.

On July 13th 2023 I filed my initial claim for $1315 — $1000 of compensation for the flight delay, plus $315 because I was reboooked into "domestic business class" (regular seats) instead of the "signature class" (lie-flat seats) which I had originally booked:

On 2023-02-28 I booked flights with Air Canada from Vancouver to Ottawa on 2023-05-16 and returning from Ottawa to Vancouver via Toronto on 2023-05-21. The ticket was #014CENSORED and booking reference was CENSORED.

My return flights were originally AC455 (YOW-YYZ) and AC119 (YYZ-YVR) arriving in Vancouver at 2023-05-21 17:56; on 2023-05-12 I was notified that AC119 had been rescheduled from 16:00 to 17:00 (arriving at 18:56 instead of 17:56), and on 2023-05-16 I was further notified that AC119 was being cancelled and I was being rebooked to AC117, also departing at 17:00 and arriving at 18:56. I have no claim concerning these routine schedule adjustments.

On 2023-05-21 I took flight AC455 from Ottawa to Toronto as scheduled, but shortly before flight AC117 was scheduled to board, received a notification that "flight AC117/21 May from Toronto to Vancouver has been cancelled due to an unforeseen aircraft maintenance issue. We are actively looking to find an alternate flight and will update you shortly." Two hours later, I was rebooked to flight AC101 on the following morning, and arrived in Vancouver at approximately 08:30 AM, about 13.5 hours after my original scheduled return.

I filed a claim under the Air Passenger Protection Regulations via the Air Canada website on 2023-05-26, and the website indicated that "We're sorry but this flight is cancelled due to unexpected business or operational constraints on our planned flight schedule". On 2023-05-27, I received an email from Air Canada stating that "This flight was delayed due to unforeseen maintenance that does not include scheduled maintenance or mechanical problems identified during scheduled maintenance." and denying my claim for compensation on the grounds that the flight cancellation was "required for safety purposes".

I dispute Air Canada's claim, on the basis that the aircraft scheduled to operate flight AC117 on May 21st -- registration C-GFUR -- proceeded to depart later on May 21st for Bogota, operating flight AC94. While Air Canada may have had an unforeseen safety-related aircraft maintenance issue on a different Airbus 330-300, the aircraft scheduled for the cancelled flight AC117 was in fact operating (presumably safely!) and could have been used to operate the flight as initially scheduled; it was in fact Air Canada's operational choice to cancel flight AC117 in order to use that aircraft to operate AC94 instead.

Since there was no safety-related issue preventing AC117 from operating, per paragraph 19(1)(a)(iii) of the Air Passenger Protection Regulations, I am entitled to $1000 compensation.

Furthermore, my flight from YYZ to YVR was originally booked in Air Canada "signature class", with lie-flat seats ("pods"), while the flight on which I was rebooked (AC101) had regular "domestic business class". Per paragraph 17(6) of the Air Passenger Protection Regulations, I am entitled to a refund of "the difference in the cost of the applicable portion of the ticket", namely $300 + GST.

As such, under the APPR I am entitled to compensation of $1315 plus interest and any relevant fees under the BC Civil Resolution Tribunal Act.

On August 16th, Air Canada's lawyer responded, essentially saying "we deny everything".

Next up in the CRT process is "negotiation", which basically didn't happen; I sent a message via the CRT website explaining again why I was entitled to compensation, but Air Canada didn't respond. So we moved on to the following stage ("facilitation") where a Tribunal representative scheduled a conference call between us.

That call was scheduled for 9 AM on January 10th, and as soon as it started Air Canada's lawyer spoke up: "We've looked into this further, and Air Canada is prepared to pay Colin the $1315 he is claiming". They didn't agree to pay the $75 CRT filing fee, which theoretically I could have recovered if I continued through the process; but getting $1315 was good enough for me, so I agreed to take the money and drop the CRT claim.

My lessons learned, for the benefit of any of my readers facing similar flight delays and cancellations:

1. The Air Passenger Protection Regulations work!
2. If you're in BC, the Civil Resolution Tribunal is fantastic. Hopefully other provinces will set up similar bodies soon.
3. If your flight is cancelled, use a site like flightradar24 to see which plane was scheduled for your flight and what it is doing instead. If it spends the next couple days on the ground, it probably needed maintenance — but if it flies somewhere else, you probably have a case that it wasn't really in need of urgent safety-related maintenance.
4. Be patient and don't give up.

Safe travels, and I hope I'll see some of my readers at this year's BSDCan — hopefully without any flight delays!

A Canadian payroll dependency chart

The Canada Revenue Agency publishes, on a regular basis (they're now up to the 119th edition), a document entitled "Payroll Deductions Formulas". This document contains all of the formulas needed to calculate payroll deductions collected by the CRA: Canada Pension Plan, Employment Insurance, and Income Tax. As someone running a small business in Canada and not wanting to use an external payroll provider, I implement these formulas myself in a spreadsheet.

Unfortunately, like most aspects of the tax system, this steadily increases in complexity, and this year with the Canada Pension Plan including not just "base" premiums and "additional" premiums but also "second additional premiums", I needed to rewrite my spreadsheet from scratch; and also very unfortunately, the authors of the document have failed to grasp an essential detail of software: You have to calculate values before you use them. This results in a document requiring constant jumping back and forth to find where each value is calculated — before or after it is used.

To help myself keep everything straight, I decided to draw a dependency chart using GraphViz. There are 79 nodes in the graph, starting with values like "Year's Annual Maximum Pensionable Earnings" ($73200 in the 2024 tax year), and ending up with "Total payroll deductions". Note that I did not write the formulas onto the chart — I simply recorded which other values they depend upon, so someone (aka myself) looking at the chart can figure out what to calculate first without going around in circles.

Oh, and to keep things simple, I assumed I wasn't dealing with commissioned employees, employees entering or exiting the Canada Pension Plan, or anyone resident in Quebec, Nova Scotia, Yukon, or Ontario (each of which has their own special tax complications).

Here's what it looks like; click for a full size version (warning, 5627x2033 image):