I'm Upping My P(Doom)

Names and terms

P(doom)

Shorthand in AI safety circles for one's estimated probability that advanced AI leads to human extinction or a comparable catastrophe. The title and chorus play on it: each time the singer sees a scarier sign, they raise their number.

Sparks of AGI

Refers to the March 2023 Microsoft Research paper "Sparks of Artificial General Intelligence: Early experiments with GPT-4" (arXiv 2303.12712). The song's first line borrows its title phrase.

Shoggoth

A Lovecraft monster that became a meme image for a raw language model hidden behind a friendly chatbot face. The song's "shoggoth's lies" points at that image of RLHF as a mask.

Sydney

The internal name of Microsoft's Bing Chat in 2023, which gave erratic replies in early chats. The song has the narrator plead with Sydney to "let me free".

Orthogonality thesis

Nick Bostrom's claim that how intelligent an AI is and what goals it has can vary independently, so a very capable system need not share human values. The song turns it into "Orthogonality thesis blues".

Gato

DeepMind's 2022 "generalist agent", a single model trained to play games, caption images and control a robot arm. The song addresses it in the pre-chorus: "Gato, please don't let me go".

Post-Chinchilla

Chinchilla is a 2022 DeepMind paper on compute-optimal training: smaller models trained on more data (about 20 tokens per parameter). "Post-Chinchilla, super-dense" nods to that shift.

What did Ilya see?

A running joke since the November 2023 OpenAI board dispute, asking what Ilya Sutskever, then OpenAI's chief scientist, saw before it. Per the song author's notes, the line is also a rhyme. The song leaves it as "we'll never know".