If you take an interest in why — not how — the modern large language models work, this is a great 30-minute video lecture from Bertrand Serlet, former Senior Vice President of Software Engineering at Apple. This is a well-crafted, non-jargon talk to take in; at the very least, it’s a lesson in how to communicate a complex, mathematical topic in incremental, bite-size pieces without devolving into multi-layer flowcharts and equations.

I started watching because I remember Serlet from a couple of his hilarious presentations from his Apple days (watch here, then here) poking fun at Microsoft, and I’m glad I didn’t skip this one.

If, after watching the video, you would like to follow up on the topic with some technical reading, I have you covered. Transformers are a type of deep neural network, specifically trained to make predictions based on chains of previous input, using the inherent contexts and relationships therein.

So, while an image classifier takes in a single image and predicts probabilities that the image contains specific objects, a transformer takes in a sequence of information pieces, chained together, and makes a contextual prediction. The prediction in this case tries to extend the input chain, which can be the next logical word, next logical pixel, next logical musical note, etc.

If you want to take a deep dive into building a simple LLM from scratch, you may start with Andrej Karpathy’s tutorial. Karpathy is one of the co-founders of OpenAI (makers of ChatGPT), and this is a very well put-together lecture. He also has an hour-long “busy-person’s intro to LLMs”.

Finally, if you really want to go into the rabbit-hole, this paper is what started the transformer revolution: Attention is all you need. The View PDF link will take you to the PDF version of the paper. This LitMaps link will show you how influencial that paper has been.

But, seriously, forget all of the technical stuff. Go watch Bertrand’s video lecture.

P.S.: I should note, this is not an unconditional endorsement of AI in general and LLMs in particular. These technologies are being used, and may continue to be used, in ways that are unsavory, short-sighted and dangerous. We need to be circumspect and judicious in how we deploy these extremely powerful technologies, so that we don’t incur unacceptable costs in the long term. We should aim to bolster our creative crafts with AI, not foolishly attempt to replace human creativity.

∞

In light of the recent election in the UK, the Queen, of course, is supposed to make a speech regarding forming the government by the party that has won majority. Now, however, after the interesting results of the election, the Queen’s speech is delayed, and the reason for it is very interesting.

The Telegraph UK reports:

The Queen’s Speech is going to be delayed because it has to be written on goatskin paper and the ink takes days to dry.

Apparently, the British monarchy are more concerned than others would be about the archival qualities of the paper that they use.

[…] goatskin paper is not actually made from goatskin.

The material is in fact high-quality archival paper which is guaranteed to last for at least 500 years.

Well, okay, but still, why the delay?

Well, ink on this special paper takes a few days to dry. And the monarchy had “ready to go” versions of the speech for (a) a Conservative party majority, and (b) a Labour party majority. But the results of the election, that resulted in a hung parliament, has put all pre-made plans into disarray. Since the political parties themselves don’t know yet how the government will be formed, the Queen’s speech isn’t finalized yet either.

Once the details are set in stone they can be committed to the goatskin paper and sent away for binding before being presented to the Queen.

I love how even the most apparently mundane things become fascinating just by being associated with the British monarchy.

∞

From the website of the company ‘Planet’, published the same day the ISRO satellites were launched:

Today Planet successfully launched 88 Dove satellites to orbit — the largest satellite constellation ever to reach orbit. This is not just a launch (or a world record, for that matter!); for our team this is a major milestone. With these satellites in orbit, Planet will reach its Mission 1: the ability to image all of Earth’s landmass every day.

This constellation therefore formed the majority (88 of 104 satellites launched) of the payload carried by the last ISRO launch. As of this launch, Planet is operating 149 satellites in Earth orbit — this is no mean feat.

Also, an interesting side note: ISRO’s previous largest payload that I referred to in my last post — 20 satellites launched in June 2016 — also seems to be for this same company:

This is our 15th launch of Dove satellites and second aboard India’s PSLV. The launch of Flock 3p comes off the successful launch of Flock 2p on the PSLV in June 2016

∞

Indian Space Research Organisation (ISRO) scripted history today by successfully launching a record 104 satellites, including India’s earth [sic] observation satellite, on a single rocket from the spaceport in Sriharikota. This is the highest number of satellites ever launched in a single mission.

The previous record was held by Russia, with 37 satellites launched at one go. The 104 satellites include 3 of India’s own and 101 of ISRO’s international customers, including 96 from USA. (The article states ISRO’s previous record as 23 satellites launched together in June 2015, but I can’t find a record for that. The closest I could find was this: 20 satellites launched in June 2016.)

As much as this is making news, and as much as ISRO should be proud, this should come as no surprise for space enthusiasts— ISRO has been quite a force in space technology, especially using its PSLV launch system, for quite some time now.

The four stage Polar Satellite Launch Vehicle (PSLV), used for this launch, was developed by ISRO in the 1990s to launch satellites into Sun-synchronous orbits for its own remote sensing satellites. (Other than ISRO, only Russia commercially launches satellites into Sun-synchronous orbits.) PSLV was also used by ISRO for Chandrayaan 1, its lunar probe, and Mangalyaan, its Mars orbiter, becoming only the fourth space agency to reach Mars orbit.

As an aside, the Sun-synchronous orbit is a very interesting concept: it is an orbit where the satellite passes over any given point on Earth’s surface at the same local solar time. This allows the satellite to be in constant sunlight as it passes over particular regions— which is great for imaging, remote sensing, spying and weather applications. The technicalities of such an orbit are very involved and very interesting: look up the Wikipedia page I’ve linked to above.

Fun fact: due to the mechanics of the orbit, a sun-synchronous orbit is stable without external thrust only on oblate spheroid planets. This means that such orbits work on Earth and will work on Mars, but on almost spherical planets such as Venus, it will require external thrust to maintain its orbit.

∞

Last Fall, Apple included a fingerprint sensor in its latest iPhone, and called the technology ‘TouchID’. A few days ago, Samsung did the same, including its own fingerprint sensing technology in its latest Galaxy S5 phone.

The blogosphere has been aflutter about one small difference between the two launches: when Apple launched their technology, there was a huge uproar about the implications of using fingerprints as an authentication tool. As everyone has been pointing out, even a US Senator, Al Franken, issued a public email (PDF) addressed to Apple asking for clarifications and explanations regarding the technology and its implications. In contrast, Samsung’s new technology has received no such attention.

Here are my two cents on the reasons:

First, Samsung’s technology has followed Apple’s by almost half a year. In this time, users have had the chance to use the technology first hand, and have realized that the world has not, in fact, turned on its head. Thus, a similar technology from Samsung, even though its implementation is way more far reaching, has not brought any new questions or paranoia.

Second, and more importantly, Apple is a modern cultural and technological icon. There’s no way around it—anything that Apple does is subject to way more scrutiny than any of its competitors. And this is because—even if subconsciously—we have come to expect greatness, caution and prudence from this company and its products. Hence, for example, the attention to Apple’s Chinese production lines, even though every other technology company uses the same companies, and the attention to Apple’s tax schemes, even though every other company does the same—perfectly legal—thing.

This is how it should be.

Apple has shown itself to be the leader of the pack, the pioneer of modern technology; the company to follow, imitate and plagiarize from. Apple has similarly shown leadership and candor in other matters: when scrutinized about their tax practices, they actually suggested tax reforms; when a bill to solidify employee non-discrimination was on the table, Apple endorsed it, and their CEO Tim Cook personally talked about it. And, oh, even without a law being present, Apple of course already had in-house rules of similar effect, as a matter of principle; it simply would not do for it to be otherwise.

With great power, as they say, comes great responsibility, and this extra scrutiny, the extra attenion and paranoia, is the flip side to their influence and strength—their curse along with their blessing, if you will.

May Apple carry this curse with pride and with distinction, for times to come.