If you take an interest in why — not how — the modern large language models work, this is a great 30-minute video lecture from Bertrand Serlet, former Senior Vice President of Software Engineering at Apple. This is a well-crafted, non-jargon talk to take in; at the very least, it’s a lesson in how to communicate a complex, mathematical topic in incremental, bite-size pieces without devolving into multi-layer flowcharts and equations.

I started watching because I remember Serlet from a couple of his hilarious presentations from his Apple days (watch here, then here) poking fun at Microsoft, and I’m glad I didn’t skip this one.

If, after watching the video, you would like to follow up on the topic with some technical reading, I have you covered. Transformers are a type of deep neural network, specifically trained to make predictions based on chains of previous input, using the inherent contexts and relationships therein.

So, while an image classifier takes in a single image and predicts probabilities that the image contains specific objects, a transformer takes in a sequence of information pieces, chained together, and makes a contextual prediction. The prediction in this case tries to extend the input chain, which can be the next logical word, next logical pixel, next logical musical note, etc.

If you want to take a deep dive into building a simple LLM from scratch, you may start with Andrej Karpathy’s tutorial. Karpathy is one of the co-founders of OpenAI (makers of ChatGPT), and this is a very well put-together lecture. He also has an hour-long “busy-person’s intro to LLMs”.

Finally, if you really want to go into the rabbit-hole, this paper is what started the transformer revolution: Attention is all you need. The View PDF link will take you to the PDF version of the paper. This LitMaps link will show you how influencial that paper has been.

But, seriously, forget all of the technical stuff. Go watch Bertrand’s video lecture.

P.S.: I should note, this is not an unconditional endorsement of AI in general and LLMs in particular. These technologies are being used, and may continue to be used, in ways that are unsavory, short-sighted and dangerous. We need to be circumspect and judicious in how we deploy these extremely powerful technologies, so that we don’t incur unacceptable costs in the long term. We should aim to bolster our creative crafts with AI, not foolishly attempt to replace human creativity.

∞

An analysis of the dagger’s blade led by Daniela Comelli, a professor of materials science at the Polytechnic University of Milan in Italy, showed that it contains 10 per cent nickel and 0.6 per cent cobalt, the researchers report in the journal Meteoritics and Planetary Science.

(Also, I really love it when a news article references the actual journal article prominently. Nothing better for the interested reader!)

∞

A few years ago, there was a competition of sorts (I don’t remember the details) where the brief was to write 500 words or less on the importance of interdisciplinary research, aimed at a “public audience”. My submission didn’t qualify for final selection, but here it is anyway. :)

Research, development and innovation plays a crucial role in human life. Not only technological and scientific advancements, but the study of the human condition, our societies, our art and culture, and what it is that makes us who we are—it all involves research, and it is all indispensable. From what makes us happy and sad, frustrated or psychotic, to how we can mimic the gecko’s foot, our research is what defines us.

The thing about education and learning is (and this becomes more applicable with more specialized training) that we tend to focus on narrow areas to become experts in. There is no other way—it is quite rare for an individual to be able to master and be experienced in diverse fields. The problem, of course, is that reality is not divided into specializations and ‘majors’, and areas of expertise.

Consider an example: designing an automobile. Does it only involve the aesthetics of how it looks? Certainly not. The designer has to consider the materials that will build the vehicle, and how strong it needs to be. More, there’s also the thought of minimizing air resistance as the vehicle speeds down the road. Even the most elementary consideration leads us in this case to the artist, the structural engineer, and the aerodynamicist. These are diverse areas of expertise, coming together to create a beautiful and functional automobile.

Consider again the psychologist who wants to improve how people manage their life’s problems. She has certain new ideas, and wants to test them out in a variety of conditions, with a variety of individuals. She makes full use of her expertise in psychology, but in addition, she must now extensively use statistics, so see if her results are meaningful! Once again, real life finds a way to bring together two completely different areas of specialization.

The best and the most productive and beneficial research has to be interdisciplinary. This has the benefit of involving individuals who are specialists in their areas, who can bring insight to a problem that no one specialist can contribute. More, the nature of a problem in one area often benefits from knowing how a similar problem is solved in another area. Most importantly, interdisciplinary research means a new body of knowledge in one field can be extended for use in another area—a benefit that wasn’t originally anticipated while doing the research.

Recently, a NASA effort to improve how jet fuels mix inside rockets when burning had an unintended consequence: the technology was used to improve firefighting equipment. Where the old system needed a few men, 220 gallons of water, and 1min 45seconds to control a fire, the new system did it in 17seconds, using 13.6 gallons of water and one fireman.

This is why research must be interdisciplinary—because you never know how the next rocket technology can save lives quicker in a fire, and conserve precious water while doing it.

I was lamenting on the scarcity of engineering blogs, even though there are a plethora of excellent science and other technical blogs on the internet.

That got me thinking about why relatively so few scientists in general, and engineers in particular, write and publish on the web. Here’s the problem, I think–

• We never receive any proper writing training throughout our careers.

  We learn the other stuff, all the theories and how they work and so on, and even how to publish our work in peer-reviewed journals, but rarely how to competently and forcefully express ourselves and communicate with the world at large. That’s a problem, isn’t it? After all, a scientist is as much a writer as anyone else—what use is my earth shattering research if I can’t explain it to everyone else?

  And no, ‘math does the talking’ is no excuse. Math isn’t for everyone, and it’s very useful to be able to communicate ideas outside of mathematical jargon. Even a brief “Here’s an idea. Now if you really want to know, go learn the math!” is extremely valuable.

  Here’s an example, via an article at Project Wordsworth: the Japanese mathematician Shinichi Mochizuki posted four papers on the internet, purporting to prove the ever-enigmatic ABC conjecture. The only problem? No one understands his work:

  The question which quickly bubbled to the top of the forum, encouraged by the community’s “upvotes,” was simple: “Can someone briefly explain the philosophy behind his work and comment on why it might be expected to shed light on questions like the ABC conjecture?” asked Andy Putman, assistant professor at Rice University. Or, in plainer words: I don’t get it. Does anyone?

  Oops! (And remember, we’re talking about the mathematics community here, not the lay public.) Dr. Mochizuki was invited to give lectures on his work, to explain and educate. He refused.

  Of course, his peers are irked:

  “You don’t get to say you’ve proved something if you haven’t explained it,” [former math professor Cathy O’Neil] says. “A proof is a social construct. If the community doesn’t understand it, you haven’t done your job.”

  If you can’t communicate, are you really a great researcher?

  Mochizuki has reported all this progress for years, but where is he going? This “inter-universal geometer,” this possible genius, may have found the key that would redefine number theory as we know it. He has, perhaps, charted a new path into the dark unknown of mathematics. But for now, his footsteps are untraceable. Wherever he is going, he seems to be travelling alone.

  This is, of course, an extreme case, but I think the larger point holds too—that engineers/scientists should be able to competently express themselves and communicate with the larger community, and not only in journal articles.

• We are not trained to be truly internet-savvy.

  I don’t mean this in terms of knowing how to navigate the internet and check email and visit websites and perform Google searches. I mean this in a larger sense—in knowing (and being comfortable with) how to create and maintain blogs, in managing our internet personas and profiles, in creating and designing websites.

  There are ample tools and resources out there, and we don’t all need to be trained in computer science to thrive—but we often rarely know how and where to begin. Some take the time to teach themselves, but what of those of us whose knack is not in internet technologies? We really do need to do more to expose ourselves more to internet publishing.

  We personally and professionally know of many scientists and researchers who are truly great teachers and communicators—but how many of these brilliant people are writing and publishing on the internet for the community at large?

If you’re an engineer or a scientist, and are a good communicator, please do consider writing and publishing on the internet! The rest of us will be the richer in experience for it. :)

I wish there were more people writing about engineering mechanics research. It’s certainly a fascinating area, and while perhaps they wouldn’t be as popular as the tech-media blogs, or the awesome science blogs that everyone can identify with, they’d still be pretty good, right?

I really like and follow Dr. Drang, who seems to occupy the perfect niche—mechanical engineering and computer programming. And through Dr. Drang I’ve recently discovered the blog of J. Ben Deaton, but haven’t had the chancce to explore in detail yet. (BTW, Deaton’s site is also powered by Octopress, with the default Octopress theme that I mentioned.) Then there’s Engineering is Awesome, which is also excellent.

But other than that, I don’t know of any engineering or mechanics blogs. There may be some great ones that don’t show up in Google searches—if you know of one, would you let me know? :)

There are quite a few science blogs though (example, example), and they are excellent and fascinating. But where are the engineers? Are engineers really that boring compared to other scientists? :)