If you take an interest in why — not how — the modern large language models work, this is a great 30-minute video lecture from Bertrand Serlet, former Senior Vice President of Software Engineering at Apple. This is a well-crafted, non-jargon talk to take in; at the very least, it’s a lesson in how to communicate a complex, mathematical topic in incremental, bite-size pieces without devolving into multi-layer flowcharts and equations.

I started watching because I remember Serlet from a couple of his hilarious presentations from his Apple days (watch here, then here) poking fun at Microsoft, and I’m glad I didn’t skip this one.

If, after watching the video, you would like to follow up on the topic with some technical reading, I have you covered. Transformers are a type of deep neural network, specifically trained to make predictions based on chains of previous input, using the inherent contexts and relationships therein.

So, while an image classifier takes in a single image and predicts probabilities that the image contains specific objects, a transformer takes in a sequence of information pieces, chained together, and makes a contextual prediction. The prediction in this case tries to extend the input chain, which can be the next logical word, next logical pixel, next logical musical note, etc.

If you want to take a deep dive into building a simple LLM from scratch, you may start with Andrej Karpathy’s tutorial. Karpathy is one of the co-founders of OpenAI (makers of ChatGPT), and this is a very well put-together lecture. He also has an hour-long “busy-person’s intro to LLMs”.

Finally, if you really want to go into the rabbit-hole, this paper is what started the transformer revolution: Attention is all you need. The View PDF link will take you to the PDF version of the paper. This LitMaps link will show you how influencial that paper has been.

But, seriously, forget all of the technical stuff. Go watch Bertrand’s video lecture.

P.S.: I should note, this is not an unconditional endorsement of AI in general and LLMs in particular. These technologies are being used, and may continue to be used, in ways that are unsavory, short-sighted and dangerous. We need to be circumspect and judicious in how we deploy these extremely powerful technologies, so that we don’t incur unacceptable costs in the long term. We should aim to bolster our creative crafts with AI, not foolishly attempt to replace human creativity.

From way back in 1996:

It took the European Space Agency 10 years and $7 billion to produce Ariane 5, a giant rocket capable of hurling a pair of three-ton satellites into orbit with each launch and intended to give Europe overwhelming supremacy in the commercial space business.

All it took to explode that rocket less than a minute into its maiden voyage last June, scattering fiery rubble across the mangrove swamps of French Guiana, was a small computer program trying to stuff a 64-bit number into a 16-bit space.

One bug, one crash. Of all the careless lines of code recorded in the annals of computer science, this one may stand as the most devastatingly efficient.

More links here, and the report of the inquiry into the incident is archived here.

A fascinating, and from a programmer’s perspective chilling, read. This is the stuff of nightmares — an apparently innocuous line of code causing an exception that leads to disaster!

I’ve been meaning to try and implement MathJax on this website for a while now. For including math equations on a website, MathJax is probably one of the more elegant ways to do it. I can write equations in TeX format, and MathJax renders the equations properly for you!

Finally, in the last couple of days I’ve been forced to get around to it, thanks to a new post that I’m writing that includes a little bit of math. So anyway, I just wanted to jot down that process.

The thing with MathJax is that it’s meant to, and does, work with HTML. But since I’m working with Octopress and Markdown, I have to ensure that the conversion from Markdown to HTML produces no unwanted syntactical errors for MathJax. To get around this problem, Zac Harmany (I hope I got the name right) suggests tweaking the Markdown rendering engine to Pandoc, so that the conversion works as desired. I’m sure that works great, but I had no intention of tweaking my Markdown conversion engine. Instead, I discovered a nifty ruby bundle (here) that serves a great purpose.

Next, the typical MathJax “installation” involves adding a line in the <head> section of your Octopress theme (Add it to %octopress_root%/source/_includes/custom/head.html), so that when every page loads, the necessary Javascript files are also loaded, ready to render your equations. I did not want the Javascript to execute to load along with all my pages, given that I don’t expect to have equations in all my posts. Instead, I’ve included the call to the script in the <body> of the post, i.e. in the meat of the post itself. Remember that the declaration needs to be before your first equation.

I also contemplated downloading the MathJax distribution and hosting it locally on my own server, but that did not work at all. There are way too many files to be uploaded (each file is small; the total package is ~50MB; there are too many files, though) and it just took forever to upload to my server until I just gave up. I’ll revisit that option if I think using MathJax’s own servers is not working well—which I doubt will happen.

With those details, here’s how I set things up:

• Install the verbatim.rb plugin from this Github repository. To do this, simply download the file (‘Gist’ in Github parlance) and place it in %octopress_root%/plugins. When inserting equations, there’s a syntax to using this plugin; I’ll demo it below.

• Create a MathJaxLocal.js file at %octopress_root%/source/javascripts/ to add local configurations for MathJax. Note that the last line of the code must point to the full path of the local file, in my case http://arnabocean/javascripts/MathJaxLocal.js

  Here’s what my MathJaxLocal.js looks like (I started with Zac Harmany’s file and modified to suit my needs.):

  MathJax.Hub.Config({
      jax: ["input/TeX","output/HTML-CSS"],
      extensions: ["tex2jax.js","MathMenu.js","MathZoom.js"],
      tex2jax: 
          {
              inlineMath: [ ['$','$'], ['\\(','\\)'] ],
              displayMath: [ ['$$','$$'], ['\\[','\\]'] ],
              skipTags: ["script","noscript","style","textarea","pre","code"],
              processEscapes: true
          },
      TeX:
          { 
              equationNumbers: { autoNumber: "AMS" },
              TagSide: "left",
  
          },
      "HTML-CSS": { availableFonts: ["TeX"] }
      });
      MathJax.Ajax.loadComplete("http://arnabocean.com/javascripts/MathJaxLocal.js");
  

• Declare the location of the MathJax files. The easiest thing to do is to use MathJax’s own servers. However, in addition to just their servers, you’ll have to link to your own local config file as well, so we’ll add both of these at the same time. In the main body of your markdown post (preferably after the “Read More” fold), add the following:

  <script type="text/javascript"
  src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML,http://arnabocean.com/javascripts/MathJaxLocal.js">
  </script>
  

In the above, the first link points to MathJax servers, the second points to my own config file.

And that’s basically it! Now you’re all set to write beautiful equations. Here’s a demo:

<!-- MathJax configuration -->
<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML,http://arnabocean.com/javascripts/MathJaxLocal.js">
</script>
<!-- End MathJax Configuration -->

{% raw %}{% verbatim tag:p %}{% endraw %}
\[ 
f(x)= a_0 + a_1\sin(x) + a_2\sin(2x) + ...
\]

\[  
+b_1\cos(x) + b_2\cos(2x) + ...
\]

\[
f(x)=a_0+\sum_{k=1}^\infty\big( a_k\cos(kx)+b_k\sin(kx) \big)
\]
{% raw %}{% endverbatim %}{% endraw %}

And here’s what that would look like:

There’s still a lot of more that I need to find out, and from the looks of it, Zac’s website is a great resource. I’ll add more posts if I find anything useful that I end up using.

Bessel filters are incredibly useful in numerical analysis, especially for acoustic-type waveforms. This is because analog Bessel filters are characterized by almost constant group delay over any frequency band, and this means that the shape of waves does not change when passed through such a filter.

Well, MATLAB provides some of the building blocks required to create a bandpass analog filter, but does not actually combine the pieces to make a usable filter function.

I created a function for my own research (sourced from pieces I found elsewhere, but it’s been too long—I don’t remember where I found each piece, sorry!), and can be found at my MATLAB repository, specifically, here.

Here’s the documentation that I included with the function:

besselfilter. Function to implement a bandpass Bessel Filter.

[filtData, b, a] = besselfilter(order,low,high,sampling,data)

Inputs:

    - order:      Number of poles in the filter. Scalar numeric value.
                    Eg.: 4  
    - low:        Lower frequency bound (Hz). Scalar numeric value.
                    Eg.: 50000 (= 50kHz)
    - high:       Upper frequency bound (Hz). Scalar numeric value.
                    Eg.: 1000000 (= 1MHz)
    - sampling:   Sampling frequency (Hz). Scalar numeric value.
                    Eg.: 25000000 (= 25MHz)
    - data:       Input data. Numeric vector.
                    Eg.: data vector of size (n x 1)

Output:

    - filtData:   Output filtered data. Numeric vector. 
                    Eg.: data vector of size (n x 1)
    - b, a:       Transfer function values for the filter. Scalar numeric.

I’ve been using MATLAB for quite a few years now, using it both for my own research as well for work at VTTI. Well, I decided to share some of the code that I’ve been writing, which may come in handy to others in the same field. I’ve long appreciated the help I’ve received from the larger MATLAB community, and I thought I should start contributing as well. :)

Here’s the link: https://bitbucket.org/arnabocean/public-matlab/

Update in 2020: Well, my Bitbucket repository was Mercurial based, as you will read below. Bitbucket has since moved away from Mercurial and became Git-only. So I moved to the more popular Git repo: Github. Here’s where the repository resides now: https://github.com/arnabocean/public-matlab

I’ve been using bitbucket as my code repository. I know, GitHub seems to be more popular in the tech and programming community, but by coincidence I happened to learn Mercurial (Hg) first, and Mercurial it has been for me since. Bitbucket supports both Hg and Git, so you may use either based on your preference.

To be sure, these are and will be scripts that I can share, and in some cases may be snippets of code that I think may be useful in implementing certain logic scenarios. They will almost certainly not be pathbreaking new pieces of programming. :) I’ll be glad if you find them useful in any way.