Hugging Face GPT With Clojure
A new age in Clojure has dawned. We now have interop access to any python library with libpython-clj.
Let me pause a minute to repeat.
You can now interop with ANY python library.
I know. It’s overwhelming. It took a bit for me to come to grips with it too.
Let’s take an example of something that I’ve always wanted to do and have struggled with mightly finding a way to do it in Clojure:
I want to use the latest cutting edge GPT2 code out there to generate text.
Right now, that library is Hugging Face Transformers.
Get ready. We will wrap that sweet hugging face code in Clojure parens!
The first thing you will need to do is to have python3 installed and the two libraries that we need:
- pytorch -
sudo pip3 install torch
- hugging face transformers -
sudo pip3 install transformers
Right now, some of you may not want to proceed. You might have had a bad relationship with Python in the past. It’s ok, remember that some of us had bad relationships with Java, but still lead a happy and fulfilled life with Clojure and still can enjoy it from interop. The same is true with Python. Keep an open mind.
There might be some others that don’t want to have anything to do with Python and want to keep your Clojure pure. Well, that is a valid choice. But you are missing out on what the big, vibrant, and chaotic Python Deep Learning ecosystem has to offer.
For those of you that are still along for the ride, let’s dive in.
Your deps file should have just a single extra dependency in it:
Diving Into Interop
The first thing that we need to do is require the libpython library.
1 2 3
It has a very nice
require-python syntax that we will use to load the python libraries so that we can use them in our Clojure code.
Here we are going to follow along with the OpenAI GPT-2 tutorial and translate it into interop code. The original tutorial is here
Let’s take the python side first:
1 2 3 4 5
This is going to translate in our interop code to:
py/$a function is used to call attributes on a Python object. We get the
transformers/GPTTokenizer object that we have available to use and call
from_pretrained on it with the string argument
Next in the Python tutorial is:
1 2 3 4 5 6
This is going to translate to Clojure:
1 2 3 4 5 6 7 8 9 10
Here we are again using
py/$a to call the
encode method on the text. However, when we are just calling a function, we can do so directly with
(torch/tensor [indexed-tokens]). We can even directly use vectors.
Again, you are doing this in the REPL, so you have full power for inspection and display of the python objects. It is a great interop experience - (cider even has doc information on the python functions in the minibuffer)!
The next part is to load the model itself. This will take a few minutes, since it has to download a big file from s3 and load it up.
1 2 3
The next part is to run the model with the tokens and make the predictions.
Here the code starts to diverge a tiny bit.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
The main differences is that we are obviously not using the python array syntax in our code to manipulate the lists. For example, instead of using
outputs, we are going to use
(first outputs). But, other than that, it is a pretty good match, even with the
Also note that we are not making the call to configure it with GPU. This is intentionally left out to keep things simple for people to try it out. Sometimes, GPU configuration can be a bit tricky to set up depending on your system. For this example, you definitely won’t need it since it runs fast enough on cpu. If you do want to do something more complicated later, like fine tuning, you will need to invest some time to get it set up.
Doing Longer Sequences
The next example in the tutorial goes on to cover generating longer text.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
The great thing is once we have it embedded in our code, there is no stopping. We can create a nice function:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
And finally we can generate some fun text!
1 2 3
Clojure is a dynamic, general purpose programming language, combining the approachability and interactive. It is a language that is easy to learn and use, and is easy to use for anyone
So true GPT2! So true!
libpython-clj is a really powerful tool that will allow Clojurists to better explore, leverage, and integrate Python libraries into their code.
I’ve been really impressed with it so far and I encourage you to check it out.
There is a repo with the examples out there if you want to check them out. There is also an example of doing MXNet MNIST classification there as well.