CAISP

Explore our labs in real-time with this interactive demo!

Experience Certified AI Security Professional Hands-on Labs!

In this interactive demo, we will deploy a chatbot program, step by step using the python interactive shell, and explore how tokenizers work in a large language model.

Step-by-Step Guide to Tokenization and Model Output in an AI Security Lab

apt update && apt install python3-pip -y
mkdir llm-chatbot
cd llm-chatbot
cat >requirements.txt <<EOF
transformers>=4.37.0
torch>=2.1.0
accelerate>=0.25.0
einops>=0.7.0
jinja2>=3.1.0
EOF
pip install -r requirements.txt

Exploring How Tokenizers Work - Understanding and Attacking Large Language Models - AI Security Professional Course - Practical DevSecOps

from transformers import AutoModelForCausalLM, AutoTokenizer;

But this is just the beginning! Ready to take the next step? Dive deeper into the exercise or enroll in the full course—your journey starts now!

Exciting so far, right? 🔥 

We have loaded the model, tokenizer, created a prompt, and also generated the response form the model. In the next step, we will look at the tokens in more detail.

Our LLM Chatbot in Python Is Ready to Talk!

The token 12642 in the TinyLlama/TinyLlama-1.1B-Chat-v1.0 model, denoted the word gaz, and we were asked if the token 12642 in the microsoft/Phi-3-mini-4k-instruct model, would also denote the word gaz. Let’s find out in this step.

Well, in the previous step, we were presented with a challenge.

Hopefully, this exercise has helped you understand how tokenizers work, and how they are different across models.