Posts on timothyckl

Tokenising FineWeb-Edu

Fri, 19 Jun 2026 10:00:00 +0800

In the previous post, I laid out the goal for this series: build a small GPT-2-style language model from scratch, train it on a subset of FineWeb-Edu, and use it as a baseline for later experiments.

The first concrete step is turning raw text into something the model can learn from. Language models do not read text directly, they operate on sequences of token IDs. So before we can train anything, we need to:

Building a Tiny GPT-2 From Scratch

Sun, 07 Jun 2026 10:53:28 +0800

In this series, I’ll build a small GPT-2-style language model from scratch, train it on a subset of the HuggingFaceFW/fineweb-edu dataset, and use it as a baseline for experiments in training, inference optimisation, architecture changes, and mechanistic interpretability.

The aim is to understand the full language modelling pipeline from end-to-end: data, tokenisation, model architecture, training, generation , evaluation, inspection.

Who should read this

This series is for developers with some programming and machine learning experience who want to understand decoder-only Transformers by building, training, modifying, and inspecting a small GPT-2- style language model.

Writing LaTeX in Neovim

Fri, 05 Sep 2025 13:15:40 +0800

Background

As a university student in the field of computer science, my workflow largely revolves around note-taking, programming, and writing math equations. I heavily rely on Neovim and its plugins as my primary development environment due to its versatility and efficiency.

Currently, I write my math notes and tutorials on an iPad, but I would prefer documents that look more professional. After some googling I discovered LaTeX and it’s superior formatting and ability to handle equations. However, one gap in my workflow was handing LaTeX documents directly within Neovim without switching to external software.