Building a transformer without training

I started by learning how a GPT works from scratch, training small models on my laptop and asking what they were actually good for. Every idea I tried, from Sigma rules to ATT&CK mapping, ran into the same question: couldn't normal code do this anyway? So I flipped it. Base64 is already done correctly by code, so could a transformer do the same thing exactly, with no training at all? I wrote one manually to find out, partly to learn, and partly because I hate waiting when the GPU is already busy training my other projects.

That became qkvc. You describe an algorithm in a short file, and its builder makes a transformer that runs it. Every weight is placed by rule and every number has a name.

The target

I'll use my own text, Hello Darpan!. First I needed the right answer, so I got it from Python:

import base64

print(base64.b64encode(b"Hello Darpan!").decode())

SGVsbG8gRGFycGFuIQ==

That's the target. The transformer has to write exactly this.

Here's how base64 gets there. It turns 3 bytes into 4 characters of 6 bits:

H         e         l
01001000  01100101  01101100
010010  000110  010101  101100
S       G       V       s

l         o         space
01101100  01101111  00100000
011011  000110  111100  100000
b       G       8       g

D         a         r
01000100  01100001  01110010
010001  000110  000101  110010
R       G       F       y

p         a         n
01110000  01100001  01101110
011100  000110  000101  101110
c       G       F       u

!
00100001
001000  010000  (pad)   (pad)
I       Q       =       =

! is only one byte, so its 8 bits are filled out with zeros to make two characters, and = marks the rest.

So how does a transformer do this with no training?

How the builder works

The builder takes a task file and works out every weight. For base64, the task file says two things that matter:

def encode_sources(n):      # which bytes output n reads
  first = 3 * (n // 4) + (0, 0, 1, 2)[n % 4]
  return first, first + 1

def encode_char(a, b, place):      # how to make the output from them
  return [a >> 2, ((a & 3) << 4) | (b >> 4), ((a & 15) << 2) | (b >> 6), a & 63][place]

Then it goes through these steps.

Step 1: lay out the stream. It writes each group's inputs, then its outputs:

ENC H e l  S G V s   ENC l o _  b G 8 g   ENC D a r  R G F y  ...

_ is the space. Every group is 8 tokens, so what an output needs is always the same distance back. That's why one set of weights works for any length.

Step 2: work out the distances. It runs encode_sources and counts how far back each byte is from the token that predicts the output:

Output in the group	First byte	Second byte
1st	2 back	1 back
2nd	3 back	2 back
3rd	3 back	2 back
4th	3 back	not needed

S is predicted from l, the token just before it, so H is 2 back and e is 1 back.

If a distance isn't the same in every group, the build stops with an error.

Step 3: name every number. Each token is 76 numbers, and each has a name and one job: bits, phase (the position in the group), mode_enc, enc_first.bits, out_bits and so on. So you can read the model's state by name.

Step 4: build the attention heads. One head for each byte an output reads. Its weights say "look at the token this many back". The query names a position and only the token there matches, so all the weight lands on it. To write S, the head lands on H.

Step 5: build the MLP. The builder doesn't read encode_char's code. It calls it with one input bit set at a time and records where each bit lands. Each result becomes a unit, a ReLU acting as an AND gate:

encode place 0: first.bit6 -> out bit 4
    fires when: mode_enc AND phase[3] AND enc_first.bits[6]

Base64 only moves bits, so this is enough. Other algorithms need more, so the builder picks the cheapest form that fits:

Form	When	Example
Wiring	each input bit lands in one output bit	base64, 54 units
Bit functions	each output bit depends on at most 4 input bits	XOR, 85 units
Lookup table	anything else, up to 4,096 units per position	ROT13, 257 units

Step 6: add the rules. "No more bytes" becomes units too. This is how = and the end get written.

Step 7: build the output layer. Each token scores by how many of its bits match the output bits, and the highest score is written.

Step 8: prove it. The proof reads the finished weights and checks each step above.

Writing S

Here's the finished model writing the first character:

output 1: S
  enc_first    reads x48 'H', 2 back
  enc_second   reads x65 'e', 1 back
  unit  encode place 0: first.bit3 -> out bit 1
  unit  encode place 0: first.bit6 -> out bit 4

H is 01001000, so bits 3 and 6 are on and exactly two units fire. That gives 010010, which is 18, which is S.

You can see this for any text:

python -m qkvc trace base64 encode "Hello Darpan!"

Running it on the whole text:

$ python -m qkvc run base64 encode "Hello Darpan!"
S G V s b G 8 g R G F y c G F u I Q = =

The same as Python's target, character for character.

So it's just a circuit?

Pretty much. It has real attention heads, an MLP and an output layer, but every weight is placed by rule. The base64 model has 5 attention heads, 54 MLP units and 70,603 parameters.

Is it a compiler?

I called it that at first. It didn't look like one, so now it's a builder. Tracr already compiles RASP programs into transformer weights. Mine takes a short Python description with no language to parse.

Does it always work?

The model does what its rules say for every valid input of any length. That's proved from the weights. The rules themselves are checked against the real algorithm: for base64, every full group of bytes, all 65,792 one- and two-byte endings, and a sample of whole inputs.

I wanted a source for each step so I wasn't guessing, and they're listed in the repo. I also planted 11 mistakes to see if the proof would catch them. It caught all 11.

What else can it do?

16 algorithms so far:

• Rearranging bits: base64, base64url, base32, hex, binary, bit reversal, byte swap
• Substitution: ROT13, Caesar, Atbash, upper/lower case
• With a key: XOR, Vigenère
• Running state: binary addition, parity, XOR checksum

Where's the limit?

Each output has to depend on a few symbols at fixed positions. I tried reversing a string and multiplying two bytes, and the builder refused both. Sorting, search and SHA-256 look out of reach too, but I haven't tried them.

It's also slower than normal code: 1 MiB takes about 40 seconds, and Python does it in milliseconds. So it isn't replacing anything.

What's next

I still wouldn't use it for base64. But now when I look at a transformer, I can see it's just a machine you can read.

It's still ongoing. More algorithms and more ways to customise the transformer are coming.

• Watch it run in your browser: the walkthrough page
• Follow every step with real values: the 10 notebooks
• The code: github.com/darpan-97/qkvc