Papuş: I Tried to Teach an Imaginary Language to a Clueless 5-Million-Parameter Parrot

As you know, after nearly three years, I've started making videos regularly again. Claude Opus getting me a little too excited had something to do with it. We started with the Mathematical Stories series, then branched into other subjects. Before I knew it, I was constantly trying to produce new videos.

But along the way, I discovered a poison that hadn't been part of my life before, and that I have now managed to inflict upon myself: AI voice cloning.

Normally, I can narrate my videos myself. But I misread a sentence, record it again, my tone changes, the microphone sounds different... Add the constant urge to make another video, and naturally you start looking for an easier way.

ElevenLabs is quite good, but expensive. Cartesia was doing the job too, but even after squeezing the free allowances out of seven separate Gmail accounts, it wasn't enough anymore. 😂

So I thought: if there are open-source voice generation models, why not build my own system?

That's how the idea for AHDStudio was born.

First, I tried an open-source voice model called Chatterbox. I could run it on my own server, but there was a problem: It was slow.

Even generating a short clip took a long time. I made some progress by connecting several servers and getting them to generate two voice clips simultaneously.

But speed wasn't the real problem.

It could imitate my voice, but it didn't speak like me.

We had created a friend who sounded like me but delivered the news of someone's death with exactly the same emotionless tone as a mathematics lecture.

To fix this, we started cutting clips of my own speech from my videos. We extracted five-second samples, transcribed what I said, and separated different speaking tones.

Our aim was to teach the model not just what my voice sounds like, but how I speak.

Then we dived into Google Colab.

For anyone unfamiliar with Colab: it's an environment where Google lets you use powerful computing resources within certain limits. Some tasks that might keep your own computer busy for days can be done much faster there.

I also learned about something called LoRA.

I can hear you saying, “What the hell is he talking about now?” 😂

In the simplest terms, LoRA is a way of teaching a huge AI model a new skill by training a relatively small part of it rather than training the whole thing from scratch.

We were planning to teach the voice model my own speaking style.

However, I wasn't entirely sure about Colab's rules on voice cloning and deepfakes. Even though I would be using my own voice, I didn't want to risk having my account blocked.

So we put the voice issue aside for the time being.

We were left with all that preparation, the technical knowledge we'd acquired, and computing resources we could use.

So what were we going to do now?

What can we teach an AI?

We started talking to AI again.

Should it solve equations? Work on mathematics problems? Learn to play games?

All of those were possible, but another idea was stuck in my head.

I had a book project called Edgium, which I'd mentioned in some earlier blog posts.

In the story, a character named Aetron finds the diary of the old Aetron. The diary describes an experiment in living designed to stop people from insulting one another and draw everyone toward the same pattern of thought.

As part of that dystopian world, I'd imagined an AI-assisted language.

I'd called it the AHD language.

And I hadn't just invented a handful of words and left it at that. I'd worked out its rules, sentence structures, tenses, moods, vocabulary, and how it would be used in different situations.

In short, there was a language with no actual speakers, governed by rules I'd written myself.

Then a question occurred to me:

Instead of teaching an AI Turkish, could we teach the AHD language to a tiny AI that already knew some Turkish?

My first thought was to take an existing small language model and teach it a new language.

But after looking into the subject, things got interesting.

Why use a ready-made model when we could train our own little one?

And that's how Papuş was born.

Meet Papuş: The clueless parrot with 5.3 million brain nodes

We started with a clueless parrot whose brain had 5.3 million nodes. 😂

Technically, these aren't nodes but parameters: the numbers that can be adjusted as a model learns. And the actual count was approximately 5.22 million.

But I prefer calling it a “clueless five-million-parameter parrot” rather than a “5.22-million-parameter Transformer encoder.”

By the way, Papuş isn't a model you can sit down and chat with like ChatGPT.

When we give it a sentence, it tries to analyze the sentence's structure.

Who did what? To whom? When? Did someone have to do something, want to do it, or have the ability to do it?

That was the first thing we wanted it to learn.

Later, we would use that information to construct an AHD sentence.

But first, we had to teach Papuş Turkish.

We taught Turkish in two different ways

We prepared two separate versions of Papuş.

We showed the first one 100,000 Turkish sentences.

Its learning method was a bit like a primary-school fill-in-the-blank exercise.

For example:

“Çocuk kit_bı okudu.” (“The child read the bo_k.”)

Papuş tried to predict the missing piece.

Of course, in reality it worked with small byte units used by computers to process text, rather than individual letters.

We showed the second Papuş 500,000 Turkish sentences.

But we made its fill-in-the-blank exercise more difficult.

“Çocuk ______ okudu.” (“The child read ______.”)

Now it couldn't get any clues from inside the missing word. It had to use the rest of the sentence to predict the word.

This method is called Whole Word Masking.

In other words, we hide the entire word.

Notice that the second Papuş didn't have a bigger brain. It was roughly the same size. It had simply seen more sentences and learned through a different method.

Which one would understand better?

We started teaching Papuş AHD

Now we had two small models that had picked up some patterns of Turkish.

But they knew absolutely nothing about AHD.

For example, certain suffixes in our language had special functions:

  • -b: Obligation. Having to do something.
  • -h: Desire. Wanting to do something.
  • -ñ: Ability or permission. Being able or allowed to do something.

A small difference in meaning in Turkish could require us to use a different suffix in AHD.

For example, “I'm protecting the sword” and “I have to protect the sword” are not the same thing.

Papuş needed to understand that.

But we had a small problem.

There weren't thousands of real human conversations written in AHD.

And there couldn't be. I invented the language. 😂

So we generated Turkish sentences and their AHD equivalents according to the rules we'd created.

For example:

Turkish: Ben kılıcı korumak zorundayım. (“I have to protect the sword.”)

AHD: mi tirçit békbim.

Or:

Turkish: Biz suyu açalım. (“Let's turn on the water.”)

AHD: mil sut luqhuml.

Instead of merely placing them side by side, we decided to teach Papuş the skeleton of each sentence too.

That's called an AST.

The name sounds intimidating, but the idea is simple.

Consider this sentence:

“Ben kılıcı korumak zorundayım.” (“I have to protect the sword.”)

Here is the information we wanted Papuş to identify:

Who? Me.

What? The sword.

What action? Protecting.

Is there an obligation? Yes.

Is it negative? No.

That information is the sentence's skeleton.

If Papuş could extract the correct skeleton, the rules we'd written could turn it into an AHD sentence.

So instead of making Papuş memorize complete AHD sentences, we were first teaching it to figure out what a sentence meant.

We prepared 4,000 training examples for this.

We also set aside 500 sentences it hadn't seen before. These were the questions we'd use to check its progress during training.

We kept other tests separate as well.

Now the exam could begin.

Papuş gets a mathematical beating

When training AI, you constantly encounter a concept called loss.

You can think of loss as a mathematical penalty that measures how many mistakes the model makes.

When Papuş gives a correct answer, its penalty decreases. When it's wrong, the system calculates which settings need to change.

Then the same process repeats.

Again.

Again.

Again.

I called this process Papuş getting a mathematical beating. 😂

Of course, there is no living creature suffering here. It's just millions of numbers being adjusted over and over.

At the beginning, there were plenty of errors, just as we'd expected.

To make matters worse, our measurement code also turned out to be faulty in the first experiments.

At one point, the model seemed more successful than it really was. Then, after a correction, it suddenly looked as though it had learned almost nothing.

It turned out that the problem wasn't Papuş. It was the code grading the exam papers.

We fixed it and reran the experiments.

We trained each model not just once but from three different initializations. After all, getting lucky on a single run doesn't necessarily mean a model learns well.

And finally, the real results arrived.

Papuş trained on 100,000 sentences: 42.2%.

Papuş trained on 500,000 sentences: 46.6%.

Let me explain what these percentages mean.

Papuş had to identify every required field in a sentence's skeleton correctly. If it got the subject and verb right but the tense wrong, we counted the entire sentence as incorrect.

So 46.6% did not mean “it knows 46.6% of Turkish.”

It meant it could extract the complete skeleton correctly for 46.6% of the sentences in our synthetic exam.

That distinction matters.

But there was still something interesting going on.

A tiny five-million-parameter model had started learning some of the rules of a language I invented.

Let's give it some private tutoring

Next we looked at what Papuş was getting wrong.

It had learned some things rather well.

But it struggled particularly with distinctions such as desire, obligation, and ability.

For instance, when we say “yapmalıyım” (“I should/must do it”) in Turkish, the meaning can vary with the context. Sometimes it's a real obligation, sometimes advice, and sometimes an inference.

Papuş could confuse these distinctions.

So we prepared some special lessons for it.

We created 261 new sentence candidates. We checked them against the rules, removed ambiguous or problematic examples, and kept 218 for the experiment.

But we're not claiming that a human individually approved every one of these examples. Most were experimental data that had passed automated rule checks.

We carried out two different training runs.

We added these 218 targeted sentences to the first group.

For the second group, we repeated 218 sentences from its original training data.

In other words, one group got new private lessons, while the other redid its old homework.

It wasn't a perfectly controlled comparison, since we were comparing new information with repeated information. But it could still give us an initial idea.

And the results came in.

The 100,000-sentence Papuş improved to 46.6% with targeted training.

The control group remained at 42.2%.

Things were more interesting with the 500,000-sentence Papuş.

The model that received private tutoring scored 48.87%, while the one repeating old examples scored 49%.

So the big brother hadn't gained any clear advantage from private lessons.

One possible explanation occurred to me:

Perhaps the Papuş that had read 500,000 sentences and learned by predicting whole words had already learned certain linguistic relationships better.

The 100,000-sentence Papuş, on the other hand, was used to filling in little gaps inside words, so perhaps it benefited more from private lessons on these differences in meaning.

It was a plausible possibility.

But we didn't have conclusive evidence, because both the number of sentences and the training methods were different.

Still, I was beginning to trust Papuş a little.

Until I actually made it speak.

Papuş, what on earth are you talking about, son?

Up to this point, all we'd done was measure whether Papuş could identify the correct sentence skeleton on an exam.

Now I wanted to type a Turkish sentence and see its AHD equivalent.

So we built a small interface.

We type a Turkish sentence into a box.

Papuş analyzes it.

It extracts the skeleton.

Then the AHD rules we've prepared construct a sentence from that skeleton.

In the first tests, the 100,000-sentence Papuş started producing an answer for almost every example.

The 500,000-sentence Papuş, however, returned errors for some examples.

At first glance, the smaller one looked better.

Then we examined the sentences it was producing.

Turkish:

“Kitabı onlar kalemle yazdılar.” (“They wrote the book with a pen.”)

The 100,000-sentence Papuş answered:

val bivzit çézbikvil.

From the outside, it looked like AHD, and it was a sentence our rules accepted.

But there was one little problem.

Papuş had added a sense of obligation to the sentence.

These people had written a book, and our Papuş had decided they were forced to write it. 😂

On top of that, the information that they'd written it with a pen had disappeared.

The correct target sentence needed to preserve those details:

val bivçit çézçiv çézkivl.

That's when we realized that producing a grammatically valid sentence doesn't mean a model has translated it correctly.

Papuş could say completely wrong things in perfectly acceptable grammar.

The 500,000-sentence Papuş was making a different kind of mistake.

We gave it this sentence:

“Biz suyu açalım.” (“Let's turn on the water.”)

One part of the model had recognized, with very high probability, that this was a sentence expressing desire.

But when constructing the sentence, it failed to use the required -h suffix.

So the AHD sentence was invalid.

In other words, the kid understood what we meant but forgot the grammatical suffix when speaking.

We ended up with two hilarious characters.

100,000-sentence Papuş: It talks, even if what it says is wrong.

500,000-sentence Papuş: It might understand correctly, but sometimes it can't talk.

A proper family drama. 😂

We straightened Papuş out a little without touching its brain

At this point, we had another option.

Without retraining, could we choose the most grammatically appropriate answer from the possibilities Papuş proposed?

After all, the model doesn't consider just one answer. It assigns scores to multiple possibilities.

For example, with “Let's turn on the water,” the 500,000-sentence Papuş strongly indicated that the sentence expressed desire, yet it still failed to select the required suffix.

So we gave it this rule:

“If your answers contradict one another, use the scores you've assigned to find the most suitable combination that's valid in AHD.”

This is called constrained decoding.

Another unnecessarily fancy name. 😂

In short, we're passing Papuş's answers through a grammar filter.

And it worked.

The 500,000-sentence Papuş could now construct the sentence it had previously failed to produce:

mil sut luqhuml.

Meaning:

“Biz suyu açalım.” (“Let's turn on the water.”)

And we hadn't retrained Papuş's brain to do it. We were simply trying to make more consistent use of the predictions it already produced.

We tested this new approach on our old 500-sentence validation exam.

When we selected one particularly successful initialization for each model:

The 100,000-sentence Papuş's exact skeleton accuracy rose from 52.8% to 54%.

The 500,000-sentence Papuş's score increased from 51.4% to 52.2%.

But let's not cheat here.

These are not the averages of three runs; they are results from one selected model of each type.

Also, because we'd repeatedly developed the system using the same 500 sentences, we can't present these as scores from an independent final examination.

Most importantly, this improvement doesn't mean new knowledge was added to Papuş's brain.

It just makes slightly fewer ridiculous mistakes when using what it already knows.

Papuş still has serious problems

For example, if you type:

“Yarın kuantum fiziği laboratuvarında bilgisayarı programlayacağım.” (“Tomorrow I will program the computer in the quantum physics laboratory.”)

A small apocalypse unfolds within Papuş's understanding of the world.

Its vocabulary doesn't include things like quantum physics, laboratories, or computer programming.

But AI systems have a funny trait.

Sometimes they insist on answering even when they don't know what they're talking about.

Papuş did exactly that.

It tried to produce a sentence that looked like AHD by substituting familiar roots for words it didn't recognize.

So we added a second check to the system.

If a Turkish sentence falls outside our supported vocabulary, it shouldn't invent a translation out of thin air.

Instead, it should say, “I can't translate this sentence.”

This check isn't perfect yet either. Since it fails to recognize some inflected Turkish words, it can reject sentences that we could actually support.

And we still can't always catch sentences that are grammatically valid but wrong in meaning.

So we haven't solved all of Papuş's problems.

We've just learned to recognize some of them.

What does all of this mean?

Right now, we have two small AI models with approximately 5.2 million parameters each.

We tried to teach them basic linguistic patterns from Turkish texts.

Then we trained them to analyze sentences using a few thousand examples prepared for a completely imaginary language.

We ran experiments with different initializations, measured error rates, and examined where they struggled.

Then we actually made them speak.

And we saw them produce AHD sentences that were sometimes correct, sometimes wrong, and sometimes complete nonsense.

There's something I especially want to emphasize here.

Papuş is not currently an AI that can reliably translate Turkish into AHD.

It can extract correct sentence skeletons in certain examples. But we haven't yet shown that it can accurately translate arbitrary Turkish sentences written in real life.

Most of our measurements come from synthetic sentences generated according to our own rules.

And the language we've created doesn't have the same richness and unpredictable range of usage as natural languages.

That will require more independent tests, better data, and more rigorous evaluation.

But what really interests me is something else.

A few million adjustable numbers can learn some of the rules of a language that has no speakers.

Sometimes it correctly recognizes that a sentence expresses obligation.

Sometimes it gets confused and thinks, “There's desire here, but which suffix am I supposed to use?”

And sometimes it forces people to write a book they'd already written. 😂

I don't see any of this as evidence that Papuş thinks like a human.

I don't expect it to think like one in the first place.

What interests me is how much of the complex structure we call language can be captured by a tiny system made of mathematical operations.

And more importantly:

How do we tell what a model has actually learned when it merely looks as though it has learned something?

Perhaps that's the most important lesson this experiment taught me.

Because at one point we thought Papuş's performance had improved, but our evaluation code was wrong.

At another stage, our AI assistant wrote success rates for experiments that hadn't even been carried out yet.

We then ran the real experiments and checked the results.

Later, we discovered that the same assistant had prepared some error analyses using estimated values instead of actual model outputs.

So while we were trying to teach Papuş not to give wrong answers, we were also trying to stop the far more powerful AI systems helping us train Papuş from producing false reports.

That meant repeatedly checking files, code, results, and calculations.

Ironically, the process also taught me something about models far larger than Papuş:

An AI sounding confident doesn't mean it's right.

What happens next?

Papuş's story isn't over yet.

There are more serious exams ahead, using sentences written by real people.

We need to get it to use the AHD dictionary and grammar more accurately.

Maybe we'll make its brain a little bigger someday.

Maybe we'll compare different training methods.

Or maybe the little Papuş that read 100,000 sentences will keep beating its 500,000-sentence big brother at certain tasks.

I don't know.

And not knowing everything in advance is precisely what makes this experiment interesting.

I'm a mathematician. I've got a language I made up myself, a couple of little models, and plenty of curiosity.

As we work on them, sometimes we're running a training experiment, sometimes debating grammar, and sometimes finding a bug in the code we wrote.

And sometimes we're trying to understand why a clueless five-million-parameter parrot thinks people who wrote a book should have been forced to write it. 😂

Want to hear the funniest part?

I started this whole thing to solve the narration problem in my videos.

And I still haven't fully solved that narration problem.

But now I have two AI models trying to speak a language I invented myself.

What's more, one sometimes sees obligation where none exists, while the other forgets its suffixes even when it seems to know what it wants to say.

And I'm having a great time developing them.

I suppose that's my idea of a hobby.

Setting out to solve one problem and ending up creating three entirely different ones.

Anyway, let's train Papuş a little more.

Maybe one day it will learn to speak properly.

And if it doesn't, at least we'll have a nice mathematical story about why it couldn't. 😂

A little side note: I should probably explain where the name Papuş came from. It actually started with a video I watched while researching voice-cloning models. As the model was trained, its strange, meaningless sounds gradually began to resemble human speech. I found this hilarious and thought to myself, “They're like parrots. They're trying to talk, but they have no idea what they're saying.” 😂

Later, when I started building my own little language model, that comparison came back to me. A tiny parrot trying to speak without fully understanding what it's saying...

And that's how Papuş got its name.