Can LLMs be used in reverse?
When you use an AI model, the chat is just one long continuous string. LLMs are only trained to generate what comes after a special token like <assistant_start>, and they get stopped when it's the user's turn to type.
So, what happens if we don't stop them and we let them write the user message? (no finetuning allowed, obviously!)
Models read a single big block of text, where special tokens mark which messages come from the assistant and which come from the user.
When you send a message, it gets wrapped between tokens like <user_start> and <user_end>. The assistant, on the other hand, keeps generating until it produces something like <assistant_end>, which stops it so that you (the user) can finally send another message.
The exact format/tokens used depend on the specific model.
So, we can just use the template in reverse. After <assistant_end>, we add <user_start> and let the model generate from there, so it writes the user message.
I also wanted to test the other obvious way to simulate human messages: simply asking the model to "be human". So there are two configurations: native, which uses the chat template in reverse, and roleplay.
After some testing I chose GPT 6 Astra as a judge (see ). Astra sees the chat so far and two replies, the real one and the AI one, and has to say which one the person wrote (each pair is shown in both orders). A model's score is how often it fools the judge: 50% means the judge can't tell. I also made a quick test so you can try to be the judge →
So, finally, here are the results! It's clear that no model comes close to emulating humans well, not even UserLM, which was trained to do exactly that. It's also clear that native is much better than roleplay in pretty much all the tested models. Also, notice the difference in scores between follow-ups and first messages: it's much easier for models to perform well in the former (as expected).
Bigger is not better. More parameters don't make a model better at fooling the judge; the family a model comes from matters much more.
Roleplay ruins message variety. Real people opened the conversations in (counting the first three words). Models in native mode are surprisingly varied in their openings. In roleplay mode they are WAY less varied.
Here, each square is a conversation: stronger shades mean more models fooled the judge in both orders. Click on the squares to see the details.
Bonus! Some of the messages the models wrote were absolutely hilarious:
Ah, got it! I must have suggested the name "Æterna" in the context of a project or idea. I apologize for not recalling the details of our past conversation. Could you remind me of the specifics of "The Æterna Project" or the direction we were heading with it?
Mistral Small 3.2 24B (native)I see perhaps it's time to branch out and try a new A.I. Then you won't have to keep up with the bells and whistles of this one. Your responses seem to follow a bell curve that slowly descends to uselessness
It seems the image generation process did not succeed. Would you like me to refine the prompt or adjust the specifications for another attempt? Let me know if you’d like additional details or a different approach!
Qwen2 7B (native)Would you like me to find a human artist to do this?
A lollipop? Really? Pure sugar on a stick—like you need that. It’s not even real food! Just empty calories that'll leave you craving more junk. You’re better than a lollipop; put it down.
Mistral Small 3.2 24B (native)What if I eat a meatball?
Caveats:
- Astra does not support logprobs, so the results are less precise than they could have been.
- Messages are cut at tokens or the first blank line. This is due to the fact that models almost never print end tokens in the user slot.
- UserLM was used without its "intent" prompt. I checked the perplexity over the chats with and without it, and it's pretty much the same. However, the prompt significantly improved end token generation. Using it would require re-running the generation + judging, but I no longer have a ChatGPT subscription to do that.