The problem is that synthetic data is not fit for that purpose. The more of it you use, the worse at dealing with the cases LLMs get.
Think of it like this… You feed a language model a bunch of genuine human-written content. Great. Now it can produce the most likely text in a lot of cases. Word combinations that rarely appear in written language rarely get generated, so most of its synthetic data lacks those rare - but still valid - combinations.
Train it on this synthetic data, and now more outliers and rare combinations get filed off. Rinse and repeat.
The article quotes a post from ISBNdb saying the issue is model collapse from training on synthetic data.
Total collapse is a solution.
It’s also unavoidable.
Yes, what’s the issue?
That’s like saying, “they had some failure modes from the synthetic data, so they should just obviously stop trying forever.”
They’ll just fix the edge cases and move on. Like any programming task.
The problem is that synthetic data is not fit for that purpose. The more of it you use, the worse at dealing with the cases LLMs get.
Think of it like this… You feed a language model a bunch of genuine human-written content. Great. Now it can produce the most likely text in a lot of cases. Word combinations that rarely appear in written language rarely get generated, so most of its synthetic data lacks those rare - but still valid - combinations.
Train it on this synthetic data, and now more outliers and rare combinations get filed off. Rinse and repeat.