inari@piefed.zip to Programming@programming.devEnglish · edit-223 days agoI built a programming language that LLMs can't useblog.picheta.meexternal-linkmessage-square32linkfedilinkarrow-up178arrow-down126
arrow-up152arrow-down1external-linkI built a programming language that LLMs can't useblog.picheta.meinari@piefed.zip to Programming@programming.devEnglish · edit-223 days agomessage-square32linkfedilink
minus-squaredan@upvote.aulinkfedilinkarrow-up50arrow-down2·23 days agoThis is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.
minus-squareOldMrFish@feddit.dklinkfedilinkarrow-up31arrow-down2·edit-223 days agoAlternative headline: Claude scores 70% on benchmark in esoteric language it hasn’t been trained on. That’s basically an ad for Claude Code…
minus-squaredan@upvote.aulinkfedilinkarrow-up6·23 days agoNot only was Claude not trained on it, but the language is actively trying to be hostile towards it.
minus-squareMirodir@discuss.tchncs.delinkfedilinkarrow-up22arrow-down1·23 days agoAnd that’s without any of this being in its training set. I wonder if they realized how big of a self own that is.
minus-squarealdhissla@beehaw.orglinkfedilinkarrow-up8·23 days agoAs well-intentioned as this might be, I doubt they understand how trained models work.
This is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.
Alternative headline: Claude scores 70% on benchmark in esoteric language it hasn’t been trained on. That’s basically an ad for Claude Code…
Not only was Claude not trained on it, but the language is actively trying to be hostile towards it.
And that’s without any of this being in its training set. I wonder if they realized how big of a self own that is.
As well-intentioned as this might be, I doubt they understand how trained models work.