I’m specifically interested in “neural networks” and transformer architecture and limitations of what it can and can’t learn, though. Specifically, if blinded to some number pairs, can it learn to multiply numbers it hasn’t seen?

Or with all number pairs but some pairs being more common than others.

The curious thing about learning, is that what can be learned in practice is only a minuscule fraction of what you could get if you could just guess the right weights somehow.

Hmm, I guess they could have written a lot of code lines, but its not like the code they write would be equivalent to million LOC of a coherent whole.

You can also do a lot with contractor code for RLVR, like have contractors make extremely comprehensive regression tests (simulate user clicking etc etc, compare screenshots) and then you can do a lot of training on replicating pieces of the app.

My understanding is that LRVR for mysterious reasons results in somewhat better generalization, probably because the training samples are always tailored to the model they are applied to.

I do actually have a decent GPU i bought 4 years ago in my PC, so maybe I’ll do a few simple experiments again… my preferred approach is to pick a simple but nonlinear problem e.g. binary multiplication of two large numbers, where you e.g. skew the training sample distribution or omit some number pairs and see if the resulting model can learn correct multiplication anyway.

If a reasonably small model doesn’t generalize multiply beyond seen numbers, then of course a large model isn’t gonna do the whole “soulstealing” thing that the techbros hope would allow jobstealing. And if it only generalizes beyond seen numbers if using RLVR, that bodes well for making various tools that could perhaps actually generate some value, which is very different from antiemployment shit techbros originally embarked on doing.

Regarding it being a compiler, I can get behind that: no compiler artifacts in the repo, lol. edit: in fact I’m pretty sure if they were putting compiler-generated assembly into the repository, which is awful and you shouldn’t do that, they would keep information what compiler shat them out.

I can see in some 5…10 years, deterministic coding tool that either converts specification to code or asks for clarification and then edits said specification to disambiguate. One way flow, generated artifacts kept out of the repos. That’s not gonna even begin to happen until the bubble pops though.

I wonder how much code they could have gotten contractors to write in a year or two, though. Got to be only a fraction of total open source code lines. Also, open source comes with existing tests that could be used for some validation.

If you actually exclude some project from the bulk pretraining data (or just use code that is after training cutoff) you could e.g. have it replace a function but still pass the test.

They might also be somehow using all of their customer’s usage as training data, which would explain why now there’s also a hard disk shortage.

I’m thinking RLVR and such got to be the largest, then open source, then RLHF and commissioned training data.

They also did a lot of training for decompilation, which makes me think they train on decompiled closed source as well. They definitely “anonymize” variable names etc etc to avoid getting nabbed for plagiarism, so de-compiled closed source may not be that much worse than “laundered” open source.

Seems like a bit of an overkill.

Also to be honest an AI surveillance state seems like a rather silly thing. Those ratting you out have to be your co-workers, your neighbors, your friends, better yet, your family.

Automating that out is sort of like digging out belamorkanal with a robotic digger. The canal isn’t the thing. The thing is making people dig it - and making other people decide who digs.

Oh another thing, purely anekdotally, coding LLMs still suck at multithreading.

I feel that’s one part where they could legitimately make actual “AI” for, basically a tool that either proves that nothing depends on exact timing, finds a counter example, or tells you it can’t do either (in which case your code is probably just very bad). But big overfunded AI labs probably feel that such tooling is beneath them.

By the way, his argument about “you wouldn’t mention tools” is just plain wrong.

A few years back, at work, I found a security bug introduced by a commit that said something like “fix Coverity warnings”. I ended up reviewing every commit that mentioned Coverity, because that legacy codebase had a lot of rather bad tricks that triggered Coverity to have a false positive.

And when a large number of false positives were getting resolved by editing code, sometimes that resulted in a false negative. E.g. Coverity complains about a memory leak, you end up with double free. Or vice versa, it complains about double free, and you end up with a leak. (The exact example was even more on the nose; Coverity complained about use-after-free, and the fix for it introduced a real use-after-free).

Bottom line is, they do not want to have those labels because that would enable someone to actually measure how good or bad LLMs are at code.

This is another thing that is extremely familiar to most commercial developers: there’s an LLM-powered code review bot, but unlike any tool ever before, nobody’s even trying to capture statistics whether it works or not. Why? Because whoever rolls out AI tooling isn’t sure that the tooling actually works for its stated purpose, and wants to for sure get their recognition and promotion for rolling out AI tooling, without jeopardizing it with the risk of adverse KPIs.

Besides advertising their bots as slaves, they may just be using prompts as training data (or the whole chat histories in reinforcement learning), in which case they probably would prefer you don’t swear.

I think it’s like their “it can kill everyone” thing. The sort of people who actually care about things like slavery or the risk of everyone dying, with the exception of an occasional crazy person, are not interested in believing complete bullshit.

They’re upselling their product as slavery. Apparently this works because there’s no intersection between the set of people who buy into corporate marketing bullshit like this, and the sort of people who would oppose a corporation if it did slavery.

I think if no actual living beings (or some future AI that isn’t this feedforward memoryless bullshit) are being harmed then it’s not so much doing as it is saying in the sense of making a statement, which is kind of unclear.

That’s my take as well.

I think this Anthropic fellow is too much of a psychopath to understand that there’s other reasons to ponder the question of consciousness, than to try to upsell a chatbot as a slave, and so he can’t understand that it would actually be very bad for him if a bunch of normies were told, with full weight of divine authority, that these things are conscious. I mean, maybe not civil war level bad, but worse than PETA on your ass level bad for sure.

And that is to me very disturbing. He can’t even imagine that a reaction to “this is a slave” could be anything other than “shut up and take my money”.

Well yeah its gonna likely fail to hack the power grid, but also just starting or stopping too many GPUs at once is a problem if they are grid connected, so I dunno. We damn near had a grid collapse from a freeze in Texas.

The emerging talk about “AI Chornobyl” is quite disturbing with regards to what this booga-booga gonna look like. If they don’t get enough booga booga from this stupid hacking incident, then what?

What’s really disturbing to me is that now they are talking about the power grid as a target. It’s not like their usual biblical stuff about plagues and locusts. Of course, power grid can’t be that insecure, or Iran would’ve went for it already, but they have privileged access courtesy of all the use of chatbots to run shell commands etc.

I think AI is kind of like that stupid universal socket wrench head that has a bunch of pins in it. The main skill in using something like that would be guessing when it’s not gonna work and not using it.

Yeah sloppers I know went from “see it proved new math! that totally clears us of plagiarism accusations we get when we repost that someone ai generated a frogger and a WoW clone” to “tried to use it for coding and it didn’t do a good job”.

The math stuff is mostly using Lean proof verifier and all that, by the way, the only use of LLM that they found that is actually legit because it doesn’t matter if its slopping, and you want your attempts randomized.

Anyone who would have tried the kind of bullshit he’s trying, a century ago, would have died of radium poisoning or something equally dreadful. The only reason this idiot is still alive is regulations; however he’s pretty good at risk compensating greater safety of quack treatments in 21th century with the sheer number of quack treatments he’s doing.

I’m going to prognosticate that he’ll die through some combination of the sheer brute force effort he’s putting into finding a lethal quack treatment, and the weakening of regulations and corresponding societal norms relating to quack treatments.

Fucking knew it. I follow some idiot slopper thread and they are all always going on about their prompting and absolutely none of them ever even considered measuring the effects.

Having actually done optimizations including professionally once upon a time, its so ridiculous to see people thinking they optimize a stochastic process [when they don’t even formally measure anything]. It is just gambling.

In addition to shoving absolute piece of shit “products” into everything. Prompt injectable “assistants”, worthless search overviews, shoveled out without any QA or security reviews or anything.

Take those doctor transcriotionists that make up whole sentences. Why in the fuck is there even an LLM in there? Oh it lowers word error rate? That’s because nobody expected having to benchmark bad faith efforts where coherent sentences can be completely made up, so nobody thought it necessary to disqualify for it.

There’s a slopper at the other end of it who believes that asking AI to find the best place to buy bitcoin is something that AIs are good at. Or perhaps asking AI to summarize the article and getting a penis enlargement ad as a summary.