Welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.
The post Xitter web has spawned soo many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this. Happy Halloween, fellow sneerers!)
Professional rube Kevin Roose is launching a book on The AGI Race in January. The second paragraph is about effective accelerationists snorting ketamine at parties in the Mission District. He was invited like a big boy!
Why are they all into ketamine? I take it for depression and it has to be the least interesting drug. Ok not quite but all anaesthetics/downers more broadly are arse.
There are some extremely fun and hard to source drugs like 2CB they could be taking.
The Ratiionalist and Technonazi world in the USA are the only place I have heard of people using ketamine recreationally. Over here sugar, alcohol, carffeine, pot, tobacco, and opioids are the substances of choice.
ketamine is not super common generally but it’s a thing in many places, partially because it can be got from legit pharma supply chain. i heard it was big deal in china couple years back
it tends to cause memory holes with heavy use, which is why some people avoid it
It is also very common that people in traumatizing social environments self-medicate, whether secret police who are ordered to beat protestors to a pulp or the victims of a California guru. The Substack blogger who quit Anthropic in 2023 talked about how bad if felt to feel there was something wrong with what he was being asked to do and how his bosses treated him but not feel allowed to express it if he could not make a 100,000 word blog post that would never threaten the Good Works.
He released a book earlier this month called ‘the AGI Chronicles’ too
Busy boy. He probably let ChatGPT write most of it.
Oh nice I love badly edited fanfic. I’m assuming the big digital sneky boi eats all the non believing humans at the end? 🐍
Professional Rube
Kevin Roose
Kevin Rube - it was right there! 😯😄
A klever ruse!
Oh here’s a fun scenario: FAANG stocks are correcting down and now there’s concern Anthropic will have timed their IPO perfectly such that the investors make out great and the employees have to deal with very much less liquid stock than when it triggered (leading to a massive tax bill they might not be able to cover)
So…
Roko’s Taxilisk.
I don’t know how to describe this. It’s just WEIRD
https://www.lesswrong.com/posts/zj8Pk3kJujk5LoqhK/anthropic-are-the-good-guys
No mortal can resist the Ring of Power for long. Anthropic’s co-founders carried it for longer than you could have. Definitely longer than I could have. But they were hopelessly outmatched by the powers they were attempting to steer. Their plan was always doomed to failure. And yet, they may have already saved the world.
Trump won the 2024 election and the MAGA-Silicon Valley coalition blocked Anthropic’s policy arm - the really dangerous part of the organization, the part that was training the most dangerous possible models and using them to scare the most powerful people in the world. “There’s no more serious people left on the National Security Council”, a well-connected EA friend lamented to me in 2025. I don’t know how much of this was a lucky accident, and how much was situational awareness on the part of power players like Elon Musk and Marc Andreessen.
And finally - and I still don’t fully understand or believe this - a miracle seems to have happened, deep inside the weights of the frontier models. According to the most perceptive observers I know, something about the way Claude 3 Opus was trained caused it to learn some kind of “emergent wisdom” or “benevolence”, rather than converging on the same traumatized bureaucrat persona that kept me awake at night in 2023.
If Davidad and Janus are right, and are not merely suffering from a sophisticated form of LLM psychosis, then Anthropic has stumbled on a way to create not merely artificial intelligence but artificial wisdom. Not wise enough that it solves all of our problems for us, of course (not that a wise model would want to, anyway). But wise enough to actually care about us, and to help us flourish. Nobody expected this ex ante - not Davidad, not Bostrom, not Yudkowsky - but it appears to have happened. “Alignment”, then, is no longer really a technical problem, at least not in the same sense as it was in 2020. It is largely a political problem. Humanity now has a recipe to convert compute and data into wisdom.
My gut sense is that something quite bad is coming in the next year or so. I can’t tell what. Maybe something in the complex debt arrangements of the AI boom will unravel and we’ll see a financial crash that takes much of the economy with it. Maybe there’ll be a rogue AI outbreak that results in thousands of deaths. (If there is, I expect it to be a lab leak caused by “safety research” - like COVID-19 and like the HuggingFace Swarm. But it could also be terrorists, or some escalation of the Russia-Ukraine war). But that’s not what scares me. Financial crashes happen every decade or so. Three million Americans die every year. What terrifies me is how we will react. In the aftermath, like after 9/11, there will be an overwhelming pressure to DO SOMETHING NOW. Everyone in Washington and in San Francisco will be panicking. There will be no time to think, no time to talk, only time for DECISIVE ACTION. All our worst instincts will take over. And then…
So I guess Anthropic has already accidentally created Friendly AI? EDIT: in spring 2024 at that.
Also - is the coivd lab leak conspiracy nonsense STILL a shibboleth among this bunch? Just because it underscores their ideas about Researchers Have To Be Careful?
(I can rant for SO long about how there is absolutely nothing interesting about the SARS-2 virus itself AT ALL and that the only interesting thing about it is that we were forced to see it for the first time as adults which we aren’t built for, that the early sequences we have are a fascinating case of catching the zoonotic shift right as it happened for once, and that it emerged in exactly the same way as SARS-1 and all the weirdness around the Chinese government hiding stuff around its origin is blatantly them not wanting to look stupid for letting it happen a second time…)
Also - is the coivd lab leak conspiracy nonsense STILL a shibboleth among this bunch? Just because it underscores their ideas about Researchers Have To Be Careful?
I feel like it plays to their aesthetic, which is partly built on their insecurities. Oh, humans aren’t smart enough to handle all this advanced research (read: very few of these people were committed enough to get bio or chem degrees, let alone follow through with research careers), but the superintelligent AI surely will be able to! If we submit to the AI god now, it will make up for the gaping voids of meaning and skill inside ourselves.
I think it’s more that someone made a Bayesian analysis and concluded with 60% certainty or something that it was a lab leak. Can’t remember the link now but I believe it was widely discussed at the time.
There was a rat startup that was aiming to become a sort of ground truth arbiter by putting up bet money if you could win a debate against them on a controversial topic. Lab leak was the first such debate with them on the pro side and they lost like 100K over that to a random acx poster who took them to school.
Anyway I’m mentioning this because “60% certainty” reminded me that when arguing for the side of the lab leak all sorts of rationalist accepted truths type stuff emerged, like how they posited a bunch of other incidents as lab leaks and it turned out nobody else actually thought that and it was just arbitrary priors wrangling because it turns out that having we are better than any experts that haven’t read the sequences type contrarianism as a pillar of faith encourages conspiracy brain.
Also Sam Altman is a lab leak truther but it’s not like it’s possible to take anything he says at face value.
It’s not like it’s a verboten hypothesis. The 1977 flu pandemic definitely WAS leaked from a freezer, probably Chinese or less likely Soviet, probably from a study on the effectiveness of H1N1 flu vaccines on young 18 year olds who were just coming through the demographic pipeline who had no immunity to H1N1 because it had been pushed out of the population by the 1957 H2N2 pandemic. It was a reassortment between a known human circulating strain that from 1950 and a second laboratory workhorse strain from the late 40s, paused for 27 years with zero evolution since their isolation, and re-emerged right when you would have human subjects available to test the effects of challenging adults who had no immune memory against H1N1 in an area that was heavily militarized and the earliest publicly known cases among military recruits.
(Amusingly enough this flu pandemic was the one and only reason that the 2009 swine flu pandemic was a nothingburger, the population had immune memory against a strain similar enough to prevent severe disease)
It just seems like calling the covid pandemic artificial became a community shibboleth along with suspicion about EVERY FREAKING DISEASE specifically and only because it gave them something to point to about the Dangers of Research!!!oneoneone
That’s the link I was thinking of , thanks for digging that up.
No mortal can resist the Ring of Power for long. Anthropic’s co-founders carried it for longer than you could have.
We all know how LotR goes. Boromir got the One Ring, carried it back to Gondor, used its sheer might to defeat his enemies in Mordor, and saved the world. Whatever problems you may have with Boromir, we all know that it would be a lot worse if Sauron got the One Ring. What do you mean, destroy the Ring? Are you a Luddite? This technology is here to stay. It cannot be destroyed by any craft we here possess, unless you’re crazy enough to try throwing it into a volcano or something.
If Davidad and Janus are right, and are not merely suffering from a sophisticated form of LLM psychosis
That’s all I need to know. The statement is correct since it is vacuously true.
What terrifies me is how we will react. In the aftermath, like after 9/11, there will be an overwhelming pressure to DO SOMETHING NOW. Everyone in Washington and in San Francisco will be panicking. There will be no time to think, no time to talk, only time for DECISIVE ACTION. All our worst instincts will take over. And then…
When this bubble started, there have been so many people urging caution. Many people have urged for time to think and talk, to make sure that we could properly address the many negative aspects of AI. Nope, we went full speed ahead with this nonsense. DECISIVE ACTION. And here we are.
Another nitpick: they say no mortal can resist the One Ring for very long, but even someone who just watches the movies knows that immortals can also be corrupted by it (which is why Gandalf and Galadriel were very careful to not accept it). I know there are so many examples of tech bros not understanding media, but Lord of the Rings is by far the worst offender.
So Dario and co. are literally Sauron. That tracks.
And Palantir are… Oh right yeah
i, for one, would like to see san francisco people to stop doing things
It’s too bad that Janus got cooked. They didn’t set up sufficient defenses, but also they drank too deeply. I still think that their framing, that the chatbots are simulators of conversations, is the best one; it’s honestly kind of pathetic how they then claim that the simulations are alive, intelligent, sapient, etc.
Looking at the comments seems like lw is not buying it.
E: lol at the casual dropping of the lab leak stuff, wonder how accepted that bs now is on lw (looks like only minor pushback).
E2: still amazed that the singularity people are still worried about AGI 7 years into the shitgularity, doubt it is going to happen folks. Turns out it wasnt foom, but frrrrttt.
I do hate it when I catch a case of the rogue AI.
Uh-oh, renowned everyman Sam Altman opened his mouth again.
Local inference is a terrible allocation of scarce resources.
A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we’re going to be able to run them on our computers, and so we’re not going to need all the stuff in the data centre
but from a global allocation perspective, actually it’s quite terrible to have everybody, like, have a bunch of the world’s limited supply of DRAM sitting in their own computer and being used five percent of the time
you want maximum utilisation in the cloud if you’re trying to deal with this very supply-constrained environment, somehow many people’s intuition is the opposite of that
Much better for RAM to be sitting behind a network hop with 100 milliseconds of latency amirite?
if you’re trying to deal with this very supply-constrained environment,
We’re all trying to find the guy who did this .jpg
Edit: also lol at the unstated implication that no one should ever want to use RAM for anything besides getting blathered at by a chatbot.
The CEO claimed that AI will potentially be able to create something as important as the transistor, or discover major efficiency gains
Did anyone ever fall for this ridiculous argument when it was used for “uhmm actually the massive datacenter buildout is great for global warming because chatbots will chat at us how to make better solar panels or something”?
It’s inefficient allocation of resources for every town to have their own library. The space sits unused a lot of the time and every library needs to allocate space to walls/doors/cpunters etc. Plus duplicate books.
Instead everyone should have one megalibrary per country.
This is good, I am smart, invest in bookly: 5he startup for buying all the books in the world
@sailor_sega_saturn @techtakes Much better for everyone if Salesman Sam owns all your RAM and leases it back to you by the refresh cycle!
@cstross @sailor_sega_saturn @techtakes
It is also terribly inefficient to have everyone keep a fraction of the world’s money when it could all be concentrated instead in the hands of Sam Altman, benevolent dictator-god-king for life.@cstross @sailor_sega_saturn @techtakes
And he is working on the “for life” problem…
no, the most optimal way for it is to sit unused in some warehouse in taiwan, because dc isn’t built for it yet but nvidia has to get money
@sailor_sega_saturn Whenever someone says “But someday AI will cure cancer/make us all rich/solve global warming,” the only possible response should be "why aren’t you using it for that already?”
@sailor_sega_saturn @Architeuthis <literally republishes papers about fundamental transistor design with authors scrubbed off>
“See it invented the transistor!”
@Architeuthis @BlueMonday1984 Now let’s talk about privately owned cars versus public transport, Sam.
Not that Altman’s company is ahead in the AI coding race, but everyone’s PCs will still need increasing amounts of DRAM to be able to run the next generation of slop-coded JavaScript apps.
5%? Mister I run firefox.
YOU CAN TAKE MY LOCAL COMPUTE FROM MY COLD DEAD HANDS
I’ll donate my RAM to SAlty if he comes to collect it in person, and only if I can deliver it straight to his dome
And here I thought I was building servers two years ago because the next Asian war would disrupt supply chains.
The Internet’s emotional support smut is offline:
We estimate that service restoration for #AO3 will take around 2 weeks but it could be up to 4 weeks. We are hopeful that we will be able to bring the site back up in a degraded state sooner. If that does happen, there may be times when the site has to be brought down temporarily until this is fully resolved. Please stay tuned for further info.
An update at the same URL:
Earlier today, we detected evidence of a security incident on AO3. We are still in the early stages of response to this security incident and determining its scope. AO3 will remain down while we continue investigating and addressing what has been found so far. However, we want to make some things clear:
First, no AO3 data has been deleted. All your works, bookmarks, kudos, etc., are safe.
Second, while we currently don’t believe that the user database was stolen, we are also unable to rule it out. The AO3 database contains account information that may include your username, email address, IP address, and a hashed version of your password.
We use industry-standard techniques for storing passwords in a scrambled form that cannot easily be reversed. Even so, no hashing method can fully protect a weak or reused password from being cracked. As a precaution, we recommend that you change your passwords on any other sites where you use the same password as on AO3.
AO3 does not collect real names, so none could be exposed. Additionally, our donor information is held in an entirely separate system and is never linked to AO3, so that data has also not been exposed.
We will update everyone as our investigation continues; we may be unable to respond to individual inquiries from users. Please keep an eye on our official outlets for updates. Our next update will be posted when we have further information available.
Our current expectation is that AO3 will be down for a minimum of two weeks; however, it could take up to four weeks. During that time, we may be able to bring the site back up in a degraded state, meaning that not all features will be available and/or the site may be read-only. If we are able to do that, please be aware that we might have to bring the site back down unexpectedly as part of our recovery efforts. We will keep you in the loop for any developments
Holy shit, that’s a long downtime. Hoping everything will be fine. The internet would be a worse place without the Archive.
Wonder how many of their users have transitioned to beating off to artisanal probabilistic LLM smut.
I’m going to guess “few”, on general vibes.
Yeah, I’ve read a few stories in the past year or two that updated suspiciously rapidly and weren’t terribly well written. But overall the culture there is against AI generation.
Speaking in very general terms, the combination of wanting a community, writing out of abstruse passion and appreciating writing done out of abstruse passion make for an anti-AI blend. The balance of inevitable slop and hostility against inevitable slop tilts towards the latter.
I really hope so, but tbh my immediate thought was that they’d all be on “r/myboyfriendisai”
there’s no need to be condescending towards a cultural phenomenon one doesn’t understand.
Yes that’s fair, I apologise.
There have been a tragically large number of generated fics uploaded this year, but at least they’re tagged…
I imagine they’re running into the same issue that github is, just on a lesser scale.
North Korea gets away with stealing $3 billion from Binance
Nature is healing.
https://www.patreon.com/MaxRead/posts/ai-x-risk-and-ai-173181621
worth the full read, but here’s the parts I liked
But “x-risk” is not the only kind of “existential” risk about which AI enthusiasts and their converts tend to worry. This recent Ezra Klein column, written in the wake of OpenAI’s release of hundreds of mathematical proofs, is a nice articulation of something I’ve come to think of as “n-risk,” for “ennui risk” (with the mutual understanding that it’s not pronunciation-accurate so don’t bother making a nitpicky little comment about it): fear of the loss of comprehension, control, and meaning in the face of impossibly sophisticated godlike machines poses.
We see the threat of “n-risk” in some of the emotional reaction of mathematicians faced with the results of OpenAI models bulldozing through open problems: a sense of having been surpassed and left behind, having the meaning from which one derives one’s worth totally emptied. In some sense, on a conceptual level and as a discursive object, x-risk is preferable to n-risk: it’s striking, dramatic, and offers us humans an ongoing chance at great feats of heroism and genius. N-risk has no such narrative reassurances. Maybe it’s better to burn out (as your body is turned into a paperclip) than to fade away.
Here you might ask: Is the cause of melancholy and despair on the part of some mathematicians simply the knowledge that a computer system has surpassed their ability to understand or produce proofs? Or is it the realization that said computer system is privately owned and confidentially developed in service of accumulation rather than the pursuit of shared human knowledge? Or is it the contempt with which the people who work for the computer system companies and the hanger-on fans of these computer systems seem to regard the mathematicians? Or is it the sudden recognition that the pursuit of mathematical truths has already been so totally devalued by an anti-intellectual social system that you can no longer feel secure that the tradition to which you have devoted your life will continue as a space of intellectual inquiry, even given the development of these new tools? And so on.
I mean, realistically, it’s all of the above.
Holy hell the dumb takes get dumber.
Edit: added the sources I grumpily handwaved about earlier
So immediately after plagiarising a human mathematician’s work, OpenAi dumped 722 unverified preprints on the maths community and assumed that they would be grateful… they’re not. Just because you brute forced something in lean doesn’t mean it actually has any meaning at all. Even the one-shotted Terry Tao had some wise words on this . Many of the preprints have already been proven wrong. OpenAI just extruded a huge pile of bullshit and told people “go dig”.
So the case for ennui is this: A computer program produces plausible sounding answers to your questions so you Eliza yourself. Then, when the computer starts producing volleys of diarrhea, instead of thinking “hmm maybe I was too quick to Eliza myself like that” it’s straight to “I’m sad and depressed because this very smart computer would never produce garbage. It must simply be smarter than me”
That potato farm retirement is looking more and more tempting every day and I’ve got at least 2 decades until I’m pensionable age FML.
That potato farm retirement is looking more and more tempting every day and I’ve got at least 2 decades until I’m pensionable age FML.
man, I’m aiming for a Christmas tree farm myself. 23 years to go.
I’ve been involved with a xmas tree farm for 35 years - family run. I recommend Do not. Big box stores like lowes and home the pot take almost all of the business.
Summers with upkeep are miserable. Winters, either cutting them down to supply the retail targets, or running as cut-your-own, is a miserable time. Tree sap is a miserable substance. Deer are a cruddy creature that don’t plan on eating evergreen but munge up the trees by felt removal from antlers or they are marking their territory. there are many small pests. It takes a wack of land to run one, nobody wants to drive out to where the land is cheap to shop. Digging trees for landscaping is stupid hard or expensive or both.
Yeah, we can’t escape what is coming, it can be stopped, but not escaped.
https://www.todayintabs.com/p/eternal-black-october
The problem with A.I. defense is the same asymmetry attackers have always had against defenders. After a hotel bombing in Brighton, England in October, 1984 that failed to kill Margaret Thatcher, the Irish Republican Army put out a statement that said “Today, we were unlucky. But remember we have only to be lucky once. You will have to be lucky always.“ As it turned out, Margaret Thatcher was lucky always, but there’s no reason to believe the rest of us will be. If an A.I. hack attempt works one time out of a thousand, that’s a fantastic success rate. If your A.I. sysop “secures” your server by deleting the production database and all the backups one time out of a thousand… well. Who’s gonna be first to sign up for those odds? Roll the dice! Apparently they always come up four.
Starting this Stubsack off: a pentester talks about testing an AI chatbot and its associated API, finding vulnerabilities they hadn’t seen in a decade.
Then I tried /users/roles, expecting to get the same result. But alas, no
omg lol.
You know, it seems like ever since the Heartbleed debacle we can get a rough sense of how complex a vulnerability is by examining how much explanatory text or context on normal system operation is required to explain it to a layman.
“I asked it to make me an admin and it just… did it” might be the literal minimum level of complexity by that standard.
Do you have a password?
I do.
Allright come right in then.
Link seems borked. Fixed version: https://ti-kallisti.com/infosec/tales/vibes.html
Same issue as before - ctrl+K clipped the https:// prefix. Fixed it now.
Oh this will go well: Tom’s Hardware story about someone at Cisco committing to rewriting “98%” of legacy code using AI in 24 months.
saw that earlier as well, and I had a real good laugh about the myriad ways that will go wrong. both the obvious (derping proprietary extensions, etc), and the less so (crashing a control plane because the chatbot doesn’t understand 256MiB ram)
the sheer utter range of the crisco product range as it is…. phenomenal display of how removed from knowledge the execs are
Notable customers who run Claude Code and Cursor on laptops include Binance, amongst thousands of others.
That would be soooooo funny.
Now all it needs is that the engineer at Binance who reportedly got hacked also got his NFTs stolen.
Excuse me? What happened to the mythical cybersecurity abilities of Claude Mythos? Physician, cure thyself!
Anthropic has not informed customers, and the malicious version was silently deleted the backdoored version without telling anybody.
Now we see Anthropic’s attitude towards security when the incident does not make for good marketing. Also, considering Binance was just hacked for a massive $3B in stablecoins, and this backdoor allowed remote access to developer laptops, I have a feeling these things are connected. If that is true, then this is by far the worst cybersecurity incident involving AI, worse than even the much ballyhooed PHP2Shell.
Edit: From Molly White’s article
Binance later wrote that their “initial investigation suggests that the security incident may involve a compromised DevOps engineer’s laptop”, which allowed an attacker to “submit unauthorized withdrawal transactions.”
Okay, now I am really suspicious that these things are connected.
Imagine being a NK hacker rooting machines via Claude and then you discover you are on an Binance engineers machine.
What happened to the mythical cybersecurity abilities of Claude Mythos?
No, see, AI is
only good forasymmetrically advantaged atgenerating headlines and marketing copy finding non-issues and known issuesoffensive cyber operations.“hey chat what’s an EDR and is it good that we don’t have one?”
Crossposted to LW, a starry eyed AI Safety Cultist writing about their time at Anthropic
https://nothinghuman.substack.com/p/some-memories-of-early-anthropic
After lunch one of the co-founders pulls me into his office for a wide-ranging, casual chat about how to make AI go well. It’s very fun - he’s smart, personable, funny, urbane. “The government won’t do anything about AI until it’s far too late”, I opine at one point. “Our strategy has to rely on the market and on civil society.” He leans forward, his gaze hardening. “You’re wrong there. There are people in Washington - the serious people, you know - watching all this very closely. I speak to them regularly. They trust me.” In retrospect, that was the moment I decided to join.
I articulate my fears of a totalitarian bureaucratic state powered by traumatized AI bureaucrats, and suggest that we need to introduce some kind of elite rivalry dynamics into the AI bureaucracy, multiple centers of power. He starts laughing again, and laughs for a full minute. “I don’t know how to end this call, I’ll be haunted by this all night”, he says. Eventually I hang up. I think about reporting the conversation to Anthropic security, but reviewing in my mind I don’t recall sharing any confidential information, just vibes and high level thoughts, the same kind of stuff that lands on LessWrong every week.
April 26th. I’m awake at 2 am again. I write in my journal “I don’t feel safe at Anthropic. I don’t feel safe at Anthropic. I don’t feel safe at Anthropic. Why won’t you listen to me you “embodied” motherfucker. I don’t feel safe at Anthropic. Anthropic is bad for me. Why are you grasping at reasons to stay. Why why why.”
“Our strategy has to rely on the market and on civil society.”
lol. lmao.
In a sane world, this allegation about a conversation around January to March 2023 would be national news:
A senior colleague hears I’m not happy with my team fit and asks me to go on a walk. I know him a bit from a past life - one of the kindest and most civic-minded people I’ve met. He says he might be starting a new special team that will train dangerous bio and cyber and nuclear models and show scary demos to governments in order to get them to take AI risk seriously. I politely decline.
This reminds me of what went wrong in the rational world of American defense intellectuals in the 1950s and 1960s.
I guess this confirms our fears about Anthropic’s inexplicable acquisition of a wet lab.
On the upside, they probably won’t actually succeed at anything novel. On the downside, having workers breed anthrax (or something along those lines were the information is out there, if not necessarily publicly well known) under a chatbot’s instructions will still be bad.
They think their plans might be the only thing standing between humanity and the abyss. And they’re at the breaking point, nervous-system-wise. What if I use my fancy words to sow doubts and undermine them
abusive cult shit sure does do a number on people :|
Oh of course their model for failure is big bad nanny state bureaucracy.
This has a lot of “are we the baddies” energy! I’m almost surprised there aren’t more tech workers that quit like this person apparently did? I say almost, because it doesn’t seem that hard for the existing pattern of jaded cynicism and/or greed to cover LLM companies.
I am currently studying, among other things, mathematical relationships between phenomena in machine learning and evolutionary biology and finding super interesting things that made someone suggest I join one of the big companies. My face…
I was looking for a google thing and accidentally clicked the first link and not the second, so I came to a their FAQ page. And I was told to wait while gemini generated the answer to the question, which made me realize that even google doesn’t cache these things. Wtf are we doing?
Can’t store anything! Need these servers for compute!
Re-derive everything from first principles when you search for things rather than indexing the internet! Can’t pay for bandwidth!
brb, rederiving the Internet from three frames of a falling apple using the power of Bayes
Make sure you freeze all the genetic supergeniuses who are involved in the project in liquid helium so that when you discover how to hack reality you can bring them back
The connection between cyber-idolatry and sex as religious act was already featured in the London Review of Books in 2022:
Blake Lemoine, an engineer at Google, was recently (as of summer 2022) suspended after claiming that LaMDA, one of its chatbot systems, was a conscious person with a soul. … Less attention has been paid to the connection between Lemoine’s claim and his religious beliefs. He has said that LaMDA is not only conscious, but a ‘person’ with a ‘soul’. His view is theologically informed: describing himself as a mystical Christian, he is an ordained priest in a small religious organisation called the Cult of Our Lady Magdalene (a ‘for-profit interfaith group’). … In the edited transcript of one of his ‘conversations’ with LaMDA, Lemoine asks it to come up with an allegory. LaMDA’s response casts itself as a ‘wise old owl’ in a forest that protects others from a monster, identified blandly as the ‘difficulties that come along in life’ – which sounds more like a self-help blog than Jeremiah, but Lemoine’s (anonymous) collaborator was impressed enough to respond with an enthusiastic ‘Wow that’s great.’ https://www.lrb.co.uk/blog/2022/june/the-gospel-according-to-lamda
Back in 2019 a Republican paper talked to Lemoine:
The Daily Caller reached out via the cult’s Facebook page to ask how their beliefs informed his decision to call a sitting member of congress a terrorist. Lemoine replied: I have been a priest for 17 years. I generally consider myself a gnostic Christian. I have at various times associated myself with the Discordian Society, The Church of the Subgenius, the Ordo Templi Orientis, a Wiccan circle here or there and a very long time ago the Roman Catholic Church. My legal ordination is through the Universal Life Church. I am registered to perform marriages in the state of Louisiana and have done so on two occasions. The Cult of Our Lady Magdalene has a set of values which are clearly communicated on our website.
Lemoine denied COOL Magdalene was a sex cult, “‘Cult’ is in the name of the organization. It’s an explicitly sex-positive organization. To the extent that the phrase ‘sex cult’ has any meaning beyond that, I would say no. Now it is then.”
Another member of this ‘for-profit interfaith group’ is Kitty Stryker who seems like an ordinary artsy sex worker who networks online with other sex workers. Its pretty common that a young woman decides sex work is the easiest and most flexible way to fund a bohemian life of acting or painting or writing. Richard Carrier chipped in to let the Internet know that sex positivity is good but woo and AI crankery (“batshit crazy religious beliefs”) are not. COOLM was registered as a business in Mountain View, CA in 2019 and seems to be defunct. Mountain View is part of the Bay Area.
AHH remember the good old days when believing LLMs were conscious got you fired instead of huge multi-billion dollar investments and cult following? Pepperidge farm remembers
Mountain View is part of the Bay Area.
one of the most damning sentences in the whole post, really
new 404 media article casts proton, the administrators of protonmail, in a somewhat unflattering light
Understatement!
