I love AI, when it works, but even I don’t want it in 99% of my life. It’s a tool, and it should be rolled back to “tool” status and not some kind of therapist or friend or fact finder.
Just use it to replace stack overflow. That was never a good thing. ;)
Just use it to replace stack overflow. That was never a good thing. ;)
It was and is a good thing. I think it’s a huge problem that when it fades away future next generations of AI will not be able to learn from it. And the culture at that point will be to depend on AI instead of having sites like those, so even getting it back isn’t going to work. Honestly I think maybe we need a new job: “experts” who just do some fun highly thematic stuff and post results online to train AI.
Scientists don’t work on mundane problems. AI need training to solve mundane tasks. Just take stackoverflow as the literal example. It’s not science. We might need a new profession to produce training material for arbitrary areas of expertise, not limited to research level topics. A job like this might be talking about such topics on a searchable forum, for example. Each post can be evaluated by a separate AI system to assign score points to it. I don’t know whether this sounds more dystopian or fun. It’s definitely not that far from social credit systems.
As a computer science professor, you are giving us scientists way too much credit. Most articles we produce are basically just: hey, you can use this framework/use this strategy to solve this problem. There is a sad, or maybe not depending on your POV, problem in computer science research that most research doesn’t follow the scientific method.
I’ve been experimenting with agentic coding the past couple of weeks. The task is to write a data scraper for a report file I get out of a commercial tool I have to use for work.
It’s a pain of a format because it’s not written with computer parsing in mind. It’s verbose, contains loads of redundant parts, and doesn’t have good delimiters around data. It’s big too. 500MB uncompressed, so we keep them gzip’d.
All reasons why I don’t want to write the code to do it.
The model identifies the file format without me saying where it came from, but it sits in this loop:
“Let me analyse the input file” - Does various greps, seds, and awks to pull out sections and find patterns in their formatting.
“I understand the format enough for now” - and then proceeds to write out a list of rules it’s discovered. This bit is actually quite impressive.
“Now I need to draft the data structures the data will go into” - …and it will write some over-decomposed objects. Not out to disk though.
“The user says they want a parser, so let me start writing the actual code” … Finally!.. But hang on…
“Actually, I need to understand the file format more” - loop to the top.
It does this for hours.
The tiny bits of code I’ve actually managed to get out of it are really bad. It’s like the code you’d get back from some race-to-the-bottom offshore software “team” you were forced to work with 10-15 years ago because your boss had found an “amazing opportunity”. In actuality it was somebody’s teenage nepo-hire. Similar adherence to rules and standards too.
I already have a rough data scraper for this file. It’s a couple of hundred lines of python. I wrote it in an afternoon. It’s not great. It doesn’t get everything I want out. However it exists and is usable. This isn’t an intractable task.
I am basically a script kiddie and Anthropic Claude helped me design a small app for myself that was way beyond my own capabilities and it even went as far as documenting in comments what each bit of the program did for me, to help me understand the logic.
I only have access because I got a free year of Perplexity.ai because I had a paypal account (barf, had to have it to get paid for gig work).
I see it as very useful for anything technical, it helped me through some troubleshooting a hardware issue with my monitor I was having trouble pinning down by browsing forums.
I would never purposely divulge personal info to an LLM, anything that’s shared has had personal info stripped.
I am basically a script kiddie and Anthropic Claude helped me design a small app for myself that was way beyond my own capabilities and it even went as far as documenting in comments what each bit of the program did for me, to help me understand the logic.
This is the good shit, right here!
AI does shit tier work, but it provides access to new skills.
If you learned anything from the experience, you’re a programmer, now.
Welcome to the crew.
Don’t be afraid to toss out the training wheels (AI) when it gets in your way, and try to enjoy the ride.
Lots of people struggle to use it. Don’t feel bad. I think to use it correctly, one must first want to use it. After that, it becomes easier.
I recall when ChatGPT first came out, a coworker was criticizing it. I asked for a demonstration, and they just kept gaming it. Just actively trying to make it fail to do things it already struggled to do. I asked them to do something I already knew worked pretty well, and they tried to game again. I asked them to stop gaming it, and they just refused.
Clearly, they were not the target audience for AI. And that’s fine!
What I find is that people who love ai, think it is the greatest thing on earth and can do all things ever better than humans. Then there are the rest of us.
The problem for me personally, is that for my job, there simply isn’t enough information for the AI companies to steal train models on. I do industrial programming. It’s programming with fucking crayons. AI is hilariously wrong every single time I have asked it anything.
Not OP, but I was pretty disappointed trying Claude 4.6
Prompted
Write a C program to find the longest word in a static5x5 arrayof characters.
These characters shall be defined in a header file, you may allocate it withany letters for now
This program should find the longest word, using words available in a file at/usr/share/dict/words
This file will have one word per line
The rules of the longest word are that you may select the next letter inany direction from your current letter onecharacter away, including diagonals
Any index may be the starting point, and you may not repeat a space on the grid
It did a breadth first search for the longest path, then checked if that longest path was a word, rather than checking each step, so it never found any words
When I asked it to fix that, it then opened and reread the entire dictionary for each character
Once I got it to fix that, I asked it to read the input array from a file, and after 30 minutes of asking it in different ways, it never managed to successfully read that file in
All in all, it took longer than just writing it myself, even for what I would call an interview question
In a single prompt I would not expect that specific exercise to produce efficient code, but within a few prompts it should. Certainly less time than it would take someone to write it themselves.
There are always creative ways to squeeze extra performance out of code if you spend enough time on it.
Certainly less time than it would take someone to write it themselves.
I mean, sure - for you and I, who aren’t qualified to write that specific code, maybe we can prompt the electronic idiot to get there. Of course, neither we nor the electroic idiot knows where there is, and at best we will copy in exisitng better code that we should have imported from a library. So we gave up automated updates to avoid reading the manual pages.
In contrast, for domains I’m an expert in, babysitting the electric idiot is always a complete waste of time. I can just call the correct library, the correct way, on the first attempt.
Today’s AI really highlights exisitng technical debt. If there’s already a mountain of it, I can see how the learning model may help wrangle it, and how it may be hard to see the added costs.
I’ll be surprised if there is any information to be had. Most people stop at this point because it either never happened or they never actually put any effort into it which is why it failed.
I usually stop at this point because it’s a complete waste of my fucking time. I already know where the relevant sources of information are, and the current AI models have proven themselves to be incapable of distinguishing between firmware versions or subtle differences in model numbers. I try things again every once in a while to see if anything has improved, and so far, no dice.
Exactly, it’s a tool. It’s potentially useful, in certain situations, but I’ll be the one deciding if I want it at all, if it’s useful and what it’s useful for, not some company. If they tell me to use a table saw to clean my teeth, I’m going to tell them to go fuck themselves. Nothing wrong with table saws, but fuck any company that tells me what I ought to be using a table saw for, because it probably isn’t in my best interest.
I love AI, when it works, but even I don’t want it in 99% of my life. It’s a tool, and it should be rolled back to “tool” status and not some kind of therapist or friend or fact finder.
Just use it to replace stack overflow. That was never a good thing. ;)
It was and is a good thing. I think it’s a huge problem that when it fades away future next generations of AI will not be able to learn from it. And the culture at that point will be to depend on AI instead of having sites like those, so even getting it back isn’t going to work. Honestly I think maybe we need a new job: “experts” who just do some fun highly thematic stuff and post results online to train AI.
It’s probably just going to be people who put code online that doesn’t work at all in an attempt to poison AI :(
ChatGPT 3.5 had a lot of bad code. It was still pretty amazing for the time. But it wasn’t at all a coding agent.
It’s truly incredible how far it has come in just a year and especially in two.
I think you just described scientists :P
Scientists don’t work on mundane problems. AI need training to solve mundane tasks. Just take stackoverflow as the literal example. It’s not science. We might need a new profession to produce training material for arbitrary areas of expertise, not limited to research level topics. A job like this might be talking about such topics on a searchable forum, for example. Each post can be evaluated by a separate AI system to assign score points to it. I don’t know whether this sounds more dystopian or fun. It’s definitely not that far from social credit systems.
As a computer science professor, you are giving us scientists way too much credit. Most articles we produce are basically just: hey, you can use this framework/use this strategy to solve this problem. There is a sad, or maybe not depending on your POV, problem in computer science research that most research doesn’t follow the scientific method.
The problem for me personally, is that AI has worked exactly 0% of the time when I try to use it.
I’ve been experimenting with agentic coding the past couple of weeks. The task is to write a data scraper for a report file I get out of a commercial tool I have to use for work.
It’s a pain of a format because it’s not written with computer parsing in mind. It’s verbose, contains loads of redundant parts, and doesn’t have good delimiters around data. It’s big too. 500MB uncompressed, so we keep them gzip’d.
All reasons why I don’t want to write the code to do it.
The model identifies the file format without me saying where it came from, but it sits in this loop:
It does this for hours.
The tiny bits of code I’ve actually managed to get out of it are really bad. It’s like the code you’d get back from some race-to-the-bottom offshore software “team” you were forced to work with 10-15 years ago because your boss had found an “amazing opportunity”. In actuality it was somebody’s teenage nepo-hire. Similar adherence to rules and standards too.
I already have a rough data scraper for this file. It’s a couple of hundred lines of python. I wrote it in an afternoon. It’s not great. It doesn’t get everything I want out. However it exists and is usable. This isn’t an intractable task.
Your mistake is knowing that you’re doing, so you catch AI’s mistakes.
Try using it for stuff you’re not remotely qualified to do in the first place, then it can look useful!
I am basically a script kiddie and Anthropic Claude helped me design a small app for myself that was way beyond my own capabilities and it even went as far as documenting in comments what each bit of the program did for me, to help me understand the logic.
I only have access because I got a free year of Perplexity.ai because I had a paypal account (barf, had to have it to get paid for gig work).
I see it as very useful for anything technical, it helped me through some troubleshooting a hardware issue with my monitor I was having trouble pinning down by browsing forums.
I would never purposely divulge personal info to an LLM, anything that’s shared has had personal info stripped.
This is the good shit, right here!
AI does shit tier work, but it provides access to new skills.
If you learned anything from the experience, you’re a programmer, now.
Welcome to the crew.
Don’t be afraid to toss out the training wheels (AI) when it gets in your way, and try to enjoy the ride.
Lots of people struggle to use it. Don’t feel bad. I think to use it correctly, one must first want to use it. After that, it becomes easier.
I recall when ChatGPT first came out, a coworker was criticizing it. I asked for a demonstration, and they just kept gaming it. Just actively trying to make it fail to do things it already struggled to do. I asked them to do something I already knew worked pretty well, and they tried to game again. I asked them to stop gaming it, and they just refused.
Clearly, they were not the target audience for AI. And that’s fine!
What I find is that people who love ai, think it is the greatest thing on earth and can do all things ever better than humans. Then there are the rest of us.
I guess you don’t know many people.
Probably not…
The problem for me personally, is that for my job, there simply isn’t enough information for the AI companies to
stealtrain models on. I do industrial programming. It’s programming with fucking crayons. AI is hilariously wrong every single time I have asked it anything.Give me an example of what you’ve asked it to do? And, what model and app did you use?
Not OP, but I was pretty disappointed trying Claude 4.6
Prompted
Write a C program to find the longest word in a static 5x5 array of characters. These characters shall be defined in a header file, you may allocate it with any letters for now This program should find the longest word, using words available in a file at /usr/share/dict/words This file will have one word per line The rules of the longest word are that you may select the next letter in any direction from your current letter one character away, including diagonals Any index may be the starting point, and you may not repeat a space on the gridIt did a breadth first search for the longest path, then checked if that longest path was a word, rather than checking each step, so it never found any words
When I asked it to fix that, it then opened and reread the entire dictionary for each character
Once I got it to fix that, I asked it to read the input array from a file, and after 30 minutes of asking it in different ways, it never managed to successfully read that file in
All in all, it took longer than just writing it myself, even for what I would call an interview question
In a single prompt I would not expect that specific exercise to produce efficient code, but within a few prompts it should. Certainly less time than it would take someone to write it themselves.
There are always creative ways to squeeze extra performance out of code if you spend enough time on it.
I mean, sure - for you and I, who aren’t qualified to write that specific code, maybe we can prompt the electronic idiot to get there. Of course, neither we nor the electroic idiot knows where there is, and at best we will copy in exisitng better code that we should have imported from a library. So we gave up automated updates to avoid reading the manual pages.
In contrast, for domains I’m an expert in, babysitting the electric idiot is always a complete waste of time. I can just call the correct library, the correct way, on the first attempt.
Today’s AI really highlights exisitng technical debt. If there’s already a mountain of it, I can see how the learning model may help wrangle it, and how it may be hard to see the added costs.
Aren’t qualified? I mean… I’m qualified. You aren’t?
What “domains” are you an expert in?
If it can’t output ~50 lines of code that is reasonably common from textbooks with one minor modification, I’m not clear what the benefit is
It’s certainly not faster
I already stated I kept prompting it for over 30 minutes and it still hadn’t fully completed the problem
Well, if it took you 30 minutes, it’s not the AI’s fault.
Kibblebits wants to make the information known so newer models can train on it and win at life
I’ll be surprised if there is any information to be had. Most people stop at this point because it either never happened or they never actually put any effort into it which is why it failed.
I usually stop at this point because it’s a complete waste of my fucking time. I already know where the relevant sources of information are, and the current AI models have proven themselves to be incapable of distinguishing between firmware versions or subtle differences in model numbers. I try things again every once in a while to see if anything has improved, and so far, no dice.
🥹
Exactly, it’s a tool. It’s potentially useful, in certain situations, but I’ll be the one deciding if I want it at all, if it’s useful and what it’s useful for, not some company. If they tell me to use a table saw to clean my teeth, I’m going to tell them to go fuck themselves. Nothing wrong with table saws, but fuck any company that tells me what I ought to be using a table saw for, because it probably isn’t in my best interest.