dudeami0

SQL is the industry standard for a reason, it’s well known and it does the job quite well. The important part of any technology is to use it when it’s advantageous, not to use it for everything. SQL works great for looking up relational data, but isn’t a replacement for a filesystem. I’ll try to address each concern separately, and this is only my opinion and not some consensus:

Most programmers aren’t DB experts: Most programmers aren’t “experts”, period, so we need to work with this. IT is a wide and varied field that requires a vast depth of knowledge in specific domains to be an “expert” in just that domain. This is why teams break up responsibilities, the fact the community came in and fixed the issues doesn’t change the fact the program did work before. This is all normal in development, you get things working in an acceptable manner and when the requirements change (in the lemmy example, this would be scaling requirements) you fix those problems.

translation step from binary (program): If you are using SQL to store binary data, this might cause performance issues. SQL isn’t an all in one data store, it’s a database for running queries against relational data. I would say this is an architecture problem, as there are better methods for storing and distributing binary blobs of data. If you are talking about parsing strings, string parsing is probably one of the least demanding parts of a SQL query. Prepared statements can also be used to separate the query logic from the data and alleviate the SQL injection attack vector.

Yes, there are ORMs: And you’ll see a ton of developers despise ORMs. They is an additional layer of abstraction that can either help or hinder depending on the application. Sure, they make things real easy but they can also cause many of the problems you are mentioning, like performance bottlenecks. Query builders can also be used to create SQL queries in a manner similar to an ORM if writing plain string-based queries isn’t ideal.

For your own sanity, please use a formatter for your IDE. This will also help when others (and you) read the code, as indentation is a convenience for understanding program flow. From what I see:

• Your enable and disable functions are never called for this portion of code
• You use a possibly undeclared enabled variable, if so it never passes scopes between the handleClick and animation methods
• You do not use any callback or await for invoke or updateCurrentBox, causing all the code after either to immediately run. As a result, enabled is never false, since it just instantly flips back to true. I’m not sure what library invoke is from, but there should be a callback or the function returns a Promise which can be awaited.

TL;DR: The bot is configured to condense certain instances and communities. At the moment, only beehaw.org is marked to be condensed.

Quickly looking at the source code, it seems ReplyToPostsCommand uses a SummaryTextWrapper, which contains an iterable for both CondensedSummaryTextWrapperProvider and DefaultSummaryTextWrapperProvider. The DefaultSummaryTextWrapperProvider has a priority of -1_000 (so it’s always checked last) and is set to always return true on the supports(Community $community): bool. CondensedSummaryTextWrapperProvider references the config/services.yaml for it’s supports(Community $community): bool call which lists 0 condensed communities and 1 condensed instance, being beehaw.org.

In my humble opinion, we too are simply prediction machines. The main difference is how efficient our brains are at the large number of tasks given for it to accomplish for it’s size and energy requirements. No matter how complex the network is it is still a mapped outcome, just the number of factors weighed is extremely large and therefore gives a more intelligent response. You can see this with each increment in GPT models that use larger and larger parameter sets giving more and more intelligent answers. The fact we call these “hallucinations” shows how effective the predictive math is, and mimics humans abilities to just make things up on the fly when we don’t have a solid knowledge base to back it up.

I do like this quote from the linked paper:

As we will discuss, we find interesting evidence that simple sequence prediction can lead to the formation of a world model.

That is to say, you don’t need complex solutions to map complex problems, you just need to have learned how you got there. It’s never purely random attempts at the problem, it’s always predictive attempts that try to map the expected outcomes and learn by getting it right and wrong.

At this point, it seems fair to conclude the crow is relying on more than surface statistics. It evidently has formed a model of the game it has been hearing about, one that humans can understand and even use to steer the crow’s behavior.

Which is to say that it has a predictive model based on previous games. This does not mean it must rigidly follow previous games, but that by playing many games it can see how each move affects the next. This is a simpler example because most board games are simpler than language with less possible outcomes. This isn’t to say that the crow is now a grand master at the game, but it has the reasoning to understand possible next moves, knows illegal moves, and knows to take the most advantageous move based on it’s current model. This is all predictive in nature, with “illegal” moves being assigned very low probability based on the learned behavior the moves never happen. This also allows possible unknown moves that a different model wouldn’t consider, but overall provides what is statistically the best move based on it’s model. This allows the crow to be placed into unknown situations, and give an intelligent response instead of just going “I don’t know this state, I’ll do something random”. This does not always mean this prediction is correct, but it will most likely be a valid and more than not statistically valid move.

Overall, we aren’t totally sure what “intelligence” is, we are just an organism that has developed more and more capabilities to process information based on a need to survive. But getting down to it, we know neurons take inputs and give outputs based on what it perceives is the best response for the given input, and when enough of these are added together we get “intelligence”. In my opinion it’s still all predictive, its how the networks are trained and gain meaning from the data that isn’t always obvious. It’s only when you blindly accept any answer as correct that you run into these issues we’ve seen with ChatGPT.

Thank you for sharing the article, it was an interesting article and helped clarify my understanding of the topic.

Disclaimer: I am not an AI researcher and just have an interest in AI. Everything I say is probably jibberish, and just my amateur understanding of the AI models used today.

It seems these LLM’s use a clever trick in probability to give words meaning via statistic probabilities on their usage. So any result is just a statistical chance that those words will work well with each other. The number of indexes used to index “tokens” (in this case words), along with the number of layers in the AI model used to correlate usage of these tokens, seems to drastically increase the “intelligence” of these responses. This doesn’t seem able to overcome unknown circumstances, but does what AI does and relies on probability to answer the question. So in those cases, the next closest thing from the training data is substituted and considered “good enough”. I would think some confidence variable is what is truly needed for the current LLMs, as they seem capable of giving meaningful responses but give a “hallucinated” response when not enough data is available to answer the question.

Overall, I would guess this is a limitation in the LLMs ability to map words to meaning. Imagine reading everything ever written, you’d probably be able to make intelligent responses to most questions. Now imagine you were asked something that you never read, but were expected to respond with an answer. This is what I personally feel these “hallucinations” are, or imo best approximations of the LLMs are. You can only answer what you know reliably, otherwise you are just guessing.

Best I could find is here, which is an article by Randall Munroe (the xkcd artist), and states:

davean (the xkcd sysadmin) wrote the patch

This blog post links to another wayback machine page (thank you archive.org!) here, which explains the sorting algorithm and states it’s original author:

Fortunately, the math for this was worked out in 1927 by Edwin B. Wilson.

Sounds like the cache got corrupted possibly? See if Ctrl+F5 clears up the issue, or try restarting your browser.

These are just my opinions on the matter at hand.

TLDR; it’s not all about growing as massive as possible and letting everyone talk to everyone. It’s about communities being able to make choices for their user base and the freedom to choose who to federate with. It’s also about users having a choice of which instance they use to interact with the fediverse, and with whom. Having Meta involved limits these choices in not so obvious ways.

Doesn’t the fediverse have an inherent protection and/or immunity from corporate take-over?

Yes, but that does not mean it is invulnerable. Take the World Wide Web as an example, over the past couple of decades the decentralized web has become increasingly centralized. Projects such as Lemmy and Mastodon are a shot back at this trend, to try and break the web up as it was. Each instance gets to decide if letting large corporations federate with them is the best choice or not. It seems that a lot do not want this, and this is exactly the kind of protection from corporate take over that is inherent. The more large central servers are allowed to take a central role, the more power they will gain to snuff small communities and instances. They will do this by fragmenting users bases and communities over time, or any other dirty tricks they can come up with.

Also, having billions of dollars at your disposal is known to increase your influence overall. They can outspend anyone to sell most people on how Threads is interconnected and fediverse friendly, if you let them sell that lie they will win in time. They’ll do this, pull the rug and say how other independent instances aren’t corporating. They will shut off access to these communities in one way or another and begin the process of centralization. It has happened before, and will happen again.

Aren’t we protected?

If you choose to not use Threads, you are not giving your information directly to Meta. But, that does not mean you are safe. Meta is a corporation, and will try to pull whatever tricks they can to take over as the dominate player. They are going head to head with Twitter, what makes you think instances a fraction of Twitters size are safe?

Also, saying we are isolated by our individual instances is a bit humorous as they are federated. If one instance pushes most of the content is that really isolated? What about upvotes, engagement and any other activity that is pushed to other servers via the ActivityPub protocol? These will all be taken in by Meta, which means you are feeding them activity. Sure it’s safer, but they are still getting more data by engaging in the ActivityPub protocol than they get via scrapping pages. Also, they don’t have to play fair with the ActivityPub protocol, there are a lot of dirty tricks that could be used to hamper content on other instances than their own.

Is there anything currently stopping Meta from scraping the Fediverse for our content?

No, and the fediverse should not care. The goal of the fediverse at the moment is to stay independent and have a user base that is not reliant on a single entity and to stay away from the influence of corporate interests. If you operate in a public space, someones always going to be able to see it. It’s all about who owns that public space.

Won’t we grow & educate?

Who is we? Users that value their freedom will stay in the independent fediverse instances. Those who are looking for a twitter alternative will probably go to Threads. Those who don’t care will probably stay on Twitter. Any of these users might have multiple accounts on some or all of these services. Trying to group this together as “we” is a bit disingenuous.

As for growth, it’s not safe to assume that independent instances will grow because of the federation of users from Threads. Users that are on Threads are likely to stay on Threads, users that join instances are likely to stay there. Look to linux users to see why you aren’t going to convert many over the virtues of freedom and decentralization, you’ll just become another “fanboy”.

Aren’t we worried we’re forcing an ultimatum while the Fediverse is still in its infancy?

What is the ultimatum? This is a pretty loaded question, since some of the fediverse is already fractured. The fact you can spin up your own instance, invite whoever you want and keep the interests of your community out of the hands of corporations is the goal. Freedom to host your own community. Anything else is just having a capitalist mindset on growth, the line doesn’t always have to go up. Getting the most users isn’t the end game, it’s having a community that you belong to and feel a part of.

What’s the harm in pulling the ripcord if we try it, and it’s truly not a good fit?

Each instance chooses what is best for their community. Being a part of the mainstream content feed isn’t the goal of most of these decentralized communities.

“What about an influx of low-quality content?”

Why do instances need to let users block Meta when they know their users want Meta blocked? What’s stopping users from going to an instance that doesn’t block Meta if their instance disagrees with their opinion? It’s all about doing what instances communities want, or users can migrate if they feel their needs aren’t being met.

“What if Meta doesn’t moderate well?”

Meta will probably be able to moderate for their advertisers better than most instance operators will be able to. But again, it’s not about moderation and sanitizing content for advertiser revenue, it’s about having a space that is for the community by the community. It doesn’t need to be a single homogeneous community so ads can sell. Some of us want that outside of a corporations control, others don’t or don’t care, all are valid. Thankfully, everyone has a choice instead of being forced to do one or the other.

The screenshot is a picture of visit duration, I would be interested to see the other statistics. Most Reddit users will return though once the sub-reddits open back up, which is why they are forcing it.

The numbers are a little higher than you mention (currently ~3.2k active users). The server isn’t very powerful either, it’s now running on a dedicated server with 6 cores/12 threads and 32 gb ram. Other public instances are using larger servers, such as lemmy.world running on a AMD EPYC 7502P 32 Cores “Rome” CPU and 128GB RAM or sh.itjust.works running on 24 cores and 64GB of RAM. Without running one of these larger instances, I cannot tell what the bottleneck is.

The issues I’ve heard with federation are currently how ActivityPub is implemented, and possibly the fact all upvotes are federated individually. This means every upvote causes a federation queue to be built, and with a ton of users this would pile up fast. Multiply this by all the instances an instance is connected to and you have an exponential increase in requests. ActivityPub is the same protocol used by other federated servers, including Mastodon which had growing pains but appears to be running large instances smoothly now.

Other than that, websockets seem to be a big issue, but is being resolved in 0.18. It also appears every connected user gets all the information being federated, which is the cause for the spam of posts being prepended to the top of the feed. I wouldn’t be surprised if people are already botting content scrapers/posters as well, which might cause a flood of new content which has to get federated which causes queues to back up; this is mostly speculation though.

As it goes with development, generally you focus on feature sets first. Optimization comes once you reach a point a code-freeze makes sense, then you can work on speeding things up without new features breaking stuff. This might be an issue for new users temporarily, but this project wasn’t expecting a sudden increase in demand. This is a great way to show where inefficiencies are and improve performance is though. I have no doubt these will be resolved in a timely manner.

My personal node seems to use minimal resources, not having even registered compared to my other services. Looking at the process manager the postgres/lemmy backend/frontend use ~250MB of RAM.

For now, staying off lemmy.ml and moving communities to other instances is probably best. The use case of large instances anywhere near the scale of reddit wasn’t the goal of the project until reddit users sought alternatives. We can’t expect to show up here and demand it work how we want without a little patience and contributing.

Neither seems to work for me, maybe federation is overloaded with the sudden influx? I’ll report back if it ends up working.