• 0 Posts
  • 20 Comments
Joined 1 year ago
cake
Cake day: July 2nd, 2023

help-circle










  • I can share it, but right now I have it grabbing the latest artifact from the lemmy-exporter which looks like as of June 3rd they stopped updating their repo with the export artifact. Meaning, I need to implement some sort of crawler myself or use a different method for generating the list of communities.

    I can share the code with this caveat if you want to use it anyway/try and change it yourself. It is written in Golang… will post back in a bit with a github repo. need to set it up


  • You won’t get much old data when you first start subscribing to new communities from your instance. You’ll only get new content. When you add a community you’ll get 20 recent topics with no comments or upvotes. As new content comes in you’ll pick up all of it. At least one member of your instance needs to subscribe to a community to begin syncing with that community. To solve the lack of content on my personal instance, I made a bot that subscribed to the top 300 communities with more than 1000 subscribers. After a day or so I now have tons of content… I may stick the bot script on a weekly cronjob to keep my instance synced with the largest communities.


  • It’s pretty lightweight. I’ve given each container 1/3Gi of memory and 1CPU limit with low requests. Utilizing kubernetes HPA to scale containers under load up to 4 replicas. It only scales when a user takes large actions (subscribing to hundreds of new to the instance communities at once). But once the initial federation begins it seems to quickly scale back down. The biggest bottleneck is pictrs since it is stateful.

    So far the database and pictrs is only about 2Gi of storage but I’ve allocated 25Gi to each since I have a lot to spare at the moment.

    I have to play with the HPA more since I’m not happy yet with my settings. I have 2 users and 1 bot on my instance.

    I’d like to start contributing to Lemmy’s codebase so I wanted to host my own instance to learn the inner workings.

    My postgres is a single replica at the moment but I may scale that if stability is becomes a problem.



  • Yes this was unclear to me until I set up my own instance is may be a big deal breaker to the average user. If you can’t view “All” and really see all communities on all federated instances then there is no ability to surf Lemmy. At the very least it should have all communities of instances from either a list or user subscriptions.

    edit: if you can share your script that would be amazing, I was thinking of having to do the same thing…