🛎️ Syncing of media files to object storage went too fast and rate limited the storage bucked during the night (CEST night), resulting in Mastodon's backlog processing/retry queue having a damn huge latency (4 hours right now).

Stopped the sync (which was mostly complete) so rate limiting would stop and Mastodon could process more without retrying.

Gonna add temporarily more workers for the queue so it processes the backlog more rapidly

interestingly even after stopping the sync we're still getting rate limited by object storage (while it shouldn't exceed 250 req/s, as there's just 50 workers)…

Show thread

New update: Scaleway Object Storage is just incredibly broken. Even without any requests it's still limiting or even returning error 400.

Show thread

Amazing! Three hours after opening the ticket at Scaleway they responded. We never hit the rate limit, it was a technical issue on their side. Should be fixed now and the processing queue should process everything now

Show thread

another update: the queue is still processing :(

Show thread

@ferh Thanks! I'm finally starting to get my first notification from out of this server (from 20 hours ago). ;)

@trinsec Weird (20 hours) because Sidekiq is saying (Latency: 4 hours). But cool. Not really solved tho, still getting a lot of rate limiting errors from S3. still on it..

@ferh Getting stuff from 7 hours ago appearing now. I daresay it's an improvement over 20 hours. ;)

@trinsec yeah :( the queue is going down very slowly. For some reason I can get Sidekiq to use all his threads nor all the DB connections.

@ferh Cool! Looking forward to up-to-date feeds soon. :D
And was that exclamation sarcastic? I can't quite figure out if 3 hours is a good or bad response time, heh.

@trinsec yes quite sarcastic. I wish they were monitoring their services a bit more.... always annoying to stumble upon an issue on their side, while they don't notice it, and act after some hours...

@ferh @trinsec well their status page says it's operational so everything must be fine 🙄

@eqe haha :-) It's not ODS I'm using, but "Object Storage". Either way -- ODS had a shit ton of issues since sept, and yeah, you can see it right ? :))) @trinsec

@ferh My feeds seem to be up-to-date again mostly. Is stuff back to regular scheduling now?

@ferh 🎉 (We so need Telegram stickers here!)

Sign in to participate in the conversation

What sets appart?

As a general instance, we are not centered on a specific theme, or a specific language. Everyone is welcome as long as they follow the few rules we have.

We always run the latest stable version of Mastodon, providing you a stock experience, with the following modifications:

  • 2048 character limit (not 500).
  • 512 character limits for the account's bio (not 160).
  • full column width (not fixed size).
  • more search results (20 instead of 5 per type of search).
  • Audio upload (not limited to videos only).
  • full text search of statuses (not restricted to searching tags and users).
  • trending tags are displayed on the main UI ( and a more complete view is available here)
  • boosts and replies are shown in the timelines (more complete view for everyone).
  • Your account is also a XMPP account (Federated Instant Messaging, take a look online to find a client and use your username and password to configure it, you can also use our online web client). Woohoo! two services in one!
  • We also have a very good view of the federation (our federated timeline gives you a more broad view of the Mastodon network than a 'normal' instance, there's a lot to read in a lot of languages).
  • We also host a Peertube instance for the convenience. Try it and let us know what you think !
  • In the same vein, we host a Funkwhale instance for the convenience. Try it and let us know what you think !
  • If you want to be on a more quiet, moderated instance, or use another AP implementation, we host a Pleroma instance for the convenience. This is the same fediverse as with Mastodon, but with a different default UI and way more optimized backend. Try and see which one you prefer !
  • Now if you want a different UI, the stock mastodon one does not please you, you can check Pinafore.
  • We do also host for the convenience a video Conference server ( Jitsi Meet )
  • We have a mastodon relay for other instances to bootstrap their timelines:

Although we are a medium+ sized instance, we'd like to keep the feeling that you are at home and safe here.