@trinsec yes! \o/
another update: the queue is still processing :(
@trinsec yeah :( the queue is going down very slowly. For some reason I can get Sidekiq to use all his threads nor all the DB connections.
@trinsec yes quite sarcastic. I wish they were monitoring their services a bit more.... always annoying to stumble upon an issue on their side, while they don't notice it, and act after some hours...
Amazing! Three hours after opening the ticket at Scaleway they responded. We never hit the rate limit, it was a technical issue on their side. Should be fixed now and the processing queue should process everything now
New update: Scaleway Object Storage is just incredibly broken. Even without any requests it's still limiting or even returning error 400.
@trinsec Weird (20 hours) because Sidekiq is saying (Latency: 4 hours). But cool. Not really solved tho, still getting a lot of rate limiting errors from S3. still on it..
interestingly even after stopping the sync we're still getting rate limited by object storage (while it shouldn't exceed 250 req/s, as there's just 50 workers)…
🛎️ Syncing of media files to object storage went too fast and rate limited the storage bucked during the night (CEST night), resulting in Mastodon's backlog processing/retry queue having a damn huge latency (4 hours right now).
Stopped the sync (which was mostly complete) so rate limiting would stop and Mastodon could process more without retrying.
Gonna add temporarily more workers for the queue so it processes the backlog more rapidly
due to an incident on Scaleway Object Storage, media uploads/medias may have issues - https://status.scaleway.com/incident/933
@ilias Had issues with mail provider bill. Should be working now,, account is reactivated
https://relay.mastodon.host has been migrated to it's new home!
the .host of mastodon.host - main account @href (contact me there, of by email: href (at) random.sh)
As a general instance, we are not centered on a specific theme, or a specific language. Everyone is welcome as long as they follow the few rules we have.
We always run the latest stable version of Mastodon, providing you a stock experience, with the following modifications:
Although we are a medium+ sized instance, we'd like to keep the feeling that you are at home and safe here.