gled is a user on mastodon.host. You can follow them or interact with them if you have an account anywhere in the fediverse. If you don't, you can sign up here.
gled @gled

Mastodon server admins that asked me the patch to use Postgresql instead of ES:
github.com/gled-rs/mastodon/bl ( Untested but seems complete ).

PR welcome as usual if you want.

Β· Web Β· 15 Β· 11

@gled

Is that all it took? Some postgresql voodoo and about 10 lines changed?

It seems too simple, but I'll give it a go. Thanks for doing this. You are right for small instances ES is just not viable.

@smallsees it is all it take yes. Hence my surprise when I read that ES implementation was 'so much easier'.

@gled @smallsees yeah, just no filtering as in upstream, though... but I'm glad someone did this

@charlag @smallsees @gled How does it perform? I thought that was one of the main justification for ES

@cbowdon @gled @smallsees the only justification I've heard is that it was super easy to add (which I think is a valid reason)

@charlag @smallsees @gled @cbowdon I don't know if the Chewy gem used in Mastodon handles this stuff, but other advantages of ES are things like homophones, diacritics, plurals, synonyms, "fuzzyness" (typos), stemming, etc., which get really hairy to do rolling your own searches in Postgres. That said, I'd like to see a simple database fulltext search as an option for those who have religious opinions about ES.

@charlag @cheesegrits @gled @cbowdon there was a handful of lines to change.

Postgresql FTS doesn't just do direct matches but also match where the root word is the same to cater for plurals etc.

Not implementing ES helps those smaller instances because of its huge resource requirements

@cheesegrits @charlag @smallsees @gled @cbowdon it is not a 'religious' opinion thing. It is a question of segmenting admins with few resources out of the feature, and that is unacceptable. FTS with pg is as good as ES for the need, even for big instances. That was either a deliberate choice of lazyness and locking out as much as possible from having the feature or something more shady as ES real strength is big data analysis ( some processing done on the toot data other that searching ? ).

@gled @charlag @smallsees @cbowdon I run ES on three instances, all "small" (single user, handful of users and a couple of hundred). Other than some extra memory, it hasn't gobbled up resources - I had enough headroom on all three, all <= $20/month Digital Ocean droplets.

I didn't mean "religious" in a pejorative sense, just "concerns other than purely technical", such as worries that Something Shady might be going on. So absolutely, yes, I'd like to see a pure database option for this.

@cbowdon @charlag @gled I got a tiny instance which means its not powered that well and it works fine

@cbowdon @charlag @smallsees @gled unless you want to do more than just searching toots, Pg is more than enough even if you have a big instance. We have FTS for months now on mastodon.host ( it started around June 2017 ) and I tried Sphinx, ES, lucene and Pgsql. We sticked with PG for performance and ease of use and have been running it since October 2017. I have instrumented heavily Mastodon, and search with PG is not an issue at all. There are other elements taking more resources.

@cbowdon @gled @charlag it works great. I mean I have a tiny instance but also means it's not very powerful hardware. I don't notice any extra delay.

I know larger instances use it too and it seems to work fine. It means I don't need to deploy ES with the associated cost. I think for most people there is no reason at all to use ES for Mastodon, just use the postgresql FTS.

@gled I installed it just now. Wasn't a problem at all. The patch didn't apply cleanly but it was such a simple one I hand patched it.