Commit graph

18 commits

Author SHA1 Message Date
tobi
ac6ed3d939
[chore] Update bun / sqlite versions; update gtsmodels ()
* upstep bun and sqlite versions

* allow specific columns to be updated in the db

* only update necessary columns for user

* bit tidier

* only update necessary fields of media_attachment

* only update relevant instance fields

* update tests

* update only specific account columns

* use bool pointers on gtsmodels
includes attachment, status, account, user

* update columns more selectively

* test all default fields on new account insert

* updating remaining bools on gtsmodels

* initialize pointer fields when extracting AP emoji

* copy bools properly

* add copyBoolPtr convenience function + test it

* initialize false bool ptrs a bit more neatly
2022-08-15 11:35:05 +01:00
tobi
91c8d5d20d
[bugfix] Fix thumbnails not taking exif rotation into account ()
* use disintegration/imaging instead of nfnt/resize

* update tests

* use disintegration lib for thumbing (if necessary)
2022-08-10 14:05:14 +02:00
kim
7cc40302a5
[chore] consolidate caching libraries ()
* add miekg/dns dependency

* set/validate accountDomain

* move finger to dereferencer

* totally break GetRemoteAccount

* start reworking finger func a bit

* start reworking getRemoteAccount a bit

* move mention parts to namestring

* rework webfingerget

* use util function to extract webfinger parts

* use accountDomain

* rework finger again, final form

* just a real nasty commit, the worst

* remove refresh from account

* use new ASRepToAccount signature

* fix incorrect debug call

* fix for new getRemoteAccount

* rework GetRemoteAccount

* start updating tests to remove repetition

* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).

* fix up tests to use main mock httpclient

* webfinger only if necessary

* cheeky linting with the lads

* update mentionName regex
recognize instance accounts

* don't finger instance accounts

* test webfinger part extraction

* increase default worker count to 4 per cpu

* don't repeat regex parsing

* final search for discovered accountDomain

* be more permissive in namestring lookup

* add more extraction tests

* simplify GetParseMentionFunc

* skip long search if local account

* fix broken test

* consolidate to all use same caching libraries

Signed-off-by: kim <grufwub@gmail.com>

* perform more caching in the database layer

Signed-off-by: kim <grufwub@gmail.com>

* remove ASNote cache

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, improve db tracing hooks

Signed-off-by: kim <grufwub@gmail.com>

* return ErrNoEntries if no account status IDs found, small formatting changes

Signed-off-by: kim <grufwub@gmail.com>

* fix tests, thanks tobi!

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-07-10 17:18:21 +02:00
Dominik Süß
9d0df426da
[feature] S3 support ()
* feat: vendor minio client

* feat: introduce storage package with s3 support

* feat: serve s3 files directly

this saves a lot of bandwith as the files are fetched from the object
store directly

* fix: use explicit local storage in tests

* feat: integrate s3 storage with the main server

* fix: add s3 config to cli tests

* docs: explicitly set values in example config

also adds license header to the storage package

* fix: use better http status code on s3 redirect

HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL

307/TemporaryRedirect would mean that this resource is usually located
here, not in this case

303/SeeOther indicates that the redirection does not link to the
requested resource but to another page

* refactor: use context in storage driver interface
2022-07-03 12:08:30 +02:00
tobi
cf5c6d724d
[chore] Validate/set account domain ()
* add miekg/dns dependency

* set/validate accountDomain
2022-06-11 11:09:31 +02:00
tobi
1cdc163276
[performance] Don't retry/backoff invalid http requests that will never succeed ()
* add httpguts (ew)

* add ValidateRequest err wrapping logic

* don't retry on unrecoverable errors

* i am very clever
2022-05-26 13:38:41 +02:00
kim
b56dae8120
[chore] Update all but bun libraries ()
* update all but bun libraries

Signed-off-by: kim <grufwub@gmail.com>

* remove my personal build script changes

Signed-off-by: kim <grufwub@gmail.com>
2022-05-02 15:05:18 +02:00
tobi
88979b35d4
[chore] Update bun and sqlite dependencies ()
* update bun + sqlite versions

* step bun to v1.1.3
2022-04-24 12:26:22 +02:00
tobi
07727753b9
[feature] Clean up/uncache remote media ()
* Add whereNotEmptyAndNotNull

* Add GetRemoteOlderThanDays

* Add GetRemoteOlderThanDays

* Add PruneRemote to Manager interface

* Start implementing PruneRemote

* add new attachment + status to tests

* fix up and test GetRemoteOlderThan

* fix bad import

* PruneRemote: return number pruned

* add Cached column to mediaattachment

* update + test pruneRemote

* update mediaTest

* use Cached column

* upstep bun to latest version

* embed structs in mediaAttachment

* migrate mediaAttachment to new format

* don't default cached to true

* select only remote media

* update db dependencies

* step bun back to last working version

* update pruneRemote to use Cached field

* fix storage path of test attachments

* add recache logic to manager

* fix trimmed aspect ratio

* test prune and recache

* return errwithcode

* tidy up different paths for emoji vs attachment

* fix incorrect thumbnail type being stored

* expose TransportController to media processor

* implement tee-ing recached content

* add thoughts of dog to test fedi attachments

* test get remote files

* add comment on PruneRemote

* add postData cleanup to recache

* test thumbnail fetching

* add incredible diagram

* go mod tidy

* buffer pipes for recache streaming

* test for client stops reading after 1kb

* add media-remote-cache-days to config

* add cron package

* wrap logrus so it's available to cron

* start and stop cron jobs gracefully
2022-03-07 11:08:26 +01:00
tobi
67ac8db190
Upstep Go dependencies ()
* Upstep Go dependencies

* tiny linter fix

* Tidy
2021-12-12 15:47:51 +01:00
tobi
182b4eea73
Update dependencies () 2021-11-27 15:26:58 +01:00
tobi
8b7c3507fe
upstep bun to v1.0.14 () 2021-10-24 13:14:37 +02:00
tobi
9a53b1a8d1
upstep bun to v1.0.9 () 2021-09-29 15:09:45 +02:00
tobi
f2e5bedea6
migrate go version to 1.17 ()
* migrate go version to 1.17

* update contributing
2021-09-10 14:42:14 +02:00
kim (grufwub)
bdcc090851 update bun library -> v1.0.4
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-09-08 20:05:26 +01:00
kim
ed46224573
Add SQLite support, fix un-thread-safe DB caches, small performance f… ()
* Add SQLite support, fix un-thread-safe DB caches, small performance fixes

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add SQLite licenses to README

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* appease the linter, and fix my dumbass-ery

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* make requested changes

Signed-off-by: kim (grufwub) <grufwub@gmail.com>

* add back comment

Signed-off-by: kim (grufwub) <grufwub@gmail.com>
2021-08-29 16:41:41 +02:00
tobi
2dc9fc1626
Pg to bun ()
* start moving to bun

* changing more stuff

* more

* and yet more

* tests passing

* seems stable now

* more big changes

* small fix

* little fixes
2021-08-25 15:34:33 +02:00
Tobi Smethurst
98263a7de6
Grand test fixup ()
* start fixing up tests

* fix up tests + automate with drone

* fiddle with linting

* messing about with drone.yml

* some more fiddling

* hmmm

* add cache

* add vendor directory

* verbose

* ci updates

* update some little things

* update sig
2021-08-12 21:03:24 +02:00