* don't serve unused fields for video attachments
* parse video bitrate + duration more accurately
* use ServeContent where appropriate to respect Range
* abstract temp file seeker into its own function
* media processor consolidation and reformatting, reduce amount of required syscalls
Signed-off-by: kim <grufwub@gmail.com>
* update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling
Signed-off-by: kim <grufwub@gmail.com>
* fix duration not being set, fix mp4 test expecting error
Signed-off-by: kim <grufwub@gmail.com>
* fix test expecting media files with different extension
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code
Signed-off-by: kim <grufwub@gmail.com>
* fix expected storage paths in tests, update expected test thumbnails
Signed-off-by: kim <grufwub@gmail.com>
* remove dead code
Signed-off-by: kim <grufwub@gmail.com>
* fix cached presigned s3 url fetching
Signed-off-by: kim <grufwub@gmail.com>
* fix tests
Signed-off-by: kim <grufwub@gmail.com>
* fix test models
Signed-off-by: kim <grufwub@gmail.com>
* update media processing to use sync.Once{} for concurrency protection
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* fix passing in KVStore GetStream() as stream to PutStream()
Signed-off-by: kim <grufwub@gmail.com>
* fix unlocks of storage keys
Signed-off-by: kim <grufwub@gmail.com>
* whoops, return the error...
Signed-off-by: kim <grufwub@gmail.com>
* pour one out for tobi's code <3
Signed-off-by: kim <grufwub@gmail.com>
* add back the byte slurping code
Signed-off-by: kim <grufwub@gmail.com>
* check for both ErrUnexpectedEOF and EOF
Signed-off-by: kim <grufwub@gmail.com>
* add back links to file format header information
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* Add local user and post count to nodeinfo responses
This fixes#1307 (at least partially). The nodeinfo endpoint should now
return the total users on an instance, along with their post count.
* Update NodeInfoUsers docstring and swagger yaml file
* launch websocket streaming in goroutine to allow upgrade handler to return
* don't send any message on ping, improved close check on failed read
* use context to signal wsconn close, ensure canceled in read goroutine
Signed-off-by: kim <grufwub@gmail.com>
* interim commit: start refactoring middlewares into package under router
* another interim commit, this is becoming a big job
* another fucking massive interim commit
* refactor bookmarks to new style
* ambassador, wiz zeze commits you are spoiling uz
* she compiles, we're getting there
* we're just normal men; we're just innocent men
* apiutil
* whoopsie
* i'm glad noone reads commit msgs haha :blob_sweat:
* use that weirdo go-bytesize library for maxMultipartMemory
* fix media module paths
* close pipereader on failed data function
* gently slurp the bytes
* readability updates
* go fmt
* tidy up file server tests + add more cases
* start moving io wrappers to separate iotools package. Remove use of buffering while piping recache stream
Signed-off-by: kim <grufwub@gmail.com>
* add license text
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* [chore] Move ShortcodeDomain to its own little util func
* [feature] Add RefetchEmojis function to media manager
* [feature] Expose admin media refresh via admin API
* update following review feedback
- change/fix log levels
- make sure not to try to refetch local emojis
- small style refactoring + comments
* log on emoji refetch start
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* Implement Bookmarks
* Update based on review comments
* Update swagger doc
* Fix argument passing to status.Bookmark
* Update changed test
* Updates based on latest PR review
* [feature] overhaul the oidc system
this allows for more flexible username handling and prevents account
takeover using old email addresses
* [feature] add migration path for old OIDC users
* [feature] nicer error reporting for users
* [docs] document the new OIDC flow
* [fix] return early on oidc error
* [docs]: add comments on the finalization logic
* start refactoring some of the search + deref logic
* add tests for search api
* rename GetRemoteAccount + GetRemoteStatus
* make search function a bit simpler + clearer
* fix little fucky wucky uwu owo i'm just a little guy
* update faulty switch statements
* update test to use storage struct
* redo switches for clarity
* reduce repeated logic in search tests
* fastfail getstatus by uri
* debug log + trace log better
* add implementation note
* return early if no result for namestring search
* return + check on dereferencing error types
* errors hah what errors
* remove unneeded error type alias, add custom error text during stringification itself
* fix a woops recursion 🙈
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* start adding admin emoji PATCH stuff
* updating works OK, now how about copying
* allow emojis to be copied
* update swagger docs
* update admin processer to use non-interface storage driver
* remove shortcode updating for local emojis
* go fmt
Co-authored-by: f0x52 <f0x@cthu.lu>
* add account block DB cache and remove reliance on relational joins
* actually include cache key arguments...
* add a PutBlock() method which also updates the block cache, update tests accordingly
* use `PutBlock` instead of `Put(ctx, block)`
* add + use functions for deleting + invalidating blocks
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
Google abandoned Floc in favour of Topics and changed the
necessary permissions policy. Currently, the Floc policy will
block Topics. This change includes switching to the updated
policy to be ahead of Google abandoning recognising the Floc
policy.
Update the function documentation to include the current relevant
documentation.
* convert most of the caches to use result.Cache{}
* add caching of emojis
* fix issues causing failing tests
* update go-cache/v2 instances with v3
* fix getnotification
* add a note about the left-in StatusCreate comment
* update EmojiCategory db access to use new result.Cache{}
* fix possible panic in getstatusparents
* further proof that kim is not stinky
* [feature] Add 'role' field to api serialization of local accounts
* [chore] Add a bit of license text while I'm here
* [frogend] render account role on same line as username in web view of profile
* style tweaking on role badges, general profile header layout
* profile stats wrapping
* don't render standard 'user' role on web view
Co-authored-by: f0x <f0x@cthu.lu>
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API
* [feature] Create (or use existing) category for new emoji uploads
* fix lint issue
* update misleading line in swagger docs
* Add instance-expose-public-timeline flag
Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.
* Update structure following review
* Add comment
* Fix linting
* Fix login on Mastodon iOS app for users with no statuses
Mastodon for iOS can't cope with an empty string for a date and
expect a JSON `null` instead.
Fixes https://github.com/superseriousbusiness/gotosocial/issues/1010
* Fix expected values in tests to match
* [feature] Make instance thumbnail configurable via admin panel
* log db errors in InstanceToAPIInstance
* only update instance in db if necessary
* start adding tests
* finish test
* re-add eslint
* fix oauth url getting too long
* actually attach single emoji get and delete routes
* basic emoji details + deletion using rtk query
* refactor emoji upload to rtk query
* clean up old redux api+reducers for custom emoji
* fix validation order
* refactor custom emoji form fields
* remove unused requires
* cleanup, fix most eslint errors
* more small eslint fixes
* fix max emoji size
* tiny bit of function documentation
* use readcloser for content.Content
* call media postdata function no matter what
* return a readcloser from data func
* tidy of logic of readertostore
* fix whoopsie
* select emoji using image_static_url
* use updated on AP emojis
* allow refetch of updated emojis
* cheeky workaround for test
* clean up old files for refreshed emoji
* check error for originalPostData
* shorten GetEmojiByStaticImageURL
* delete kirby (sorry nintendo)
* add admin emojis get path + model + docs
* stub admin emojis get processor function
* add id + disabled fields to admin emoji
* add emoji -> api admin emoji converter
* tidy up a bit
* add GetEmojis function
* finish up get emojis function
* order by shortcodedomain
* ASC
* tidy up + explain
* update to allow paging
* make admin emojis pageable
* fix mixed case paging
* normalize emoji queries a bit better
* test emoji get paging
* make limit optional
* fix incorrect path in media cleanup tests
* i have bad coder syndrome
* don't trimspace
* rename -> GetUseableEmojis
* wrap emoji query in subquery
avoid selecting more than we need
* fix a bit of sillyness teehee
* fix subquery postgres woes
* start adding rss functionality
* add gorilla/feeds dependency
* first bash at building rss feed
still needs work, this is an interim commit
* tidy up a bit
* add publicOnly option to GetAccountLastPosted
* implement rss endpoint
* fix test
* add initial user docs for rss
* update rss logo
* docs update
* add rssFeed to frontend
* feed -> feed.rss
* enableRSS
* increase rss logo size a lil bit
* add rss toggle
* move emojify to text package
* fiddle with rss feed formatting
* add Text field to test statuses
* move status to rss item to typeconverter
* update bun schema for enablerss
* simplify 304 checking
* assume account not rss
* update tests
* update swagger docs
* allow more characters in title, trim nicer
* update last posted to be more consistent
* vastly simplify logging caller information
Signed-off-by: kim <grufwub@gmail.com>
* fix failing test due to multiple calls to processor.Start()
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* update dependencies, bump Go version to 1.19
* bump test image Go version
* update golangci-lint
* update gotosocial-drone-build
* sign
* linting, go fmt
* update swagger docs
* update swagger docs
* whitespace
* update contributing.md
* fuckin whoopsie doopsie
* linterino, linteroni
* fix followrequest test not starting processor
* fix other api/client tests not starting processor
* fix remaining tests where processor not started
* bump go-runners version
* don't check last-webfingered-at, processor may have updated this
* update swagger command
* update bun to latest version
* fix embed to work the same as before with new bun
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* Start adding account emoji
* get emojis serialized + deserialized nicely
* update tests
* set / retrieve emojis on accounts
* show account emojis in web view
* fetch emojis from db based on ids
* fix typo in test
* lint
* fix pg migration
* update tests
* update emoji checking logic
* update comment
* clarify comments + add some spacing
* tidy up loops a lil (thanks kim)
* log internal server errors from 500 api calls
* don't exec into nil dest
* don't exec into nil dest
* log error in router logger not api errorhandling
* update logging a tad
* linter
* add custom css account property + db func to fetch
* allow account to get/set custom css
* serve custom css for an account
* go fmt
* use monospace for customcss, add link
* add custom css to account cache
* fix broken field
* add custom css docs to user guide
* add `accounts-allow-custom-css` config flag
* add allow custom css to /api/v1/instance response
* only show/set custom css if allowed to do so
* only set/serve custom account css if enabled
* update swagger docs
* chain promise
* make bool a bit clearer
* use cache for GetAccountCustomCSSByUsername
* start implementing remote emoji fetcher
* update status where pk
* aaa
* tidy up a little
* check size limits for emojis
* thank you linter, i love you <3
* update swagger docs
* add emoji dereference test
* make emoji max sizes configurable
* normalize db.ErrAlreadyExists
* migrate emojis
* add get emoji to s2s (federation) API
* add new emoji db + cache functions
* add shortcodeDomain lookup for emojis
* check existing emojis w/cache, not w/constraints
* go fmt
* add putEmoji func
* use new db emoji funcs instead of where
* remove emojistringstotags func
* add unique constraint back in
* fix up broken migration
* update index
* validate web-asset-base-dir
* move default icons into converter
* always ensure avatar + header on api accounts
* update tests
* add default header
* don't return error from web module creation anymore
* tidy a bit
* use pngs for default avatars rather than svgs
* use 'test' value for testrig storage backend
* update test dependency
* add WaitFor func in testrig
* use WaitFor function instead of time.Sleep
* tidy up tests
* make SentMessages a sync.map
* go fmt
* upstep bun and sqlite versions
* allow specific columns to be updated in the db
* only update necessary columns for user
* bit tidier
* only update necessary fields of media_attachment
* only update relevant instance fields
* update tests
* update only specific account columns
* use bool pointers on gtsmodels
includes attachment, status, account, user
* update columns more selectively
* test all default fields on new account insert
* updating remaining bools on gtsmodels
* initialize pointer fields when extracting AP emoji
* copy bools properly
* add copyBoolPtr convenience function + test it
* initialize false bool ptrs a bit more neatly
* start work on user panel
* parse source first before checking if empty form
* newline
* set avi + header nicely
* add posts settings
* render signin a bit nicer on mobile
* return OK json on successful change
* return unauthorized on bad password
* clarify message on insecure password
* make login a bit prettier
* add alt text + border round image previews
* add logout button
* add password change
* styling updates
* redirect /auth/edit to /user
* update tests
* fix validation tests
* better labels, link to more info
* make submit button generic component
* move submit button inside forms
* add autocomplete labels to password fields
* fix indentation (thx eslint)
* update eslintrc
* eslint: no-unescaped-entities
* initial deduplication between user and admin panel
* add default status/post format setting
* user panel styling for inputs
* update user panel styling, include normalize css
* add placeholder text
* input padding
Co-authored-by: f0x <f0x@cthu.lu>
* add post_format to acct & use it when making post
* update swagger docs
* add status_format updating to frontend
* fix up tests
* post_format => status_format
* add status_format to account validation
* first commit
Signed-off-by: kim <grufwub@gmail.com>
* replace logging with our own log library
Signed-off-by: kim <grufwub@gmail.com>
* fix imports
Signed-off-by: kim <grufwub@gmail.com>
* fix log imports
Signed-off-by: kim <grufwub@gmail.com>
* add license text
Signed-off-by: kim <grufwub@gmail.com>
* fix package import cycle between config and log package
Signed-off-by: kim <grufwub@gmail.com>
* fix empty kv.Fields{} being passed to WithFields()
Signed-off-by: kim <grufwub@gmail.com>
* fix uses of log.WithFields() with whitespace issues and empty slices
Signed-off-by: kim <grufwub@gmail.com>
* *linter related grumbling*
Signed-off-by: kim <grufwub@gmail.com>
* gofmt the codebase! also fix more log.WithFields() formatting issues
Signed-off-by: kim <grufwub@gmail.com>
* update testrig code to match new changes
Signed-off-by: kim <grufwub@gmail.com>
* fix error wrapping in non fmt.Errorf function
Signed-off-by: kim <grufwub@gmail.com>
* add benchmarking of log.Caller() vs non-cached
Signed-off-by: kim <grufwub@gmail.com>
* fix syslog tests, add standard build tags to test runner to ensure consistency
Signed-off-by: kim <grufwub@gmail.com>
* make syslog tests more robust
Signed-off-by: kim <grufwub@gmail.com>
* fix caller depth arithmatic (is that how you spell it?)
Signed-off-by: kim <grufwub@gmail.com>
* update to use unkeyed fields in kv.Field{} instances
Signed-off-by: kim <grufwub@gmail.com>
* update go-kv library
Signed-off-by: kim <grufwub@gmail.com>
* update libraries list
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter get nerfed
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
* add GetAccountWebStatuses to db
* add WebStatusesGet func to processor
* don't add limit to next/prev links if 0
* take query params for next/prev statuses
* add separate next + prev links for convenience
* show 'nothing here' message if no statuses exist
* add back / next links to profiles
* allow paging down only
* go fmt ./...
* 'recent public toots' -> 'latest public toots'
* move panic recovery to logging middleware, improve logging + panic recovery logic
Signed-off-by: kim <grufwub@gmail.com>
* remove dead code
Signed-off-by: kim <grufwub@gmail.com>
* remove skip paths code
Signed-off-by: kim <grufwub@gmail.com>
* re-enable log quoting
Signed-off-by: kim <grufwub@gmail.com>
* use human-friendly bytesize in logging body size
Signed-off-by: kim <grufwub@gmail.com>
* only disable quoting in debug builds
Signed-off-by: kim <grufwub@gmail.com>
* use logrus level instead of debug.DEBUG() to enable/disable quoting
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* fix instance tests
Signed-off-by: kim <grufwub@gmail.com>
* fix gin test contexts created with missing engine HTML renderer
Signed-off-by: kim <grufwub@gmail.com>
* add note regarding not logging query parameters
Signed-off-by: kim <grufwub@gmail.com>
* better explain 'DisableQuoting' logic
Signed-off-by: kim <grufwub@gmail.com>
* add license text
Signed-off-by: kim <grufwub@gmail.com>
* add miekg/dns dependency
* set/validate accountDomain
* move finger to dereferencer
* totally break GetRemoteAccount
* start reworking finger func a bit
* start reworking getRemoteAccount a bit
* move mention parts to namestring
* rework webfingerget
* use util function to extract webfinger parts
* use accountDomain
* rework finger again, final form
* just a real nasty commit, the worst
* remove refresh from account
* use new ASRepToAccount signature
* fix incorrect debug call
* fix for new getRemoteAccount
* rework GetRemoteAccount
* start updating tests to remove repetition
* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).
* fix up tests to use main mock httpclient
* webfinger only if necessary
* cheeky linting with the lads
* update mentionName regex
recognize instance accounts
* don't finger instance accounts
* test webfinger part extraction
* increase default worker count to 4 per cpu
* don't repeat regex parsing
* final search for discovered accountDomain
* be more permissive in namestring lookup
* add more extraction tests
* simplify GetParseMentionFunc
* skip long search if local account
* fix broken test
* panics get logged at error level, now include stacktrace
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* feat: vendor minio client
* feat: introduce storage package with s3 support
* feat: serve s3 files directly
this saves a lot of bandwith as the files are fetched from the object
store directly
* fix: use explicit local storage in tests
* feat: integrate s3 storage with the main server
* fix: add s3 config to cli tests
* docs: explicitly set values in example config
also adds license header to the storage package
* fix: use better http status code on s3 redirect
HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL
307/TemporaryRedirect would mean that this resource is usually located
here, not in this case
303/SeeOther indicates that the redirection does not link to the
requested resource but to another page
* refactor: use context in storage driver interface
* add configuration object to api instance model
* regenerate swagger docs
* add func to return all supported mimes for media
* add instance configuration to api serialization
* fix json tags
* update instance endpoint tests
* fix typeutils tests
* final regen of swagger docs
* omitempty instance configuration
* add missing license headers
* start adding instance peers get
* rename domainblock.go
* embed domain in domainblock so it can be reused
* update swagger docs
* add test instances to db
* update tests
* add/update instancepeersget
* update domain model
* add getinstancepeers to db
* instance-expose-peers, instance-expose-suspended
* add auth checks for both current filters
* attach endpoint to router
* include public comment
* obfuscate domain if required
* go mod tidy
* update swagger docs
* remove unnecessary comment
* return 'flat' peerlist if no query params provided
* move finger to dereferencer
* totally break GetRemoteAccount
* start reworking finger func a bit
* start reworking getRemoteAccount a bit
* move mention parts to namestring
* rework webfingerget
* use util function to extract webfinger parts
* use accountDomain
* rework finger again, final form
* just a real nasty commit, the worst
* remove refresh from account
* use new ASRepToAccount signature
* fix incorrect debug call
* fix for new getRemoteAccount
* rework GetRemoteAccount
* start updating tests to remove repetition
* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).
* fix up tests to use main mock httpclient
* webfinger only if necessary
* cheeky linting with the lads
* update mentionName regex
recognize instance accounts
* don't finger instance accounts
* test webfinger part extraction
* increase default worker count to 4 per cpu
* don't repeat regex parsing
* final search for discovered accountDomain
* be more permissive in namestring lookup
* add more extraction tests
* simplify GetParseMentionFunc
* skip long search if local account
* fix broken test
* start fiddling about with oauth server
* start returning more helpful errors from oauth
* test helpful(ish) token errors
* add missing license header
* update templates
* start reworking api error handling
* update template
* return AP status at web endpoint if negotiated
* start making api error handling much more consistent
* update account endpoints to new error handling
* use new api error handling in admin endpoints
* go fmt ./...
* use api error logic in app
* use generic error handling in auth
* don't export generic error handler
* don't defer clearing session
* user nicer error handling on oidc callback handler
* tidy up the sign in handler
* tidy up the token handler
* use nicer error handling in blocksget
* auth emojis endpoint
* fix up remaining api endpoints
* fix whoopsie during login flow
* regenerate swagger docs
* change http error logging to debug
* test link headers
* page get account statuses properly
* page get notifications
* add util func for packaging timeline responses
* return timelined stuff from accountstatusesget
* rename timeline response
* use new convenience function
* go fmt
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct
Signed-off-by: kim <grufwub@gmail.com>
* improved code comment
Signed-off-by: kim <grufwub@gmail.com>
* linter
Signed-off-by: kim <grufwub@gmail.com>
* fix unmarshaling
Signed-off-by: kim <grufwub@gmail.com>
* remove kim's custom go compiler changes
Signed-off-by: kim <grufwub@gmail.com>
* generate setter and flag-name functions, implement these in codebase
Signed-off-by: kim <grufwub@gmail.com>
* update deps
Signed-off-by: kim <grufwub@gmail.com>
* small change
Signed-off-by: kim <grufwub@gmail.com>
* appease the linter...
Signed-off-by: kim <grufwub@gmail.com>
* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync
Signed-off-by: kim <grufwub@gmail.com>
* lint
Signed-off-by: kim <grufwub@gmail.com>
* update code comments
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* improved version string (removes time + go version)
Signed-off-by: kim <grufwub@gmail.com>
* fix version string build to pass test script + consolidate logic in func
Signed-off-by: kim <grufwub@gmail.com>
* add license text, update config.Defaults comment
Signed-off-by: kim <grufwub@gmail.com>
* add license text to generated config helpers file
Signed-off-by: kim <grufwub@gmail.com>
* defer unlock on config.Set___(), to ensure unlocked on panic
Signed-off-by: kim <grufwub@gmail.com>
* make it more obvious which cmd flags are being attached
Signed-off-by: kim <grufwub@gmail.com>
* don't omitempty on description
* don't omitempty on any fields
* add ms to timestamp format
* don't omitempty on text_url
* rearrange attachment fields a bit
* just give URL again as attachment text url
* update tests
* fix accidental replace
* move caption sanitization -> sanitize.go
* use sanitizeplaintext rather than removehtml
* rename sanitizecaption to sanitizeplaintext
* avoid removing html twice from statuses
* unexport remoteHTML
it's no longer used outside the text package so this
makes it less confusing
* test instance PATCH
* add some announces to test models
* start on announce test logic
* test federatingDB.Announce
* change signature of GetRemoteStatus
* remove 'refresh' logic and replace it with refetch
* go fmt
* remove timeline manager from processor test
* make zork created at determinate
* test get account statuses
* test get + serialize zork
* make account keys determinate
* make admin accountCreate time determinate
* test account to as
* init test config before test log
* test status to frontend
* remove daft Within check
* hack around a bit
* use index of slice
* implement custom_emojis api endpoint
* add tests for getting custom emoji out of the database and converting to api emoji
* change sort direction of emoji query
* change logging level and initialize array with known length as per kim's suggestions
* add continue to lessen risk of making a malformed struct during conversion from db to api emojis
* rename + tidy up remote pruning
* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships
* update logging to new function name
* add get avatars and headers to bun
* add pruneallmeta function
* don't set uncached since we're deleting anyway
* fix totalPruned being in wrong place
* test pruning meta
* go fmt ./...
* rename mediaprune
* add meta pruning to routine mediaprune
* tidy up cleanup job scheduling
* rename adminmediaremoteprune
* update mediacleanup to use renamed prune func
* update swagger docs a little bit
* reuse cancel + context
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests
Signed-off-by: kim <grufwub@gmail.com>
* update code comments, defer mutex unlocks
Signed-off-by: kim <grufwub@gmail.com>
* add count to 'performing request' log message
Signed-off-by: kim <grufwub@gmail.com>
* reduce repeated conversions of same url.URL object
Signed-off-by: kim <grufwub@gmail.com>
* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue
Signed-off-by: kim <grufwub@gmail.com>
* fix security advisories regarding max outgoing conns, max rsp body size
- implemented by a new httpclient.Client{} that wraps an underlying
client with a queue to limit connections, and limit reader wrapping
a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}
Signed-off-by: kim <grufwub@gmail.com>
* add httpclient tests, move ip validation to separate package + change mechanism
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* use singular mutex in transport rather than separate signer mus
Signed-off-by: kim <grufwub@gmail.com>
* improved useragent string
Signed-off-by: kim <grufwub@gmail.com>
* add note regarding missing test
Signed-off-by: kim <grufwub@gmail.com>
* remove useragent field from transport (instead store in controller)
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* reset other signing headers on each loop iteration
Signed-off-by: kim <grufwub@gmail.com>
* respect request ctx during retry-backoff sleep period
Signed-off-by: kim <grufwub@gmail.com>
* use external pkg with docs explaining performance "hack"
Signed-off-by: kim <grufwub@gmail.com>
* use http package constants instead of string method literals
Signed-off-by: kim <grufwub@gmail.com>
* add license file headers
Signed-off-by: kim <grufwub@gmail.com>
* update code comment to match new func names
Signed-off-by: kim <grufwub@gmail.com>
* updates to user-agent string
Signed-off-by: kim <grufwub@gmail.com>
* update signed testrig models to fit with new transport logic (instead uses separate signer now)
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter
Signed-off-by: kim <grufwub@gmail.com>
* fix error check of prune to allow missing files
* update go-store library, add test for pruning item with db entry but no file
Signed-off-by: kim <grufwub@gmail.com>
* remove now-unneccessary error check
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* fix existing bio text showing as HTML
- updated replaced mentions to include instance
- strips HTML from account source note in Verify handler
- update text formatter to use buffers for string writes
Signed-off-by: kim <grufwub@gmail.com>
* go away linter
Signed-off-by: kim <grufwub@gmail.com>
* change buf reset location, change html mention tags
Signed-off-by: kim <grufwub@gmail.com>
* reduce FindLinks code complexity
Signed-off-by: kim <grufwub@gmail.com>
* fix HTML to text conversion
Signed-off-by: kim <grufwub@gmail.com>
* Update internal/regexes/regexes.go
Co-authored-by: Mina Galić <mina.galic@puppet.com>
* use improved html2text lib with more options
Signed-off-by: kim <grufwub@gmail.com>
* fix to produce actual plaintext from html
Signed-off-by: kim <grufwub@gmail.com>
* fix span tags instead written as space
Signed-off-by: kim <grufwub@gmail.com>
* performance improvements to regex replacements, fix link replace logic for un-html-ing in the future
Signed-off-by: kim <grufwub@gmail.com>
* fix tag/mention replacements to use input string, fix link replace to not include scheme
Signed-off-by: kim <grufwub@gmail.com>
* use matched input string for link replace href text
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code (to appease linter :sobs:)
Signed-off-by: kim <grufwub@gmail.com>
* improve hashtagFinger regex to be more compliant
Signed-off-by: kim <grufwub@gmail.com>
* update breakReplacer to include both unix and windows line endings
Signed-off-by: kim <grufwub@gmail.com>
* add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts
Signed-off-by: kim <grufwub@gmail.com>
* drop unnecessary code
Signed-off-by: kim <grufwub@gmail.com>
* update text package tests to fix logic changes
Signed-off-by: kim <grufwub@gmail.com>
* add raw note content testing to account update and account verify
Signed-off-by: kim <grufwub@gmail.com>
* remove unused modules
Signed-off-by: kim <grufwub@gmail.com>
* fix emoji regex
Signed-off-by: kim <grufwub@gmail.com>
* fix replacement of hashtags
Signed-off-by: kim <grufwub@gmail.com>
* update code comment
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: Mina Galić <mina.galic@puppet.com>
* remove unnecessary LOWER() db calls
* warn during slow db queries
* use bundb built-in exists function
* add db block test
* update account block query
* add domain block db test
* optimize domain block query
* fix implementing wrong test
* exclude most columns when checking block
* go fmt
* remote more unnecessary use of LOWER()
* tidy up some federation tests
* add missing license to following.go
* give zork some followers, as a treat
* test InboxesForIRI
* Go fmt
* update timeline tests
* bump superseriousbusiness/activity version
* fetch list of follower inbox iris
* add test for getting account statuses
* fix incorrect publicOnly parsing
* update media not null query
* remove mention util function from db
* add ParseMentionFunc to gtsmodel
* add parseMentionFunc to processor
* refactor search to simplify it a bit
* add parseMentionFunc to account
* add parseMentionFunc to status
* some renaming for clarity
* test dereference of unknown mentioned account
* add accountAction to the admin API
* model admin account action
* add admin account action to the processor
* add migration for new AdminAccountActions table
* fix accounts admin path
* Update swagger docs
* Add whereNotEmptyAndNotNull
* Add GetRemoteOlderThanDays
* Add GetRemoteOlderThanDays
* Add PruneRemote to Manager interface
* Start implementing PruneRemote
* add new attachment + status to tests
* fix up and test GetRemoteOlderThan
* fix bad import
* PruneRemote: return number pruned
* add Cached column to mediaattachment
* update + test pruneRemote
* update mediaTest
* use Cached column
* upstep bun to latest version
* embed structs in mediaAttachment
* migrate mediaAttachment to new format
* don't default cached to true
* select only remote media
* update db dependencies
* step bun back to last working version
* update pruneRemote to use Cached field
* fix storage path of test attachments
* add recache logic to manager
* fix trimmed aspect ratio
* test prune and recache
* return errwithcode
* tidy up different paths for emoji vs attachment
* fix incorrect thumbnail type being stored
* expose TransportController to media processor
* implement tee-ing recached content
* add thoughts of dog to test fedi attachments
* test get remote files
* add comment on PruneRemote
* add postData cleanup to recache
* test thumbnail fetching
* add incredible diagram
* go mod tidy
* buffer pipes for recache streaming
* test for client stops reading after 1kb
* add media-remote-cache-days to config
* add cron package
* wrap logrus so it's available to cron
* start and stop cron jobs gracefully
* add ensureUserIsAuthorizedOrRedirect to /oauth/authorize
* adding authorization (email confirm, account approve, etc) to TokenCheck
* revert un-needed changes to signin.go
* oops what happened here
* error css
* add account.SuspendedAt check
* remove redundant checks from oauth util Authed function
* wip tests
* tests passing
* stop stripping useful information from ErrAlreadyExists
* that feeling of scraping the dryer LINT off the screen
* oops I didn't mean to get rid of this NewTestRouter function
* make tests work with recorder
* re-add ConfigureTemplatesWithGin to handle template path err
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* add preparable and timelineable interfaces
* initialize timeline manager within the processor
* generic renaming
* move status-specific timeline logic into the processor
* refactor timeline to make it useful for more than statuses
* start centralizing negotiation logic for API
* swagger document nodeinfo endpoint
* go fmt
* document negotiate function
* use content negotiation
* tidy up negotiation logic
* negotiate content throughout client api
* swagger
* remove attachment on Content
* add accept header to test requests
* start pulling out + replacing urfave and config
* replace many many instances of config
* move more stuff => viper
* properly remove urfave
* move some flags to root command
* add testrig commands to root
* alias config file keys
* start adding cli parsing tests
* reorder viper init
* remove config path alias
* fmt
* change config file keys to non-nested
* we're more or less in business now
* tidy up the common func
* go fmt
* get tests passing again
* add note about the cliparsing tests
* reorganize
* update docs with changes
* structure cmd dir better
* rename + move some files around
* fix dangling comma
* move token checker to security package
* update tests with new security package
* add oauth token checking to security package
* check if user email confirmed when parsing token
* start with a default config, not an empty config.
* some data structures were present on Empty config but not Default config
* the monkey patched CLIContext is working
* remove print debugging log
* make the behaviour of the flags consistent across all data types
Conflicts:
internal/config/config.go
* try to fix accidentally broken test
* add smtp configuration
* add email confirm + reset templates
* add email sender to testrig
* flesh out the email sender interface
* go fmt
* golint
* update from field with more clarity
* tidy up the email formatting
* fix tests
* add email sender to processor
* tidy client api processing a bit
* further tidying in fromClientAPI
* pin new account to user
* send msg to processor on new account creation
* generate confirm email uri
* remove emailer from account processor again
* add processCreateAccountFromClientAPI
* move emailer accountprocessor => userprocessor
* add email sender to user processor
* SendConfirmEmail function
* add noop email sender
* use noop email sender in tests
* only assemble message if callback is not nil
* use noop email sender if no smtp host is defined
* minify email html before sending
* fix wrong email address
* email confirm test
* fmt
* serve web hndler
* add email confirm handler
* init test log properly on testrig
* log emails that *would* have been sent
* go fmt ./...
* unexport confirm email handler
* updatedAt
* test confirm email function
* don't allow tokens older than 7 days
* change error message a bit
* add basic smtp docs
* add a few more snippets
* typo
* add email sender to outbox tests
* don't use dutch wikipedia link
* don't minify email html
* add statusesvisible convenience function
* add minID + onlyPublic to account statuses get
* move swagger collection stuff to common
* start working on Outbox GETting
* move functions into federationProcessor
* outboxToASCollection
* add statusesvisible convenience function
* add minID + onlyPublic to account statuses get
* move swagger collection stuff to common
* start working on Outbox GETting
* move functions into federationProcessor
* outboxToASCollection
* bit more work on outbox paging
* wrapNoteInCreate function
* test + hook up the processor functions
* don't do prev + next links on empty reply
* test get outbox through api
* don't fail on no status entries
* add outbox implementation doc
* typo
* reference logrus' global logger instead of passing and storing a logger reference everywhere
* always directly use global logrus logger instead of referencing an instance
* test suites should also directly use the global logrus logger
* rename gin logging function to clarify that it's middleware
* correct comments which erroneously referenced removed logger parameter
* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
* use pub public const
* don't error on no summary
* move extract visibility to separate function
* extract visibility test
* add addressable interface
* tidy up streaming
* cut down code duplication
* test get followers/following
* test streaming processor
* fix some test models
* add TimeMustParse
* fix uri / url typo
* make trace logging less verbose
* make logging more consistent
* disable quote on logging
* remove context.Background
* remove many extraneous mastodon references
* regenerate swagger
* don't log query on no rows result
* log latency first for easier reading
* only fetch status parents / children if explicity requested when dereferencing
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* Remove recursive DB GetStatus logic, don't fetch parent unless requested
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* StatusCache copies status so there are no thread-safety issues with modified status objects
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* remove sqlite test files
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* fix bugs introduced by previous commit
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* fix not continue on error in loop
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* use our own RunInTx implementation (possible fix for nested tx error)
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* fix cast statement to work with SQLite
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* be less strict about valid status in cache
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add cache=shared ALWAYS for SQLite db instances
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* Fix EnrichRemoteAccount when updating account fails
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add nolint tag
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* ensure file: prefixes the filename in sqlite addr
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add an account cache, add status author account from db
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* Fix incompatible SQLite query
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* *actually* use the new getAccount() function in accountsDB
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* update cache tests to use test suite
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add RelationshipTestSuite, add tests for methods with changed SQL
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* Add SQLite support, fix un-thread-safe DB caches, small performance fixes
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add SQLite licenses to README
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* appease the linter, and fix my dumbass-ery
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* make requested changes
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add back comment
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* start moving some database stuff around
* continue moving db stuff around
* more fiddling
* more updates
* and some more
* and yet more
* i broke SOMETHING but what, it's a mystery
* tidy up
* vendor ttlcache
* use ttlcache
* fix up some tests
* rename some stuff
* little reminder
* some more updates
* start fixing up tests
* fix up tests + automate with drone
* fiddle with linting
* messing about with drone.yml
* some more fiddling
* hmmm
* add cache
* add vendor directory
* verbose
* ci updates
* update some little things
* update sig
* decided where to put reply dereferencing
* fiddling with dereferencing threads
* further adventures
* tidy up some stuff
* move dereferencing functionality
* a bunch of refactoring
* go fmt
* more refactoring
* bleep bloop
* docs and linting
* start implementing replies collection on gts side
* fiddling around
* allow dereferencing our replies
* lint, fmt
* start experimenting with swagger documentation
* further adventures in swagger
* do a few more api paths
* account paths documented
* go fmt
* fix up some models
* bit o lintin'
* start work on admin domain blocking
* move stuff around + further work on domain blocks
* move + restructure processor
* prep work for deleting account
* tidy
* go fmt
* formatting
* domain blocking more work
* check domain blocks way earlier on
* progress on delete account
* delete more stuff when an account is gone
* and more...
* domain blocky block block
* get individual domain block, delete a block
* start messing about with timeline manager
* i have no idea what i'm doing
* i continue to not know what i'm doing
* it's coming along
* bit more progress
* update timeline with new posts as they come in
* lint and fmt
* Select accounts where empty string
* restructure a bunch, get unfaves working
* moving stuff around
* federate status deletes properly
* mention regex better but not 100% there
* fix regex
* some more hacking away at the timeline code phew
* fix up some little things
* i can't even
* more timeline stuff
* move to ulid
* fiddley
* some lil fixes for kibou compatibility
* timelines working pretty alright!
* tidy + lint
1. Proper DELETE of federated statuses (not yet deleting all the media and stuff -- i still have to implement this -- but the actual status is toast).
2. Proper UPDATE of profiles. When you change your profile picture on your remote instance, that will now register properly in GoToSocial.
3. Scrolling down the home timeline - it no longer just sort of ends, and will keep loading older statuses as you scroll.
4. Little bugfixes -- still had some nil pointer errors when dereferencing remote accounts.
Remote media is now dereferenced and attached properly to incoming federated statuses.
Mentions are now dereferenced and attached properly to incoming federated statuses.
Small fixes to status visibility.
Allow URL params for filtering statuses:
// ExcludeRepliesKey is for specifying whether to exclude replies in a list of returned statuses by an account.
// PinnedKey is for specifying whether to include pinned statuses in a list of returned statuses by an account.
// MaxIDKey is for specifying the maximum ID of the status to retrieve.
// MediaOnlyKey is for specifying that only statuses with media should be returned in a list of returned statuses by an account.
Add endpoint for fetching an account's statuses.
Inbox POST from federated servers now working for statuses and follow requests.
Follow request client API added.
Start work on federating outgoing messages.
Other fixes and changes/tidying up.
* add host and protocol options
* some fiddling
* tidying up and comments
* tick off /oauth/token
* tidying a bit
* tidying
* go mod tidy
* allow attaching middleware to server
* add middleware
* more user friendly
* add comments
* comments
* store account + app
* tidying
* lots of restructuring
* lint + tidy
