Mirrors/ArchiveBox
2
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox synced 2024-11-23 04:33:11 +00:00
Code Issues Projects Releases Packages Wiki Activity
3883 commits 6 branches 78 tags 19 MiB
Commit graph

1496 commits

Author SHA1 Message Date
Nick Sweeting
33ec2117e9
Update main.py 2022-06-08 19:50:45 -07:00
Nick Sweeting
dd29e1bf78
clean up first line of CLI version output for easier downstream parsing 2022-06-08 19:46:09 -07:00
Nick Sweeting
0c6d4c82c3
Update config.py 2022-06-08 19:11:02 -07:00
Nick Sweeting
f9c5808940
Update config.py 2022-06-08 19:09:11 -07:00
Nick Sweeting
5509b5cd8b
Update main.py 2022-06-08 19:08:33 -07:00
Nick Sweeting
19b88d30b2
fix missing brace 2022-06-08 19:06:03 -07:00
Nick Sweeting
31d5fbbf17
Update config.py 2022-06-08 19:04:06 -07:00
Nick Sweeting
6b019da3e9
Update config.py 2022-06-08 19:01:55 -07:00
Nick Sweeting
c752c7053d
Update main.py 2022-06-08 18:59:08 -07:00
Nick Sweeting
f9c82841ad
fix sqlite option detection 2022-06-08 18:58:15 -07:00
Nick Sweeting
1fd5830f58
enforce UTC timezone on server 2022-06-08 18:41:22 -07:00
Nick Sweeting
3e3c011f86
enforce UTC timezone on server 2022-06-08 18:40:48 -07:00
Nick Sweeting
e06717419c
fix sqlite3 version detection 2022-06-08 18:35:31 -07:00
Nick Sweeting
d0f129295f
move sqlite3 checks up a level 2022-06-08 18:29:53 -07:00
Nick Sweeting
0c7d7deb32
add missing brace 2022-06-08 18:26:42 -07:00
Nick Sweeting
ca16c88a3d
show PUID, PGID, ENFORCE_ATOMIC_WRITES, and OUTPUT_PERMISSIONS in version output header 2022-06-08 18:24:58 -07:00
Nick Sweeting
89175ccb22
check SQLite3 version and enabled extensions on startup 2022-06-08 18:24:17 -07:00
Nick Sweeting
c245d36e44
add PUID and PGID as config options in archivebox 2022-06-08 17:42:52 -07:00
Nick Sweeting
c5fc3e1e65 --ammend 2022-05-09 23:59:27 -07:00
Nick Sweeting
0b4df768ba hack to check for generator type cause too lazy to import 2022-05-09 23:50:56 -07:00
Nick Sweeting
5e4ddbbf25 fix mercury bin parsing back 2022-05-09 21:58:17 -07:00
Nick Sweeting
e96c1bcf13 bump mercury parser to git head version 2022-05-09 21:48:41 -07:00
Nick Sweeting
d581a5081f correctly handle bytes strings in hints 2022-05-09 21:29:37 -07:00
Nick Sweeting
a6767671fb append content of referenced files to imports 2022-05-09 21:21:39 -07:00
Nick Sweeting
f6d6a06c78 always show all totals in log output 2022-05-09 21:21:26 -07:00
Nick Sweeting
d05510f844 fix version string parsing on macOS in some cases where LANG C is not supported 2022-05-09 21:21:08 -07:00
Nick Sweeting
4b8b17e788 add update flag support to archivebox schedule 2022-05-09 20:18:43 -07:00
Nick Sweeting
8cfe6f4afb cleanup update flag handling and show better logging to clarify when its working 2022-05-09 20:15:55 -07:00
Nick Sweeting
38e54b93fe allow parsing to continue even when fetching URL contents fails 2022-05-09 19:56:24 -07:00
Nick Sweeting
ecbcb6a1b3 fix bracing in template tag for PREVIEW_ORIGINALS 2022-05-09 19:56:08 -07:00
Nick Sweeting
8ebf3e2f93 add config option PREVIEW_ORIGINALS to hide original iframes in snapshot detail pages 2022-05-09 19:31:41 -07:00
Nick Sweeting
acd53c854d handle new wallabag export format with newlines mid-tag attributes 2022-05-09 19:07:48 -07:00
Nick Sweeting
950b5cbbb6
Merge pull request #924 from prnake/dev 
improve title extractor
 2022-05-09 18:38:12 -07:00
Nick Sweeting
6e66863871 add max 5s writing delay for concurrent writers and flush WAL slower 2022-05-09 18:36:40 -07:00
Nick Sweeting
57df65f28f use yt-dlp for media archiving instead of youtube-dl 2022-04-21 07:11:35 -07:00
Nick Sweeting
eb81d41f84 bump Dockerfile base image version and install yt-dlp 2022-04-21 07:11:35 -07:00
Ross
c63822a5e5
Fix missing input redirection in a hint text 2022-04-19 22:25:49 +01:00
Igor Rzegocki
d4f534e612
add LOGOUT_REDIRECT_URL 2022-03-31 21:40:14 +02:00
Pellaeon Lin
5e9d05483e Fix bin_version: set LANG=C when calling executables to avoid parsing 
localized output.
 2022-02-24 17:01:00 +08:00
prnake
011bd104cb
remove unused import 2022-02-09 10:48:51 +08:00
papersnake
de8e22efb7 improve title extractor 2022-02-08 23:17:52 +08:00
Nick Sweeting
666ab20df5
Update archivebox/config.py 2022-01-10 20:42:09 -05:00
hannah98
fc3d2bb4dc rename TAG_SEPARATORS to TAG_SEPARATOR_PATTERN 2022-01-06 14:14:41 +00:00
hannah98
049f88def9 Added TAG_SEPARATORS option to supply a regex of characters to use when splitting tags 2021-12-30 20:19:48 +00:00
Nick Sweeting
d7f01922f3
fix direct assignment of tags to many-to-many set 2021-12-23 12:29:17 -05:00
Nick Sweeting
b1b7ee2b85
Update sql.py 2021-12-23 12:17:55 -05:00
hannah98
4b8962b60b Fix #725 - correctly parse tags on json import 2021-12-20 08:58:58 -06:00
Mika Tuupola
f14a861605 Change logfile open to write mode only 2021-12-19 23:17:33 +02:00
TheCakeIsNaOH
decab91ea2
(#847) Decode error output hints to string if needed 2021-12-16 16:46:12 -06:00
Nick Sweeting
44f5338470
fix typo in pocket_api articl variable name 2021-11-12 19:23:47 -05:00
Nick Sweeting
8878dcc5e8
Merge pull request #843 from bltavares/patch-1 2021-11-12 15:57:19 -08:00
Nick Sweeting
d5240f1a1d
Merge pull request #885 from adamwolf/safari-admin-actions 2021-11-12 08:56:31 -08:00
Adam Wolf
18e1fb0d96 Fixes Add button behavior on Safari 
Previously, when you clicked the Add button, the page wouldn't change.

It looked like it wasn't doing anything, as noted by @rcarmo
(https://github.com/ArchiveBox/ArchiveBox/issues/658#issuecomment-948300055)

I didn't track it down the exact reason why. It may be that Safari
didn't like the two opening <h3>s, but I was able to find a bunch of
people complaining about Safari being very finicky with innerHTML
and actually repainting the page, enough that I decided to try just
extending the block hide/show behavior already done with the
delay-warning, and it works for me now in both Chrome and Safari.

For #658.
 2021-10-28 22:31:54 -05:00
Adam Wolf
83731f5a68 Tweak JS so Safari can choose admin actions 
I noticed that Safari was submitting both the empty option and the
selected options back to the server.

Digging into it, I was able to get Safari to deselect the ---------
option by using '[selected]' as the selector.

For #658
 2021-10-28 22:22:46 -05:00
Igor Rzegocki
05de1c9fe6
healthcheck endpoint 2021-10-03 19:12:03 +02:00
Igor Rzegocki
95cf85f8cf
Support for Reverse Proxy authentication backends (like authelia) 2021-09-30 17:40:13 +02:00
Bruno Tavares
bb2a2e758a
Avoid KeyError on Pocket API parser 
When trying to import my pocket library I got a lot of ` KeyError`  on Python. Pocket API has a few idiosyncrasies, such as sometimes returning the keys on json, sometimes not.

` ` ` sh
archivebox add --parser pocket_api pocket://my_username
` ` ` 

Gave me this errors
` ` ` 
  File "/app/archivebox/parsers/pocket_api.py", line 54, in link_from_article
    title = article['resolved_title'] or article['given_title'] or url
KeyError: 'resolved_title'
` ` ` 

This commit are the patches I've changed to successfully import my library
 2021-09-07 21:53:36 -03:00
Ross Williams
f6cf35a45d Fix Pinboard RSS parsing valid links as None 
`item.find(p)` returns either an `ElementTree.Element` or `None`.  The
[lambda on line 24][lambda] coerces the return value to a bool, which is
`False` if the `<link>` element has no children (see
[`ElementTree.py` line 207][etbooldef]), so the lambda returns `None`.

Further, returning a `Link` with `url=None` violates
[an assertion in `index/schema.py`][assertion], which crashes
the `archivebox add` command.

[lambda]: 3d54b1321b/archivebox/parsers/pinboard_rss.py (L24)
[etbooldef]: 3d8993a744/Lib/xml/etree/ElementTree.py (L207)
[assertion]: 3d54b1321b/archivebox/index/schema.py (L165)
 2021-08-04 10:13:37 -04:00
Inndy
0e81a0722e Discard Referer header from iframe and link to original URL 2021-07-19 21:48:01 +08:00
Nick Sweeting
5a2c78e14b add proper support for URL_WHITELIST instead of using negation regexes 2021-07-06 23:42:00 -04:00
Nick Sweeting
e4974d3536 support negation patterns by checking both re.search and re.match 2021-07-06 23:17:05 -04:00
TJ Horner
cdcfb7fa44 Exempt /add route from CSRF 2021-07-01 20:55:51 -04:00
Nick Sweeting
e0a2b2e252 ominous warnings 2021-06-01 03:03:42 -04:00
Nick Sweeting
aa53fe653c fix use of uneeded perms arg 2021-06-01 02:58:36 -04:00
Nick Sweeting
c2d1a57581 fix umask dir permissions 2021-06-01 00:50:18 -04:00
Nick Sweeting
4715ace7dd ignore BaseException lgtm errors 2021-05-31 20:59:05 -04:00
Nick Sweeting
9f1470cf03 fix output permissions tests 2021-05-31 20:57:46 -04:00
Nick Sweeting
8230f88d80 change default OUTPUT_PERMISSIONS to disallow execution except on dirs 2021-05-31 19:31:51 -04:00
Nick Sweeting
1112526543 add option ENFORCE_ATOMIC_WRITES to allow disabling forced FSYNC writes on network drives 2021-05-31 19:31:51 -04:00
Nick Sweeting
49faec8f6d
add no-zygote and single-process args to try and prevent orphan chrome processes after exit 2021-05-13 05:04:23 -04:00
Nick Sweeting
eb4d3bca9d
Update readability.py 2021-05-13 00:13:32 -04:00
Nick Sweeting
c3d009e4ec fix python file execution checking 2021-04-24 04:43:02 -04:00
Nick Sweeting
79e19ecd47 shield pwd import for windows 2021-04-24 03:51:38 -04:00
Nick Sweeting
3db77fd1a2 fix lint errors 2021-04-24 03:49:01 -04:00
Nick Sweeting
3992e0fee3 auto prepend python binary to args when running system command with python file as first argument 2021-04-24 03:29:22 -04:00
Nick Sweeting
fa84f749ff run archivebox version using python binary 2021-04-24 03:29:22 -04:00
Nick Sweeting
226e26852c dont try to autodetect whether node, chrome, etc are needed during setup 2021-04-24 03:29:22 -04:00
Nick Sweeting
f129b9f443 allow executing main 2021-04-24 03:11:06 -04:00
Nick Sweeting
208f866fc4 fix user detection on freebsd always returning root 2021-04-24 02:52:09 -04:00
Nick Sweeting
06f6084e3b ignore stdin when passed instead of throwing an error 2021-04-24 00:09:52 -04:00
Nick Sweeting
eb80dc26a1 cleanup config files 2021-04-23 22:58:44 -04:00
Nick Sweeting
4f91af0ce9 TODO create admin user from env vars 2021-04-23 15:50:34 -04:00
Nick Sweeting
9563e74f7c enforce minimum django version 2021-04-23 15:02:26 -04:00
Kunwoo Ahn
b6ee7d291e
fix bookmarklet script 
Fix '&' bug in parameter
 2021-04-16 21:24:51 +09:00
Nick Sweeting
63552be0e3 whoops 2021-04-13 04:03:47 -04:00
Nick Sweeting
7d1f8cb496 no longer recommend snap chromium, just use playwright 2021-04-13 03:59:17 -04:00
Nick Sweeting
2c6f0a96bf fix extra arg 2021-04-13 02:21:51 -04:00
Nick Sweeting
54cab113f3 allow static_index.json in dir root from old static exports 2021-04-12 17:06:32 -04:00
Nick Sweeting
50b341baab bail out if old index.json is found during init but doesnt contain links 2021-04-12 16:51:45 -04:00
Nick Sweeting
217952c8d0 sigh linters 2021-04-11 12:51:50 -04:00
Nick Sweeting
bfefe042bf fix noqa 2021-04-11 03:57:04 -04:00
Nick Sweeting
335e4658c8 keep grid header on one line 2021-04-11 00:49:33 -04:00
Nick Sweeting
52429b8d15 dont prevent search when backend throws an exception or times out 2021-04-10 08:18:13 -04:00
Nick Sweeting
9f05cf8283 virtual-time-budget doesnt work with some chrome stuff 2021-04-10 08:04:59 -04:00
Nick Sweeting
62078a77f8 show run duration after each archived link in cli output 2021-04-10 07:52:01 -04:00
Nick Sweeting
6a6ef63775 enforce utf8 on more file reads 2021-04-10 07:22:32 -04:00
Nick Sweeting
193df5c8d3 add video subtitles and description to full-text index 2021-04-10 07:22:20 -04:00
Nick Sweeting
1e3c1e3c83 also write autogenerated subtitles 2021-04-10 06:56:19 -04:00
Nick Sweeting
e7473c4e3e add some more padding to public index columns 2021-04-10 06:38:33 -04:00
Nick Sweeting
6949803395 enforce new models to use uuid keys 2021-04-10 06:32:45 -04:00
Nick Sweeting
1b67dd5c45 fix issues with django 3.2 2021-04-10 06:22:07 -04:00
Nick Sweeting
0c321a06d0 hide scrollbars in screenshots 2021-04-10 05:45:19 -04:00
Nick Sweeting
c5a9b77f4f update user agents to mimic latest chrome 2021-04-10 05:45:02 -04:00
Nick Sweeting
bf0da664be fix favicons on public index 2021-04-10 05:23:02 -04:00
Nick Sweeting
3093057675 bail out on sonic indexing after 5 errors 2021-04-10 05:18:03 -04:00
Nick Sweeting
89158d5dc7 allow full-text search from public index 2021-04-10 05:14:02 -04:00
Nick Sweeting
d1051e40bd remove tzlocal 2021-04-10 04:42:39 -04:00
Nick Sweeting
a9986f1f05 add timezone support, tons of CSS and layout improvements, more detailed snapshot admin form info, ability to sort by recently updated, better grid view styling, better table layouts, better dark mode support 2021-04-10 04:21:36 -04:00
Nick Sweeting
cf7d7e4990 add new timezone autosetting and cache header setting middlewares 2021-04-10 04:21:36 -04:00
Nick Sweeting
1977ae8962 add more helper props to snapshot for getting latest headers, favicon, etc 2021-04-10 04:21:36 -04:00
Nick Sweeting
8d68f1744e tweak columns in private snapshots list and shorten action names 2021-04-10 04:21:36 -04:00
Nick Sweeting
722f530c80 support adding and removing multiple tags at once using autocomplete multiselect field 2021-04-10 04:21:36 -04:00
Nick Sweeting
06c58ac42a fix unreachable config backup clear 2021-04-09 12:15:47 -04:00
Nick Sweeting
32764347ce add new SEARCH_BACKEND_TIMEOUT config option defaulted to 90sec 2021-04-09 00:27:12 -04:00
Nick Sweeting
2e57df917e handle BaseExceptions properly 2021-04-08 06:08:17 -04:00
Nick Sweeting
2c07d8f1f3 fix lgtm alerts 2021-04-06 13:47:26 -04:00
Nick Sweeting
21bb84f627 fix lint errors 2021-04-06 02:50:51 -04:00
Nick Sweeting
e92db03f5f fallback to using bin_path to find archivebox binary if sys.argv is empty 2021-04-06 02:45:57 -04:00
Nick Sweeting
c98053a8ca allow filtering by archiveresult status 2021-04-06 02:07:22 -04:00
Nick Sweeting
f67a5a215a fix readability indexing process and implement a max total character length on indexed content 2021-04-06 02:01:38 -04:00
Nick Sweeting
b3a89172ab add original src url to dosctring 2021-04-06 01:38:24 -04:00
Nick Sweeting
e7c7a8f71c force kill forked child processes after each extractor command 2021-04-06 01:35:21 -04:00
Nick Sweeting
af6561b527 remove CDN jquery and bootstrap by inlining them instead 2021-04-06 01:13:31 -04:00
Nick Sweeting
e4823cc5d7 add back CUSTOM_TEMPLATES_DIR config option 2021-04-06 00:57:20 -04:00
Nick Sweeting
4b098bea15 fix navbar showing user info even when not logged in 2021-04-06 00:44:01 -04:00
Nick Sweeting
68a506e1a7 fix Snapshot admin actions buttons submitting on Safari 2021-04-06 00:43:49 -04:00
Nick Sweeting
5a9f27204a dont use chrome when its not available on windows systems 2021-04-05 23:33:08 -04:00
Nick Sweeting
1485e4eb90 use older docker compose version for more features 2021-04-05 23:22:05 -04:00
Nick Sweeting
cadac4838e fix searching by url contents 2021-04-05 23:21:07 -04:00
Nick Sweeting
2fc9f1cfa3 dont depend on README in Dockerfile install step 2021-04-05 23:17:07 -04:00
Nick Sweeting
8b4f84959a new setup flag on init command to autosetup on init 2021-04-05 21:15:32 -04:00
Nick Sweeting
5c181532b5 only install dependencies in setup if they are missing 2021-04-05 21:15:10 -04:00
Nick Sweeting
21820acecd import User model into core models for convenience 2021-04-05 21:14:22 -04:00
Nick Sweeting
bbebc33b7a better autodetection of node modules dependencies 2021-04-05 21:14:06 -04:00
Nick Sweeting
211bf86f4a fix bug checking value in subcommand_args when None 2021-04-05 21:13:49 -04:00
Nick Sweeting
79fb60a47a dont allow caching when installing packages 2021-04-05 20:08:48 -04:00
Nick Sweeting
05c17af4dc fix lint errors 2021-04-05 20:00:16 -04:00
Nick Sweeting
ef76f7c490 require running setup in existing data dir 2021-04-05 19:59:33 -04:00
Nick Sweeting
8c1d12020f fix lint errors 2021-04-05 19:53:42 -04:00
Nick Sweeting
3449bac5bc add new setup command to autoinstall dependencies 2021-04-05 19:51:57 -04:00
Nick Sweeting
24f7dd9c05 dont raise systemexit on succesful config 2021-04-05 19:51:40 -04:00
Nick Sweeting
ffd119c66c fix linter 2021-04-03 15:51:27 -04:00
Nick Sweeting
7162649b03 fix search fields on admin snapshots list 2021-04-01 14:00:07 -04:00
Nick Sweeting
5492c204b8 tweak add page copy 2021-04-01 14:00:07 -04:00
Nick Sweeting
f59b6d4189 only add url-list lines that are real urls 2021-04-01 14:00:07 -04:00
Nick Sweeting
d73f7d7d96 add db_index on url field 2021-04-01 14:00:07 -04:00
Nick Sweeting
c438e49871 fix clicking on tags on snapshot list 2021-04-01 14:00:07 -04:00
Nick Sweeting
0e39a2098d split migrations and data dir check 2021-04-01 14:00:07 -04:00
Nick Sweeting
5d7ad9b1b4 add form parser option 2021-04-01 02:34:16 -04:00
Nick Sweeting
59d5423483 fix snapshot icon caching and ordering 2021-04-01 02:22:15 -04:00
Nick Sweeting
fc5d99b4b4 fix missing package.json repo 2021-04-01 01:40:02 -04:00
Nick Sweeting
ff9445648b fix navbar and tags display 2021-04-01 01:39:34 -04:00
Nick Sweeting
7ee7a94e67 fix str split using old overwrite pipe instead of append pipe 2021-03-31 11:30:15 -04:00
Nick Sweeting
005da31f8f add overwrite flag option to schedule cmd 2021-03-31 11:29:51 -04:00
Nick Sweeting
534ae60ab0 dont overwrite the log on each run, and use dedicated log for scheduled archiving 2021-03-31 11:20:42 -04:00
Nick Sweeting
5d3a03b299 use stderr and hint in case of parser returning no urls instead of bare exception 2021-03-31 01:39:01 -04:00
Nick Sweeting
8ce93ff787 use KEY, NAME, and PARSER to define parsers instead of hardcoding in init 2021-03-31 01:05:49 -04:00
Nick Sweeting
36f0646501
Merge pull request #669 from FliegendeWurst/fix-issue-235 
add command: --parser option (fixes #235)
 2021-03-31 00:53:47 -04:00
Nick Sweeting
2656e59215
change list style 2021-03-31 00:47:42 -04:00
Nick Sweeting
bd8a01e677 sigh vendor yet another dependency 2021-03-30 14:49:42 -04:00
FliegendeWurst
60bd9a902e add command: --parser option 2021-03-28 10:09:11 +02:00
Nick Sweeting
a58ad5b272 allow larger tags 2021-03-27 05:52:42 -04:00
Nick Sweeting
7e7c134ec0 use exc 2021-03-27 05:32:07 -04:00
Nick Sweeting
5568b3b612 fix adding and removing tags via admin snapshot list 2021-03-27 05:27:02 -04:00
Nick Sweeting
49b77b91bb fix unused var 2021-03-27 05:12:54 -04:00
Nick Sweeting
3e26ae4a66 support finding multiple urls as substrings in text 2021-03-27 04:30:40 -04:00
Nick Sweeting
f3a3d76439 support adding urls with tags directly via CLI and add page 2021-03-27 04:30:15 -04:00
Nick Sweeting
5fb9ca389f check more url parsing invariants on startup 2021-03-27 03:57:22 -04:00
Nick Sweeting
fea0b89dbe add tag cli option 2021-03-27 03:57:05 -04:00
Nick Sweeting
d486f8c162 use tempfile when logs dir is not available 2021-03-27 02:17:12 -04:00
Nick Sweeting
3c2f4c95cf fix unused import 2021-03-27 01:50:26 -04:00
Nick Sweeting
56c6b11b14 add comment 2021-03-27 01:19:32 -04:00
Nick Sweeting
bd6d9c165b enforce utf8 on literally all file operations because windows sucks 2021-03-27 01:16:29 -04:00
Nick Sweeting
185d2f9f9b check for JSON1 extesnion in Sqlite on startup 2021-03-27 00:27:58 -04:00
Nick Sweeting
177d22da50 fix clicking add tag when no tag is selected throwing 500 2021-03-01 01:39:06 -05:00
Nick Sweeting
76ff789164 fix tags page rendering when snapshot has no updated timestamp 2021-03-01 00:40:41 -05:00
Nick Sweeting
0036e9cce2 add migration 2021-02-28 22:55:12 -05:00
Nick Sweeting
9fc9f52cfb increase sqlite timeout and dont check for same thread condition 2021-02-28 22:55:00 -05:00
Nick Sweeting
1cabde3ccd remove atomic transactions 2021-02-28 22:54:40 -05:00
Nick Sweeting
844b5c5e20 minor db fix 2021-02-28 22:54:04 -05:00
Nick Sweeting
62089fdb22 close sqlite3 connections if unused 2021-02-28 22:53:34 -05:00
Nick Sweeting
6aef1dd98d dont check db if init is pending 2021-02-28 22:53:23 -05:00
Nick Sweeting
acbce25201 missing migrations 2021-02-18 08:05:05 -05:00
Nick Sweeting
9ead76f9f8 cross link snapshot index, admin, and actions pages using uuids 2021-02-18 08:04:50 -05:00
Nick Sweeting
c257ae240e only enable debug toolbar if nothreading is on 2021-02-18 08:04:26 -05:00
Nick Sweeting
ca2bb673ea improve inline admin forms and add tags autocomplete 2021-02-18 08:04:14 -05:00
Nick Sweeting
33df9c1ebe fix after and before in remove tests 2021-02-18 06:21:44 -05:00
Nick Sweeting
9c07fbdc0b unused import fix 2021-02-18 05:56:00 -05:00
Nick Sweeting
106f6adc59 fix lgtm alerts 2021-02-18 04:29:24 -05:00
Nick Sweeting
46a4197514 fix tests 2021-02-18 04:26:56 -05:00
Nick Sweeting
13c2b7c8eb fix url used by gridview 2021-02-18 02:38:30 -05:00
Nick Sweeting
e3b69c450c move db checks under check_db if statement 2021-02-18 02:37:33 -05:00
Nick Sweeting
74a9dd8880 add missing migrations 2021-02-18 02:36:21 -05:00
Nick Sweeting
7282f4a42a add favicon and robots to internal static dir 2021-02-18 02:35:51 -05:00
Nick Sweeting
dcc812b9f5 make output link easier to see on archiveresult admin list 2021-02-18 02:35:33 -05:00
Nick Sweeting
df47486d31 make admin link easier to find on snapshot detail page 2021-02-18 02:35:11 -05:00
Nick Sweeting
8369ca0b87 fix snapshot detail page titles garbled due to double escaping 2021-02-18 02:34:56 -05:00
Nick Sweeting
ba2afa0672 allow cancelling init partway through and show hints to resume 2021-02-18 02:34:42 -05:00
Nick Sweeting
75e1bfd0a9 create_or_update ArchiveResults from history instead of get_or_create 2021-02-18 02:34:20 -05:00
Nick Sweeting
69e0302c2a only add https to url if needed when prompting to save new snapshot 2021-02-18 02:34:03 -05:00
Nick Sweeting
9918bddb7e remove emptystring tags before saving 2021-02-18 02:33:42 -05:00
Nick Sweeting
f1823381d0 improve latest title logic to take longest title 2021-02-18 02:33:28 -05:00
Nick Sweeting
ad4bd49163 increase max title length to 512 2021-02-18 02:33:08 -05:00
Nick Sweeting
4e5671dda9 add ability to add and remove tags directly from snapshot list 2021-02-18 02:32:58 -05:00
Nick Sweeting
c877d67e0c check if pragma is already wal before setting 2021-02-18 02:31:42 -05:00
Nick Sweeting
7b7aa239fd autocreate db cache table when needed and reenable WAL mode 2021-02-17 18:26:04 -05:00
Nick Sweeting
d60d73754d fix favicon.ico and robots.txt served by runserver 2021-02-17 18:25:47 -05:00
Nick Sweeting
a8a6752b06 add CACHE_BACKEND options to settings for easier dev 2021-02-17 18:25:23 -05:00
Nick Sweeting
3457773988 fix ArchiveResult extractor showing up on top of dropdown in admin inline form 2021-02-17 18:25:01 -05:00
Nick Sweeting
084cf7ff51 add more explanation about snapshot.save timestamp bump 2021-02-17 13:34:46 -05:00
Nick Sweeting
265bcc0264 fix lint errors2 2021-02-16 16:29:41 -05:00
Nick Sweeting
6f0eec92eb fix lint errors 2021-02-16 16:26:48 -05:00
Nick Sweeting
d89034dcde disable debug_toolbar by default 2021-02-16 16:23:09 -05:00
Nick Sweeting
71cf8d5224 add migrations 2021-02-16 15:57:13 -05:00
Nick Sweeting
118bddb41a fix admin title and url truncation and add edit/admin link and ID to Snapshot detail page 2021-02-16 15:55:07 -05:00
Nick Sweeting
05e891632c add snapshot_id to Link and uuid to ArchiveResult 2021-02-16 15:54:27 -05:00
Nick Sweeting
0b8bdd8cc5 show dir size after each snapshot archiving process in cli 2021-02-16 15:53:52 -05:00
Nick Sweeting
acb932ba12 improve readability and mercury error handling and fix output path to be relative 2021-02-16 15:53:11 -05:00
Nick Sweeting
c95698e608 bump Snapshot.updated time after each extractor, change extractor order 2021-02-16 15:52:18 -05:00
Nick Sweeting
28e50c5e49 tweak snapshot asset serving logic to show multiple choices in case of conflict 2021-02-16 15:51:56 -05:00
Nick Sweeting
a6c64f2560 only get archive_size if index.html is present within dir 2021-02-16 15:51:17 -05:00
Nick Sweeting
c24e4bf11d tweak django debug toolbar panels and add djdt_flamegraph 2021-02-16 15:51:02 -05:00
Nick Sweeting
1e2188517b speed up latest_title by preferring db title if present 2021-02-16 15:49:53 -05:00
Nick Sweeting
8b236b9367 cache dir size, snapshot icons, tags str, and title in django cache 2021-02-16 15:49:29 -05:00
Nick Sweeting
51440ede3a show extra fields on ArchiveResult admin list and detail page 2021-02-16 15:47:49 -05:00
Nick Sweeting
74c882ef16 add nothreading arg to archivebox server for perf profiling 2021-02-16 15:46:58 -05:00
Nick Sweeting
8cde6bc676 add gist.github.com to git domains 2021-02-16 07:36:09 -05:00
Nick Sweeting
bdf1b102be load ArchiveResults from orphaned links history during init 2021-02-16 06:20:05 -05:00
Nick Sweeting
d0f8a5e710 change mercury atomic_write output order 2021-02-16 06:19:16 -05:00
Nick Sweeting
e62ecffeb0 disable django caches explicitly 2021-02-16 06:18:39 -05:00
Nick Sweeting
988a10a9f6 fix warc path in snapshot_icons 2021-02-16 06:18:05 -05:00
Nick Sweeting
48eddadc9b inline archive_size and archive_path snapshot methods 2021-02-16 06:16:02 -05:00
Nick Sweeting
46a7b972ee disable WAL and cache table for now 2021-02-16 06:13:50 -05:00
Nick Sweeting
24ae01cdbe add ArchiveResult admin 2021-02-16 06:13:27 -05:00
Nick Sweeting
87069dd41e fix wsgi.py 2021-02-16 04:16:51 -05:00
Nick Sweeting
8c2d81c041 always serve static files 2021-02-16 04:16:24 -05:00
Nick Sweeting
fb8e6cabcb fix error log location 2021-02-16 04:15:09 -05:00
Nick Sweeting
26fa63749d add debug toolbar 2021-02-16 02:50:05 -05:00
Nick Sweeting
fad2620c62 minor tweaks 2021-02-16 02:49:31 -05:00
Nick Sweeting
ae43d35e65 fix color option in docker 2021-02-16 01:26:26 -05:00
Nick Sweeting
4eac9d0ec1 show extra system setup info in version output 2021-02-16 01:23:58 -05:00
Nick Sweeting
82de67db34 fix missing/outdated template variables 2021-02-16 01:23:31 -05:00
Nick Sweeting
22da885148 log every archivebox command run to the errors.log 2021-02-16 01:23:01 -05:00
Nick Sweeting
19f7c907e0 exec archivebox from docker entrypoint script to avoid nesting pid under bash 2021-02-16 01:22:36 -05:00
Nick Sweeting
10d687c55c add new MEDIA_MAX_SIZE=750m size option for limiting youtubedl downloads 2021-02-16 01:21:19 -05:00
Nick Sweeting
49939f3eaa only accept stdin if args are not passed, fix stdin hang in docker 2021-02-16 01:20:47 -05:00
Nick Sweeting
de1a939df4 fix public index pagination links and total count 2021-02-15 21:07:12 -05:00
Nick Sweeting
108cb100b3 hotlink direclty to specific archivebox release from footer info 2021-02-15 21:06:56 -05:00
Nick Sweeting
56ec19fb29 allow hotlinking directly to a specific extractor output using hash on Snapshot detail page 2021-02-15 20:55:22 -05:00
Nick Sweeting
a089136fea fix missing stdout template var on /add when not rendering success page 2021-02-15 20:54:47 -05:00
Nick Sweeting
b6d7c74680 speed up the Snapshot handling view and show index page when extractor output is missing or multiple snapshots returned 2021-02-15 20:52:08 -05:00
Nick Sweeting
0375853683 log error tracebacks to logs/errors.log file and filter noisy 404s and 200s from log output 2021-02-15 20:51:23 -05:00
Nick Sweeting
0ec9bfb971 fix dead missing template variables 2021-02-15 20:50:12 -05:00
Nick Sweeting
b3a50a2c10 fix server quick-init param not being passed properly to subcommand 2021-02-15 20:49:40 -05:00
Nick Sweeting
b06e256ad9 fix add command not updating snapshot detail index pages when passed index-only and overwrite flags together 2021-02-15 20:49:23 -05:00
Nick Sweeting
8e98cef7ad fix after and before args flipped when filtering 2021-02-15 20:48:51 -05:00
Nick Sweeting
33d180afe7 allow filtering snapshots by timestamp in list, update, and remove cmds 2021-02-15 20:48:35 -05:00
Nick Sweeting
0c9db1c554 remove symbols from random secret key for easier copy pastin 2021-02-15 20:45:42 -05:00
Nick Sweeting
4faef03ba3 compute snapshot properties directly without loading whole Link 2021-02-15 20:44:08 -05:00
Nick Sweeting
9ce3bd5bdc use index.LINK_FILTERS to validate filter-type args instead of hardocding them twice 2021-02-15 20:43:36 -05:00
Nick Sweeting
c28ad8bd1b fix AddLinkForm widget complaining about missing template var class 2021-02-15 20:42:59 -05:00
Nick Sweeting
78463c243a remove unused GIT_SHA config option 2021-02-15 20:42:33 -05:00
Nick Sweeting
9cd4ba38f0 add new SNAPSHOTS_PER_PAGE pagination limit config 2021-02-15 20:42:00 -05:00
Nick Sweeting
00ae1f15a7 ignore shm db file and config files in archivebox data dir on init 2021-02-15 14:52:37 -05:00
Nick Sweeting
3c3bae02d2 add quick-init option to skip reimporting all snapshot dirs on init 2021-02-15 14:52:10 -05:00
Nick Sweeting
e61e12c889 use setup.py to determine dependencies in Dockerfile instead of egg-info requires.txt 2021-02-15 14:51:32 -05:00
Nick Sweeting
0407d03b6b add cli tests file back 2021-02-15 13:39:49 -05:00
Nick Sweeting
611216765d switch sqlite to use WAL mode by default to prevent database locked errors 2021-02-15 13:39:03 -05:00
Nick Sweeting
683a08772b change wording of db not found error 2021-02-08 23:27:46 -05:00
Nick Sweeting
6705354e57 fix assertion 2021-02-08 23:24:48 -05:00
Nick Sweeting
a49884ade8 fix emptystrings in cmd_version causing exception 2021-02-08 23:22:02 -05:00
Nick Sweeting
171bbeb69b catch exception on import of old index.json into ArchiveResult 2021-02-01 16:31:29 -05:00
Nick Sweeting
0aea5ed3e8 fix handling of skipped ArchiveResult entries with null output 2021-02-01 14:37:34 -05:00
Nick Sweeting
c4b02be24d remove dead tests code 2021-02-01 05:14:43 -05:00
Nick Sweeting
783f597955 minor build fixes 2021-02-01 05:13:46 -05:00
Nick Sweeting
aa84a7ff2b fix migration creating conflicting tags based on slug 2021-02-01 05:13:23 -05:00
Nick Sweeting
7d0f5653c3 fix lgtm alerts 2021-02-01 02:27:24 -05:00
Nick Sweeting
04c951cdd5 fix alerts 2021-02-01 02:22:02 -05:00
Nick Sweeting
534ead2440 use the db exclusively for icons instead of hammering filesystem 2021-02-01 02:18:13 -05:00
Nick Sweeting
923f517a8f minor fixes 2021-02-01 02:17:54 -05:00
Nick Sweeting
560d3103a8 cleanup snapshot detail page UI 2021-01-30 22:04:24 -05:00
Nick Sweeting
54c5331693 check for output existance when rendering files icons 2021-01-30 22:04:14 -05:00
Nick Sweeting
15e87353bd only show archive.org if enabled 2021-01-30 22:03:59 -05:00
Nick Sweeting
846c966c4d use globbing to find wget output path 2021-01-30 22:02:39 -05:00
Nick Sweeting
e6fa16e13a only chmod wget output if it exists 2021-01-30 22:02:11 -05:00
Nick Sweeting
385daf9af8 save the url as title for staticfiles or non html files 2021-01-30 22:01:49 -05:00
Nick Sweeting
24e24934f7 add headers.json and fix relative singlefile path resolving for sonic 2021-01-30 21:59:34 -05:00
Nick Sweeting
c089501073 add response status code to headers.json 2021-01-30 20:44:49 -05:00
Nick Sweeting
b9b1c3d9e8 fix singlefile output path not relative 2021-01-30 20:44:49 -05:00
Nick Sweeting
d072f1d413 hide ssl warnings when checking SSL is disabled 2021-01-30 20:44:49 -05:00
Nick Sweeting
9d24bfd0dc disable progress bars on mac again 2021-01-30 20:44:49 -05:00
Nick Sweeting
326ce78496 simplify debug 2021-01-30 06:09:26 -05:00
Nick Sweeting
d6de04a83a fix lgtm errors 2021-01-30 06:07:35 -05:00
Nick Sweeting
cc80ceb0a2 fix icons in public index 2021-01-30 05:47:55 -05:00
Nick Sweeting
1ce0eca217 add trailing slashes to canonical paths 2021-01-30 05:47:55 -05:00
Nick Sweeting
6edae6a17f add future api spec design 2021-01-30 05:47:55 -05:00
Nick Sweeting
a98298103d cleanup templates and views 2021-01-30 05:47:55 -05:00
Nick Sweeting
ed13ec7655 remove active theme 2021-01-30 05:47:55 -05:00
Nick Sweeting
c2aaa41c76 fix missing str path 2021-01-30 01:25:08 -05:00
Nick Sweeting
ff7d2ffa09 fix version in legacy footer 2021-01-29 09:18:38 -05:00
Nick Sweeting
6e84890abd improve loading snapshots tooltips 2021-01-29 09:09:23 -05:00
Nick Sweeting
8a4edb45e7 also search url, timestamp, tags on public index 2021-01-29 09:08:03 -05:00
Nick Sweeting
f6c3683ab8 fix snapshot favicon loading spinner height 2021-01-29 00:15:32 -05:00
Nick Sweeting
3227f54b52 limit youtubedl download size to 750m and stop splitting out audio files 2021-01-29 00:15:32 -05:00
Nick Sweeting
d7df9e58ea hide footer on add page 2021-01-28 23:15:05 -05:00
Nick Sweeting
5c54bcc1f3 fix files icons greying out on public index 2021-01-28 22:57:12 -05:00
Nick Sweeting
7d8fe66d43 consistent tags styling 2021-01-28 22:35:21 -05:00
Nick Sweeting
6a8f6992d8 reuse admin styling for pubic index and add page 2021-01-28 22:28:10 -05:00
Nick Sweeting
f0040580c8 fix files icons escaping 2021-01-28 22:27:17 -05:00
Nick Sweeting
39ec77e46c add createsuperuser flag to server command 2021-01-28 22:27:02 -05:00
Nick Sweeting
4b7550c23f
Merge pull request #632 from aggroskater/bugfix/issue-617 2021-01-28 17:03:57 +02:00
Nick Sweeting
15e58bd366 fix using os.path calls on pathlib paths 2021-01-27 11:27:40 -05:00
Preston Maness
1810426774 Remove now-unused mark_safe import 2021-01-25 21:16:06 -06:00
Preston Maness
b647581115
Update archivebox/index/html.py 
mark_safe is dangerous, as the URL's filename could have malicious HTML fragments in it.

Co-authored-by: Nick Sweeting <git@sweeting.me>
 2021-01-25 20:47:57 -06:00
Nick Sweeting
9764a8ed9b check for non html files from wget 2021-01-25 18:15:16 -05:00
Preston Maness
1989275944 Fix issue #617 by using mark_safe in combination with format_html 
I have no experience with Django, so all I'm really going off of is this
stackoverflow

https://stackoverflow.com/a/64498319

which cited this bit of Django documentation:

https://docs.djangoproject.com/en/3.1/ref/utils/#django.utils.html.format_html

After using this method, I no longer get the 500 error or KeyError
exception, and can browse the local server and interact with the single
entry in it (the problematic URL in ArchiveBox#617 with curly braces).

Whether this is the "right" method or not, I have no idea. But it is at
least a start.
 2021-01-23 20:32:56 -06:00
Dan Arnfield
5420903102 Refactor should_save_extractor methods to accept overwrite parameter 2021-01-21 15:56:32 -06:00
Nick Sweeting
ef7711ffa0 fix cookies file arg is path 2021-01-20 19:13:53 -05:00
Nick Sweeting
a07ed3989e fix import path 2021-01-20 19:02:31 -05:00
Nick Sweeting
72e2c7b95d use relative imports for util 2021-01-20 18:44:28 -05:00
Nick Sweeting
02bdb3bdeb fix DATABASE_NAME posixpath 2021-01-20 18:42:12 -05:00
jdcaballerov
14df0cbb7c
Update sonic.py 
Sonic buffer accepts 20.000 bytes not unicode characters, since the chunking here is on unicode characters, sending 20.000 characters will overflow sonic's buffer.
UTF-8 can take up to 6 bytes, so sending less than (20.000 / 6) rounded minus should be ok.
 2021-01-20 14:51:46 -05:00
Dan Arnfield
5c7842ffb3 Fix dependency dict entries 2021-01-20 09:24:34 -06:00
Nick Sweeting
a3008c8189 fix migration failing due to null cmd_versions in older archives 2021-01-12 12:56:06 +02:00
Nick Sweeting
f2a0068c17
Merge pull request #608 from cdvv7788/extractor-bugs 2021-01-07 16:38:56 +02:00
Cristian
6031ffa3b2 fix: Mercury extractor error was incorrectly initialized 2021-01-07 09:22:46 -05:00
Cristian
e9e4adfc34 fix: wget_output_path failing on some extractors. Add a new condition 2021-01-07 09:07:29 -05:00
Cristian
14d1b3209e fix: Make cmd_version nullable 2021-01-06 20:03:40 +02:00
Cristian
c21af37ed4 fix: Give cmd_version a default value in case it is not present 2021-01-06 20:03:40 +02:00
Tim Gates
7bf63d91ff docs: fix simple typo, timstamp -> timestamp 
There is a small typo in archivebox/index/__init__.py.

Should read `timestamp` rather than `timstamp`.
 2021-01-06 20:03:40 +02:00
Nick Sweeting
9784dcb816 better config comments and docstrings 2020-12-20 03:11:19 +02:00
Nick Sweeting
72b8119881
Merge pull request #587 from jdcaballerov/move-vendored-as-submodules 
Add submodules and links
 2020-12-16 09:59:56 -05:00
jdcaballerov
a2694a3e8a Add submodules and links 2020-12-16 08:53:59 -05:00
jdcaballerov
c29ce7e7f0 Add border for card select 2020-12-14 16:00:59 -05:00
jdcaballerov
7b66e1514d Merge branch 'v0.5.0' of github.com:ArchiveBox/ArchiveBox into feat-snapshots-grid 2020-12-14 15:05:19 -05:00
jdcaballerov
243fcccd89 Allow actions on grid view 2020-12-14 15:01:24 -05:00
jdcaballerov
6b5c881555 Fix search to include filters 2020-12-14 13:40:38 -05:00
jdcaballerov
45e97ea278 Reverse test condition to avoid redirects with change details 2020-12-14 13:27:06 -05:00
jdcaballerov
d4255be077 use localStorage var 2020-12-14 13:00:13 -05:00
jdcaballerov
8fca36a7cd Restore preferred snapshots view from localstorage 2020-12-14 12:52:15 -05:00
jdcaballerov
7db6b0a8a6 Preserve query string between snapshot list views 2020-12-14 12:11:44 -05:00
Nick Sweeting
326fe69eea
fix lint error 2020-12-12 12:35:32 -05:00
jdcaballerov
9b6afa36a3
Update archivebox/search/backends/ripgrep.py 
Co-authored-by: Nick Sweeting <git@sweeting.me>
 2020-12-12 08:36:08 -05:00
jdcaballerov
aa53f4f088
Update archivebox/search/backends/ripgrep.py 
Co-authored-by: Nick Sweeting <git@sweeting.me>
 2020-12-12 08:36:01 -05:00
jdcaballerov
50df108863
Update archivebox/config.py 
Co-authored-by: Nick Sweeting <git@sweeting.me>
 2020-12-12 08:34:00 -05:00
jdcaballerov
24d4c44624 Add ripgrep configs 2020-12-12 07:36:31 -05:00
jdcaballerov
254d2502fd Feature implementation 2020-12-11 23:03:46 -05:00
Cristian
a57a5b6b83 refactor: call setup_django with the check_db attribute for the commands that actually need the database 2020-12-11 18:02:56 -05:00
Cristian
57d1a3d4e5 refactor: Remove setup_django from html.py 2020-12-11 17:49:16 -05:00
Cristian
ce53b0220c refactor: Remove setup_django from index 2020-12-11 17:36:31 -05:00