Mirrors/ArchiveBox
2
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox synced 2024-11-22 20:23:12 +00:00
Code Issues Projects Releases Packages Wiki Activity
4100 commits 6 branches 78 tags 19 MiB
Commit graph

405 commits

Author SHA1 Message Date
Nick Sweeting
9c07fbdc0b unused import fix 2021-02-18 05:56:00 -05:00
Nick Sweeting
106f6adc59 fix lgtm alerts 2021-02-18 04:29:24 -05:00
Nick Sweeting
13c2b7c8eb fix url used by gridview 2021-02-18 02:38:30 -05:00
Nick Sweeting
74a9dd8880 add missing migrations 2021-02-18 02:36:21 -05:00
Nick Sweeting
69e0302c2a only add https to url if needed when prompting to save new snapshot 2021-02-18 02:34:03 -05:00
Nick Sweeting
9918bddb7e remove emptystring tags before saving 2021-02-18 02:33:42 -05:00
Nick Sweeting
f1823381d0 improve latest title logic to take longest title 2021-02-18 02:33:28 -05:00
Nick Sweeting
ad4bd49163 increase max title length to 512 2021-02-18 02:33:08 -05:00
Nick Sweeting
4e5671dda9 add ability to add and remove tags directly from snapshot list 2021-02-18 02:32:58 -05:00
Nick Sweeting
d60d73754d fix favicon.ico and robots.txt served by runserver 2021-02-17 18:25:47 -05:00
Nick Sweeting
a8a6752b06 add CACHE_BACKEND options to settings for easier dev 2021-02-17 18:25:23 -05:00
Nick Sweeting
3457773988 fix ArchiveResult extractor showing up on top of dropdown in admin inline form 2021-02-17 18:25:01 -05:00
Nick Sweeting
265bcc0264 fix lint errors2 2021-02-16 16:29:41 -05:00
Nick Sweeting
6f0eec92eb fix lint errors 2021-02-16 16:26:48 -05:00
Nick Sweeting
d89034dcde disable debug_toolbar by default 2021-02-16 16:23:09 -05:00
Nick Sweeting
71cf8d5224 add migrations 2021-02-16 15:57:13 -05:00
Nick Sweeting
05e891632c add snapshot_id to Link and uuid to ArchiveResult 2021-02-16 15:54:27 -05:00
Nick Sweeting
28e50c5e49 tweak snapshot asset serving logic to show multiple choices in case of conflict 2021-02-16 15:51:56 -05:00
Nick Sweeting
a6c64f2560 only get archive_size if index.html is present within dir 2021-02-16 15:51:17 -05:00
Nick Sweeting
c24e4bf11d tweak django debug toolbar panels and add djdt_flamegraph 2021-02-16 15:51:02 -05:00
Nick Sweeting
1e2188517b speed up latest_title by preferring db title if present 2021-02-16 15:49:53 -05:00
Nick Sweeting
8b236b9367 cache dir size, snapshot icons, tags str, and title in django cache 2021-02-16 15:49:29 -05:00
Nick Sweeting
51440ede3a show extra fields on ArchiveResult admin list and detail page 2021-02-16 15:47:49 -05:00
Nick Sweeting
e62ecffeb0 disable django caches explicitly 2021-02-16 06:18:39 -05:00
Nick Sweeting
48eddadc9b inline archive_size and archive_path snapshot methods 2021-02-16 06:16:02 -05:00
Nick Sweeting
24ae01cdbe add ArchiveResult admin 2021-02-16 06:13:27 -05:00
Nick Sweeting
87069dd41e fix wsgi.py 2021-02-16 04:16:51 -05:00
Nick Sweeting
8c2d81c041 always serve static files 2021-02-16 04:16:24 -05:00
Nick Sweeting
fb8e6cabcb fix error log location 2021-02-16 04:15:09 -05:00
Nick Sweeting
26fa63749d add debug toolbar 2021-02-16 02:50:05 -05:00
Nick Sweeting
ae43d35e65 fix color option in docker 2021-02-16 01:26:26 -05:00
Nick Sweeting
22da885148 log every archivebox command run to the errors.log 2021-02-16 01:23:01 -05:00
Nick Sweeting
a089136fea fix missing stdout template var on /add when not rendering success page 2021-02-15 20:54:47 -05:00
Nick Sweeting
b6d7c74680 speed up the Snapshot handling view and show index page when extractor output is missing or multiple snapshots returned 2021-02-15 20:52:08 -05:00
Nick Sweeting
0375853683 log error tracebacks to logs/errors.log file and filter noisy 404s and 200s from log output 2021-02-15 20:51:23 -05:00
Nick Sweeting
0c9db1c554 remove symbols from random secret key for easier copy pastin 2021-02-15 20:45:42 -05:00
Nick Sweeting
4faef03ba3 compute snapshot properties directly without loading whole Link 2021-02-15 20:44:08 -05:00
Nick Sweeting
c28ad8bd1b fix AddLinkForm widget complaining about missing template var class 2021-02-15 20:42:59 -05:00
Nick Sweeting
9cd4ba38f0 add new SNAPSHOTS_PER_PAGE pagination limit config 2021-02-15 20:42:00 -05:00
Nick Sweeting
e61e12c889 use setup.py to determine dependencies in Dockerfile instead of egg-info requires.txt 2021-02-15 14:51:32 -05:00
Nick Sweeting
611216765d switch sqlite to use WAL mode by default to prevent database locked errors 2021-02-15 13:39:03 -05:00
Nick Sweeting
171bbeb69b catch exception on import of old index.json into ArchiveResult 2021-02-01 16:31:29 -05:00
Nick Sweeting
0aea5ed3e8 fix handling of skipped ArchiveResult entries with null output 2021-02-01 14:37:34 -05:00
Nick Sweeting
aa84a7ff2b fix migration creating conflicting tags based on slug 2021-02-01 05:13:23 -05:00
Nick Sweeting
923f517a8f minor fixes 2021-02-01 02:17:54 -05:00
Nick Sweeting
b9b1c3d9e8 fix singlefile output path not relative 2021-01-30 20:44:49 -05:00
Nick Sweeting
326ce78496 simplify debug 2021-01-30 06:09:26 -05:00
Nick Sweeting
d6de04a83a fix lgtm errors 2021-01-30 06:07:35 -05:00
Nick Sweeting
6edae6a17f add future api spec design 2021-01-30 05:47:55 -05:00
Nick Sweeting
a98298103d cleanup templates and views 2021-01-30 05:47:55 -05:00
Nick Sweeting
ed13ec7655 remove active theme 2021-01-30 05:47:55 -05:00
Nick Sweeting
8a4edb45e7 also search url, timestamp, tags on public index 2021-01-29 09:08:03 -05:00
Nick Sweeting
d7df9e58ea hide footer on add page 2021-01-28 23:15:05 -05:00
Nick Sweeting
a07ed3989e fix import path 2021-01-20 19:02:31 -05:00
Nick Sweeting
72e2c7b95d use relative imports for util 2021-01-20 18:44:28 -05:00
Nick Sweeting
02bdb3bdeb fix DATABASE_NAME posixpath 2021-01-20 18:42:12 -05:00
Nick Sweeting
a3008c8189 fix migration failing due to null cmd_versions in older archives 2021-01-12 12:56:06 +02:00
Cristian
14d1b3209e fix: Make cmd_version nullable 2021-01-06 20:03:40 +02:00
Cristian
c21af37ed4 fix: Give cmd_version a default value in case it is not present 2021-01-06 20:03:40 +02:00
jdcaballerov
7b66e1514d Merge branch 'v0.5.0' of github.com:ArchiveBox/ArchiveBox into feat-snapshots-grid 2020-12-14 15:05:19 -05:00
jdcaballerov
6b5c881555 Fix search to include filters 2020-12-14 13:40:38 -05:00
jdcaballerov
254d2502fd Feature implementation 2020-12-11 23:03:46 -05:00
Nick Sweeting
1c87c27105 patch migration JSONField as well 2020-12-11 20:50:45 +02:00
Nick Sweeting
081d94d799 fallback to old JSONField from lib if django version is old 2020-12-11 20:45:44 +02:00
Nick Sweeting
c084e70ea8 fix TEMPLATES_DIR location 2020-12-11 16:21:09 +02:00
Nick Sweeting
a194bb6301
Merge pull request #580 from BlipRanger/master 2020-12-10 12:48:30 -05:00
BlipRanger
6f462b45d7
Update archivebox/core/forms.py 
Cleaner handling of the ARCHIVE_METHODS values

Co-authored-by: Nick Sweeting <git@sweeting.me>
 2020-12-10 12:46:16 -05:00
BlipRanger
35809eab1c
Update archivebox/core/views.py 
Cleaner handling of the archive methods input

Co-authored-by: Nick Sweeting <git@sweeting.me>
 2020-12-10 12:45:30 -05:00
BlipRanger
7ce1f63183
Update archivebox/core/forms.py 
Format cleanup

Co-authored-by: Nick Sweeting <git@sweeting.me>
 2020-12-10 12:44:38 -05:00
BlipRanger
8b0ff2dfee update instead of append 2020-12-10 11:08:27 -05:00
BlipRanger
d9fd1e3811 Add selector for archive modes 2020-12-10 10:51:57 -05:00
Cristian
9aac09a5e1 feat: Patch setup_django so we can use an inmemory db in specific commands 2020-12-08 18:42:25 -05:00
Cristian
8d22ebf988 feat: Remove walrus operator (we still need to support python3.7) 2020-12-06 12:23:02 -05:00
Nick Sweeting
a0a79cead8 move utils and vendored libs into subfolders 2020-12-06 02:01:18 +02:00
JDC
7903db6dfb Add ArchiveResult Manager and sorted indexable filter 2020-12-06 01:13:39 +02:00
JDC
823df34080 Use QuerySets for search backend API instead of pks 2020-12-06 01:12:47 +02:00
JDC
47daa038eb Implement flush for search backend after remove command 2020-12-06 01:12:47 +02:00
JDC
5f6673c72c Implement backend architecture for search engines 2020-12-06 01:12:46 +02:00
JDC
b1f70b2197 Initial implementation 2020-12-06 01:12:45 +02:00
jdcaballerov
c8d8346e4d Remove duplicate context after rebase 2020-12-03 15:44:21 -05:00
jdcaballerov
8ac7a760c9 Fix num_links missing in public index 2020-12-03 08:32:49 -05:00
jdcaballerov
69897f6121 Hotfix public page search 
No ordering causes warning and fallback to default unfiltered QuerySet
 2020-12-03 08:32:49 -05:00
Nick Sweeting
193dde03f0
Merge pull request #559 from jdcaballerov/hotfix-public-search 
Hotfix public page search
 2020-12-01 10:56:32 -05:00
jdcaballerov
4d972571d0 Hotfix public page search 
No ordering causes warning and fallback to default unfiltered QuerySet
 2020-12-01 10:46:11 -05:00
Nick Sweeting
104553489f remove redundant utils file 2020-11-28 02:12:27 -05:00
Nick Sweeting
7fdea91311 fix static html num_outputs info 2020-11-28 02:01:53 -05:00
Nick Sweeting
fde65c3b7d fix public index missing template context 2020-11-28 01:29:34 -05:00
Nick Sweeting
46a53eafdb simplify history helper 2020-11-28 01:14:45 -05:00
Nick Sweeting
9661c863b3 css style tweaks for icons 2020-11-28 01:06:23 -05:00
Nick Sweeting
910f3d65c7 default function args can never be mutable 2020-11-28 01:06:11 -05:00
Nick Sweeting
411fdcac87 use database for num_outputs instead of legacy json 2020-11-28 01:05:53 -05:00
Nick Sweeting
00bb55203e
always show WARC icon with opacity set based on exists 2020-11-27 23:45:49 -05:00
Nick Sweeting
efe3027797
Merge branch 'master' into archive-result 2020-11-27 23:18:11 -05:00
Cristian
4b3f72202b feat: Bump django, update migration and change cmd to use JSONField 2020-11-27 16:23:27 -05:00
Cristian
f61e6a74bb feat: Re-add unused icons in list view 2020-11-27 15:55:37 -05:00
Nick Sweeting
f84f288bef
Apply suggestions from code review 
minor nit
 2020-11-27 00:01:34 -05:00
Cristian
34a1a6d30d fix: Update model according to code review 2020-11-23 18:28:43 -05:00
Nick Sweeting
0e2ccbc10d update urls to new repo path 2020-11-23 02:06:46 -05:00
Nick Sweeting
afe9319c25
Merge pull request #537 from TrAyZeN/master 2020-11-18 23:20:41 -05:00
JDC
8b0250caeb Fixes 500 error on search 
The class SnapshotAdmin search_fields includes the
tags ManyToMany field causing a
django.core.exceptions.FieldError: Related Field got invalid lookup: icontains
error.
A related search field tags__name should be used.
 2020-11-17 08:36:03 -05:00
TrAyZeN
a05485f85c Fix file icons order 2020-11-14 17:44:06 +01:00
Cristian
0f13087a09 refactor: Remove unneeded prefetch related 2020-11-12 13:58:13 -05:00
Cristian
c565fad75c feat: Use prefetch related to reduce the number of queries to the database on public index view 2020-11-12 11:37:56 -05:00
Cristian
8cfad64271 feat: Add specific logic for archive_org icon 2020-11-12 11:09:34 -05:00
Cristian
e594e6a75a feat: WARC link points to the first warc result in target path 2020-11-12 10:57:31 -05:00
Cristian
b237e412df feat: Finish reversal. Add ArchiveResults that are not found in the index.json 2020-11-12 10:30:41 -05:00
Cristian
f7f0bebdcc feat: Modify migration reverse function to restore index (WIP) 2020-11-11 15:26:54 -05:00
Cristian
508a0bb06e refactor: Unpack extractors tuple instead of using the index to access the relevant information 2020-11-10 12:38:29 -05:00
Cristian
71655220ad feat: Add warc to list and limit check to succeeded archive results 2020-11-05 07:54:40 -05:00
Cristian
4484491fb7 feat: Create ArchiveResult after finishing an extractor process 2020-11-04 11:22:55 -05:00
Cristian
b3e0400bc0 feat: initial functional version with icons calculated based on archive results 2020-11-04 10:31:20 -05:00
Cristian
309a87e8fe feat: Add extractor field to the database 2020-11-04 07:28:02 -05:00
Cristian
8f3c03a0f9 feat: Initial (and naive) ArchiveResult model 2020-11-03 09:54:02 -05:00
Cristian
ac0ec160d1 lint: Fix warnings in master branch 2020-11-02 08:51:48 -05:00
Nick Sweeting
22fb9c2ad7 tweak icons 2020-10-31 19:32:43 -04:00
Nick Sweeting
651d6c4447 bold snapshots over 50MB 2020-10-31 07:57:11 -04:00
Nick Sweeting
ac9e0e356d config fixes 2020-10-31 07:57:11 -04:00
Nick Sweeting
18355dc2c6 clean up config loading in settings and config file layout 2020-10-31 03:08:03 -04:00
Nick Sweeting
aede134ab3 temporarily disable icon highlighting in favor of performance 2020-10-30 05:12:33 -04:00
Nick Sweeting
e727af6f22 allow Path args to get_dir_size and copy_and_overwrite 2020-10-24 22:47:18 -04:00
Cristian Vargas
a850b4a9d9
Merge branch 'master' into tags 2020-10-20 08:23:25 -05:00
Cristian
62c78e1d10 refactor: Remove django-taggit and replace it with a local tags setup 2020-10-12 13:47:03 -05:00
Nick Sweeting
6c704fa8cf
Merge pull request #498 from adamwolf/bookmarklet 
Add a bookmarklet
 2020-10-09 21:59:07 -04:00
Cristian
10384a8a6f style: Improve look of tags in admin list 2020-10-07 10:15:56 -05:00
Cristian
b9e5b781a7 fix: Avoid creating empty tag on migration 2020-10-07 09:59:49 -05:00
Cristian
62f3d648d4 fix: reverse_func functional 2020-10-07 09:46:10 -05:00
Adam Wolf
8d3295458c Add a bookmarklet 
The bookmarklet lets you quickly open the Add page
with the URL already populated in the URLs box.
 2020-10-03 14:57:55 -05:00
Angel Rey
897bace84d Fixed paths in settings 2020-10-02 15:46:39 -05:00
Angel Rey
0e7c337dcb Replaced os.path in settings.py 2020-10-02 15:46:39 -05:00
Cristian
5975c27a6a fix: Remove trailing slash from public index 2020-09-25 13:48:19 -05:00
Angel Rey
4581ea956f Fixed empty tags 2020-09-24 15:34:23 -05:00
Angel Rey
f26c0c6cd8 Fix serialization 2020-09-24 15:34:23 -05:00
Angel Rey
62c9028212 Improved tags 2020-09-24 15:34:23 -05:00
Cristian
b2ed96c35a feat: Redirect old add view to the main one 2020-09-17 09:08:20 -05:00
Cristian
b3ec170e39 fix: Remove unused imports 2020-09-16 08:50:56 -05:00
Cristian
bc116c25f8 refactor: Change View to FormView 2020-09-16 08:50:56 -05:00
apkallum
1cdaad00a8 no more oldhome, cbvs uniform across views 2020-09-16 08:50:56 -05:00
apkallum
5e8c115f3f unify public archive view 2020-09-16 08:50:56 -05:00
apkallum
3288f8579b add public add view + toggle setting 2020-09-16 08:50:56 -05:00
apkallum
6f7cc2b3ef ensure results have icons 2020-09-16 08:50:56 -05:00
apkallum
3048c0f6dc add icons to new public view 2020-09-16 08:50:56 -05:00
apkallum
c50af04cce search view inherits from modified public view 2020-09-16 08:50:56 -05:00
apkallum
5c4ac3cf3d new public view derived from django 2020-09-16 08:50:56 -05:00
Cristian
779a446085 feat: Make title and tags editable in admin 2020-09-15 08:05:46 -05:00
Cristian
2aa8d69b72 fix: Save history in main index (to mimic previous behaviour) 2020-09-15 08:05:46 -05:00
Cristian
a8ed72501d feat: Refactor remove command to use querysets 2020-09-15 08:05:46 -05:00
Nick Sweeting
273588c75a change main link to point to link details index to reduce confusion 2020-08-18 09:17:21 -04:00
Nick Sweeting
6087e30d38 dont allow ui editing of db fields for now 2020-08-18 09:17:01 -04:00
Nick Sweeting
c9b3bab84d fix pull title not working 2020-08-18 08:49:26 -04:00
Cristian
a40e337280 feat: Add link to admin list of files 2020-08-03 13:22:06 -05:00
Nick Sweeting
a160e6bf20 fix None canon output to be emptystring 2020-07-29 23:54:50 -04:00
Nick Sweeting
9dedcdd577 remove inaccurate updated ts from main index UI 2020-07-29 23:54:50 -04:00
Nick Sweeting
9806ed8d8c fix circular import 2020-07-28 06:50:03 -04:00
Nick Sweeting
5a30e03778 rearrange tags column and improve files icons 2020-07-28 05:59:54 -04:00
Nick Sweeting
943453a9a8 pass overwrite properly 2020-07-28 05:54:42 -04:00
Nick Sweeting
313fcd0501 change defalt date format to ISO 2020-07-28 05:51:18 -04:00
Nick Sweeting
ece6d43078 hide builtin delete button 2020-07-28 05:51:02 -04:00
Nick Sweeting
d70bb7980e use proper url naming instead of hardcoding paths 2020-07-27 23:56:35 -04:00
Nick Sweeting
3aeca0e450 fix pending titles and favicons, improve add page, custom admin 2020-07-27 23:26:45 -04:00
Nick Sweeting
022231b362 fix favicon url and show size in separate column 2020-07-27 19:30:40 -04:00
Cristian
6006b4f93b refactor: Organize code to remove flake8 issues 2020-07-24 12:25:25 -05:00
Cristian
a5550b2105 fix: Rename logging folder to avoid naming conflicts (and circular import issues) 2020-07-22 11:02:13 -05:00
Cristian
f4d1b5121e refactor: Move logging.py to main module to avoid circular import issues 2020-07-17 18:00:04 -05:00
Cristian
83e5b019e4 feat: Add canonical link http header to the static response 2020-07-16 12:49:26 -05:00
Nick Sweeting
a79dd4685a make snapshots unique again 2020-07-13 12:21:52 -04:00
Nick Sweeting
ae208435c9 fix the add links form 2020-07-13 12:21:37 -04:00
Nick Sweeting
7cbd068c95 add flake8 2020-07-13 11:22:23 -04:00
Nick Sweeting
5b571aa166
Merge pull request #356 from cdvv7788/depth-flag 2020-07-13 05:05:36 -04:00
Cristian
d476b13007 fix: Add missing permission to add view (post) 2020-07-08 14:46:31 -05:00
Cristian
f12bfeb322 refactor: Change add() to receive url and depth instead of import_str and import_path 2020-07-08 08:17:47 -05:00
Cristian
8bdfa18a3f feat: Allow feed loading from the add links view 2020-07-02 15:54:25 -05:00
Nick Sweeting
8840ad72bb remove circular import possibilities 2020-07-02 03:13:35 -04:00
Cristian
364c5752d8 feat: Handle empty URL case 2020-07-01 13:14:20 -05:00
Cristian
c971e00c9c feat: Add stdout from process to the template 2020-07-01 12:23:59 -05:00
Nick Sweeting
264cae2b6a add migration 2020-06-30 06:42:17 -04:00
Nick Sweeting
c415420f33 improve sort columns and UI placeholders 2020-06-30 06:41:48 -04:00
Nick Sweeting
178f6ac1a6 switch to django admin snapshots list as new homepage 2020-06-30 05:56:17 -04:00
Nick Sweeting
df593dea0a fix missing imports 2020-06-30 05:55:34 -04:00
Nick Sweeting
094beac1d9 fix add page redirect 2020-06-29 21:26:59 -04:00
Nick Sweeting
b9e17fa0d1 rename archivebox-info to archivebox-status 2020-06-25 23:32:01 -04:00
Nick Sweeting
43c471e4af cli experience improvements 2020-06-25 17:47:55 -04:00
Nick Sweeting
afacc5c5da use fallback random secret key 2020-04-22 21:14:43 -04:00
Nick Sweeting
7d699578e8 implement add page 2020-03-27 08:38:22 -04:00
Nick Sweeting
3c3b2ee621 expose more django server config options 2019-05-02 19:15:16 -04:00
Nick Sweeting
cb2dd1ee28 rename model Page to Snapshot 2019-04-30 23:44:51 -04:00
Nick Sweeting
d26f87efef make archivebox server work for urls, hashes, and timestamps 2019-04-30 23:13:21 -04:00
Nick Sweeting
95007d9137 split up utils into separate files 2019-04-30 23:13:04 -04:00
Nick Sweeting
1b8abc0961 move everything out of legacy folder 2019-04-27 17:26:24 -04:00
Nick Sweeting
c43a0e58df create config file on archivebox init 2019-04-26 18:31:50 -04:00
Nick Sweeting
d248684839 add index-only option to archivebox add and update 2019-04-25 18:59:41 -04:00
Nick Sweeting
ba7144f6f3 customize django admin ui chrome 2019-04-22 21:40:42 -04:00
Nick Sweeting
b44281919d add todo comment for mlazana 2019-04-22 19:10:22 -04:00
Nick Sweeting
71402c3d09 add manage command and shell welcome message 2019-04-22 19:08:01 -04:00
Nick Sweeting
834aaa1591 better template staticfile management with themes dir 2019-04-22 19:07:39 -04:00
Nick Sweeting
29ced7b5c8 allow running archivebox core commands from manage.py 2019-04-22 13:36:27 -04:00
Nick Sweeting
354895aef1 django admin to view links now working 2019-04-22 13:20:19 -04:00
Nick Sweeting
39a0ab3013 add pipenv, schedule cmd, logs dir, and lots more 2019-04-18 21:10:00 -04:00
Nick Sweeting
669bd6bee4 first views for archivebox server 2019-04-17 05:42:21 -04:00
Nick Sweeting
289a6ea30f fix database file location and init process 2019-04-17 05:42:09 -04:00
Nick Sweeting
35aa8c8902 clearer sql parsing and dumping 2019-04-17 03:50:41 -04:00
Nick Sweeting
cdb70c73df first working django model with archivebox-shell command and sql exporting 2019-04-17 03:49:18 -04:00
Nick Sweeting
bcfe17bc87 define database file in config.py 2019-04-11 03:40:37 -04:00
Nick Sweeting
51ae634ec9 working argparse based CLI with most commands implemented 2019-04-03 00:27:37 -04:00
Nick Sweeting
68b4c01c6b working archivebox command inside django legacy folder 2019-04-02 18:53:21 -04:00
Nick Sweeting
27708152d2 wip initial django setup 2019-04-02 16:36:41 -04:00