Apkallum
1f91f5b102
remove commented lines
2020-07-16 19:42:20 -04:00
apkallum
b7785c4138
use dateparser for parsing, let it handle error
2020-07-16 19:38:38 -04:00
Nick Sweeting
58ac44c867
Merge pull request #365 from cdvv7788/hotfix/#330
...
fix: htmlencode titles before rendering the static html index and detail
2020-07-16 14:45:00 -04:00
Cristian
83e5b019e4
feat: Add canonical link http header to the static response
2020-07-16 12:49:26 -05:00
Cristian
f845224d6f
fix: htmlencode titles before rendering the static html index and detail
2020-07-16 09:20:33 -05:00
apkallum
98dda68897
fix: timestamp comparison in to_json function
2020-07-14 10:26:33 -04:00
Cristian
5e2bf73f04
fix: Bugs related to add() refactor
2020-07-13 14:48:25 -05:00
Nick Sweeting
a79dd4685a
make snapshots unique again
2020-07-13 12:21:52 -04:00
Nick Sweeting
ae208435c9
fix the add links form
2020-07-13 12:21:37 -04:00
Nick Sweeting
215d5eae32
normal git clone instead of mirror
2020-07-13 11:41:37 -04:00
Nick Sweeting
b4ce20cbe5
write link details json before and after archiving
2020-07-13 11:41:27 -04:00
Nick Sweeting
d159e674e1
write stderr instead of stdout for version info
2020-07-13 11:41:18 -04:00
Nick Sweeting
4c4b1e6a4b
fix link creation
2020-07-13 11:33:38 -04:00
Nick Sweeting
d3bfa98a91
fix depth flag and tweak logging
2020-07-13 11:26:34 -04:00
Nick Sweeting
354a63ccd4
dont dedupe snapshots in sqlite on every run
2020-07-13 11:25:43 -04:00
Nick Sweeting
dfb83b4f27
add AttributeDict
2020-07-13 11:24:49 -04:00
Nick Sweeting
16f3746712
check source dir at the end of checking data dir
2020-07-13 11:24:36 -04:00
Nick Sweeting
96b1e4a8ec
accept local paths as valid link URLs when parsing
2020-07-13 11:22:58 -04:00
Nick Sweeting
7cbd068c95
add flake8
2020-07-13 11:22:23 -04:00
Nick Sweeting
5b571aa166
Merge pull request #356 from cdvv7788/depth-flag
2020-07-13 05:05:36 -04:00
Nick Sweeting
26e97f242a
Merge pull request #360 from apkallum/django
...
fix legacy index.html
2020-07-08 19:51:37 -04:00
Apkallum
09b4438c9f
fix legacy index.html
2020-07-08 17:54:01 -04:00
Cristian
d476b13007
fix: Add missing permission to add view (post)
2020-07-08 14:46:31 -05:00
Cristian
4ebf929606
refactor: Change wording on CLI help
2020-07-08 08:30:07 -05:00
Cristian
f12bfeb322
refactor: Change add() to receive url and depth instead of import_str and import_path
2020-07-08 08:17:47 -05:00
Cristian
c1d8a74e4f
feat: Make input sent via stdin behave the same as using args
2020-07-07 15:49:40 -05:00
Cristian
b68c13918f
feat: Disable stdin from archivebox add
2020-07-07 12:39:36 -05:00
Cristian
a6940092bb
feat: Make sure that depth can only be either 1 or 0
2020-07-07 10:25:02 -05:00
Cristian
32e790979e
feat: Enable depth=1 functionality
2020-07-07 10:07:44 -05:00
Cristian
2db0324539
feat: depth=0 crawls the current page only
2020-07-07 09:49:28 -05:00
Cristian
8b22a2a7dd
feat: Enable --depth flag (still does nothing)
2020-07-07 09:10:36 -05:00
Nick Sweeting
ea93e05c3c
Merge pull request #351 from cdvv7788/view_feed_support
...
Allow feed loading from the add links view
2020-07-02 17:18:16 -04:00
Cristian
8bdfa18a3f
feat: Allow feed loading from the add links view
2020-07-02 15:54:25 -05:00
Nick Sweeting
7c428f40c8
fix stdin link importing
2020-07-02 13:31:05 -04:00
Nick Sweeting
f373df7bd4
update helptext to clarify adding links
2020-07-02 13:23:40 -04:00
Cristian
528fc8f1f6
fix: Improve encoding detection for rss+xml content types
2020-07-02 12:11:23 -05:00
Nick Sweeting
0c48449aa6
fix subcommand and args not being passed
2020-07-02 04:00:51 -04:00
Nick Sweeting
322be6b292
move main into cli init and remove circular import layer
2020-07-02 03:53:39 -04:00
Nick Sweeting
3ec97e5528
fix git conflict commited by accident
2020-07-02 03:22:37 -04:00
Nick Sweeting
8840ad72bb
remove circular import possibilities
2020-07-02 03:13:35 -04:00
Cristian
364c5752d8
feat: Handle empty URL case
2020-07-01 13:14:20 -05:00
Cristian
c971e00c9c
feat: Add stdout from process to the template
2020-07-01 12:23:59 -05:00
Nick Sweeting
bc1f925542
fix snapshot getter in sql saving
2020-06-30 15:24:23 -04:00
Nick Sweeting
dda3542d60
bump sql updated time after every link details save
2020-06-30 13:45:47 -04:00
Nick Sweeting
264cae2b6a
add migration
2020-06-30 06:42:17 -04:00
Nick Sweeting
c415420f33
improve sort columns and UI placeholders
2020-06-30 06:41:48 -04:00
Nick Sweeting
178f6ac1a6
switch to django admin snapshots list as new homepage
2020-06-30 05:56:17 -04:00
Nick Sweeting
9f440c2cf8
use requests.get to fetch and decode instead of urllib
2020-06-30 05:55:54 -04:00
Nick Sweeting
df593dea0a
fix missing imports
2020-06-30 05:55:34 -04:00
Nick Sweeting
e6830284c5
add script to auto-generate config subs
2020-06-30 03:35:20 -04:00
Nick Sweeting
602e141f08
fix config file atomic writing bugs
2020-06-30 02:04:16 -04:00
Nick Sweeting
4da1d82b86
use python builtin os.chmod instead of subprocess call
2020-06-30 01:49:08 -04:00
Nick Sweeting
44bd6866ac
colorize hints in CLI output more consistently
2020-06-30 01:48:41 -04:00
Nick Sweeting
c68543af74
fix text encoding for subprocesses
2020-06-30 01:14:35 -04:00
Nick Sweeting
79b19ddf35
use atomic writes for config file writing as well
2020-06-30 01:12:06 -04:00
Nick Sweeting
cf01eff9c7
tweak log output text
2020-06-30 01:10:54 -04:00
Nick Sweeting
912cbb22d5
fix CURL_USER_AGENT missing from config stubs
2020-06-30 01:10:21 -04:00
Nick Sweeting
4783daa6fa
use full python version instead of only major,minor
2020-06-30 01:08:14 -04:00
Nick Sweeting
1ecb94fe09
check PYTHON_ENCODING correclty when lowercase
2020-06-30 01:07:38 -04:00
Nick Sweeting
094beac1d9
fix add page redirect
2020-06-29 21:26:59 -04:00
Nick Sweeting
b9e17fa0d1
rename archivebox-info to archivebox-status
2020-06-25 23:32:01 -04:00
Nick Sweeting
5c2bbe7efe
bufixes
2020-06-25 22:14:40 -04:00
Nick Sweeting
cb67b09f9d
Merge branch 'master' into django
2020-06-25 21:30:29 -04:00
Nick Sweeting
fae2fdaf2b
fix if to elif bug in index snapshot sql writing
2020-06-25 17:48:27 -04:00
Nick Sweeting
43c471e4af
cli experience improvements
2020-06-25 17:47:55 -04:00
Nick Sweeting
31a454db36
show loading message in title before its fetched
2020-04-22 21:15:42 -04:00
Nick Sweeting
9fc431102b
better timestamp handling
2020-04-22 21:15:15 -04:00
Nick Sweeting
afacc5c5da
use fallback random secret key
2020-04-22 21:14:43 -04:00
Nick Sweeting
2829b18b0b
new save playlists option
2020-04-22 21:14:20 -04:00
misha
a3a048d43e
16042020
2020-04-16 14:48:23 +03:00
misha
1aa2a5b069
15042020
2020-04-15 11:54:53 +03:00
comsomisha
bb580533f7
0504202002
2020-04-05 18:35:13 +03:00
misha
18f0f66f1e
05042020
2020-04-05 18:22:59 +03:00
Nick Sweeting
7d699578e8
implement add page
2020-03-27 08:38:22 -04:00
michael.bub
c79ce2b1f5
guess encoding via chardet if available
2020-02-15 13:58:07 +01:00
Nick Sweeting
464b5b7391
Update archive.py
2020-02-03 19:25:08 -05:00
Julian Berman
2e4b506c21
Update archivebox/index.py
...
Co-Authored-By: Nick Sweeting <git@sweeting.me>
2019-10-19 12:47:42 -04:00
Julian Berman
eb641b3e35
Fix a leaking file descriptor.
2019-10-13 18:18:26 -04:00
Mashiat Sarker Shakkhar
2ba17136e0
index/schema.py: Make cmd_version actually optional.
...
During migration, we may have to import json index file which was
produced with an older version. If the index file is missing
cmd_version, migration will fail.
2019-09-10 23:51:49 -04:00
Mashiat Sarker Shakkhar
0bb216ce02
util.py: Use dateparser to parse date strings.
2019-09-10 23:51:09 -04:00
Nick Sweeting
bfa1f0a217
Update __init__.py
2019-07-09 13:21:01 -04:00
Nick Sweeting
e2b714ba95
Add chrome binary path for BSD chromium
2019-05-20 14:29:28 -04:00
t-mw
78bc52abf2
Fix json export date parsing
...
The format string for `datetime.strptime` did not match the example line in the comment below, since `%z` matches ["UTC offset in the form ±HHMM"](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior ).
2019-05-18 22:32:09 +02:00
Starbeamrainbowlabs
3b0236b087
Add prefers-color-scheme: dark support
2019-05-14 23:54:17 +01:00
Nick Sweeting
ca9c9ef956
add warning about running manage.py directly
2019-05-06 17:16:20 -04:00
Nick Sweeting
3c3b2ee621
expose more django server config options
2019-05-02 19:15:16 -04:00
Nick Sweeting
d016f1efb5
bump version
2019-05-02 17:52:43 -04:00
Nick Sweeting
e0489d77e7
bump the version
2019-05-02 17:39:45 -04:00
Nick Sweeting
ecfca13b6d
fix present folders docstring
2019-05-02 15:20:21 -04:00
Nick Sweeting
500534f4be
fix missing comma in staticfile extensions list
2019-05-02 15:17:16 -04:00
Nick Sweeting
204de37eb9
fix parsing errors for older archive index formats
2019-05-01 02:28:48 -04:00
Nick Sweeting
1ac99621ab
show progress during validate_links
2019-05-01 02:28:26 -04:00
Nick Sweeting
f60b5ed867
better stdin handling
2019-05-01 02:28:12 -04:00
Nick Sweeting
ad3898add6
allow forcing init in dirty directory
2019-05-01 02:27:50 -04:00
Nick Sweeting
cb2dd1ee28
rename model Page to Snapshot
2019-04-30 23:44:51 -04:00
Nick Sweeting
d26f87efef
make archivebox server work for urls, hashes, and timestamps
2019-04-30 23:13:21 -04:00
Nick Sweeting
95007d9137
split up utils into separate files
2019-04-30 23:13:04 -04:00
Nick Sweeting
daf5951897
allow passing debug flag to archivebox server
2019-04-30 23:11:41 -04:00
Nick Sweeting
0ba821b351
better return status from list cmd
2019-04-30 23:11:07 -04:00
Nick Sweeting
158f145d9a
move docstrings to main.py out of cli files
2019-04-30 23:10:48 -04:00
Pig Monkey
34270b2b12
only use stdin if it has a value
...
Closes #228
2019-04-30 17:25:41 -07:00
Nick Sweeting
4c924ccf7d
add instructions for building sphinx docs
2019-04-28 16:46:25 -04:00
Nick Sweeting
b86f1c3cbb
fix python dir getting docs dir
2019-04-27 21:13:56 -04:00
Nick Sweeting
94c76d7b55
update version path
2019-04-27 21:11:17 -04:00
Nick Sweeting
27fcb088af
read version from version file in conf.py
2019-04-27 21:09:32 -04:00
Nick Sweeting
4235b38009
move version file
2019-04-27 17:57:35 -04:00
Nick Sweeting
905998fb41
fix uneeded import in sphinx conf
2019-04-27 17:52:46 -04:00
Nick Sweeting
248a4da99d
fix docs to use read-the-docs theme
2019-04-27 17:43:14 -04:00
Nick Sweeting
d4c4a2d4ac
add sphinx autogenerated documentation
2019-04-27 17:27:25 -04:00
Nick Sweeting
1b8abc0961
move everything out of legacy folder
2019-04-27 17:26:24 -04:00
Nick Sweeting
c43a0e58df
create config file on archivebox init
2019-04-26 18:31:50 -04:00
Nick Sweeting
9dcf7972fa
fix bug where USE_THING configs were being overwritten later
2019-04-26 14:43:47 -04:00
Nick Sweeting
1edb14b229
wip config reset command
2019-04-26 14:43:23 -04:00
Nick Sweeting
2b4244bb52
allow setting config via aliases and show side effect changes in stderr
2019-04-26 14:43:13 -04:00
Nick Sweeting
4d6ad7a65d
add archivebox config command and move config into sections
2019-04-25 19:03:38 -04:00
Nick Sweeting
583d77bc31
use exit status 2 for errors to avoid confusion with no results status
2019-04-25 19:02:44 -04:00
Nick Sweeting
ce33a46866
make index_only option skip archive updating
2019-04-25 19:01:58 -04:00
Nick Sweeting
d2a51f68c0
fix bug in archivebox info using exausted generator
2019-04-25 19:01:22 -04:00
Nick Sweeting
df4307c4ff
move config loading into function for easier runtime reloading
2019-04-25 19:00:25 -04:00
Nick Sweeting
d248684839
add index-only option to archivebox add and update
2019-04-25 18:59:41 -04:00
Nick Sweeting
95a7d3d1de
add space in human readable sizes
2019-04-24 11:45:04 -04:00
Nick Sweeting
56d0b2c088
new methods for detecting valid/invalid data dirs on init
2019-04-24 11:40:10 -04:00
Nick Sweeting
ae782a1a0c
allow filtering for specific files when getting dir size
2019-04-24 11:39:34 -04:00
Nick Sweeting
168d253c17
show human readable filesize in version info
2019-04-24 11:39:02 -04:00
Nick Sweeting
b0ace38273
warn user that no admins exist when starting runserver
2019-04-24 11:38:37 -04:00
Nick Sweeting
03470699d3
ignore json parsing errors when loading link jsons
2019-04-24 11:38:13 -04:00
Nick Sweeting
e91cdfbc88
add rudimentary method to parse back html index into urls
2019-04-24 11:37:51 -04:00
Nick Sweeting
3825ddc095
add method to get admins list
2019-04-24 11:37:30 -04:00
Nick Sweeting
327da95928
add classes to make it easier to parse html links index
2019-04-24 11:37:08 -04:00
Nick Sweeting
8f63f24002
better Link string and is_archived methods
2019-04-24 11:36:47 -04:00
Nick Sweeting
11fd436305
use new mypy TypedDict to manage config typing
2019-04-24 11:36:14 -04:00
Nick Sweeting
461a8b0d71
use new config vars for html index writing
2019-04-24 04:10:02 -04:00
Nick Sweeting
6235d859dc
seamless archive init and migration process across versions
2019-04-24 04:09:50 -04:00
Nick Sweeting
5ef5415f3f
rename archive toggles from FETCH_ to SAVE_ for clarity
2019-04-24 04:09:25 -04:00
Nick Sweeting
0f2497a2a6
add functions to parse link details jsons and list+apply migrations
2019-04-24 04:07:46 -04:00
Nick Sweeting
0b27f33d2e
fix sql link parsing and writing
2019-04-24 04:06:54 -04:00
Nick Sweeting
735db84022
add some missing defaults to Archivebox.conf.default
2019-04-23 22:36:43 -04:00
Nick Sweeting
ba7144f6f3
customize django admin ui chrome
2019-04-22 21:40:42 -04:00
Nick Sweeting
b44281919d
add todo comment for mlazana
2019-04-22 19:10:22 -04:00
Nick Sweeting
71402c3d09
add manage command and shell welcome message
2019-04-22 19:08:01 -04:00
Nick Sweeting
834aaa1591
better template staticfile management with themes dir
2019-04-22 19:07:39 -04:00
Nick Sweeting
f0f516e853
check for data folder when running most subcommands
2019-04-22 19:06:48 -04:00
Nick Sweeting
2f0dbeebc1
update docstrings and comments
2019-04-22 14:42:04 -04:00
Nick Sweeting
ab68819332
add archivebox info command to scan data dir
2019-04-22 14:34:30 -04:00
Nick Sweeting
50b947f41d
add md5 hashes to dependencies dict
2019-04-22 14:34:12 -04:00
Nick Sweeting
29ced7b5c8
allow running archivebox core commands from manage.py
2019-04-22 13:36:27 -04:00
Nick Sweeting
168e578ea4
fix bad default in scheduler
2019-04-22 13:21:08 -04:00
Nick Sweeting
354895aef1
django admin to view links now working
2019-04-22 13:20:19 -04:00
Nick Sweeting
f489dd96a9
fix archivebox remove rejecting stdin patterns
2019-04-22 13:19:47 -04:00
Drewry Pope
332a32f4f9
Resolve 3 typos in util.py
2019-04-20 02:59:44 -05:00