105 commits

Author	SHA1	Message	Date
Folyd	926f1c0f68	Fix compile error	2020-03-13 23:20:50 +08:00
Folyd	640396a289	Skip when crate name equals description, otherwise will cause Unexpected token '[' error.	2020-03-13 22:44:23 +08:00
Folyd	5bc022f5bc	Change base_url to root path	2020-03-13 11:54:08 +08:00
Folyd	fc93600c15	Remove clean_tokens step when minify js	2020-03-12 23:13:38 +08:00
Folyd	6fe58a7936	Clean token when minify js	2020-03-06 00:54:12 +08:00
Folyd	6aad602f55	Add clippy lints search	2020-03-05 23:33:32 +08:00
Folyd	2280911226	Generate lints index	2020-03-04 01:01:05 +08:00
Folyd	ba523999d5	Split dependencies with feature flags	2020-02-27 17:12:21 +08:00
Folyd	2f449ff599	Parse csv from tar file directly to generate crates index	2020-02-27 15:45:15 +08:00
Folyd	aa1548276d	Speed up crates index	2020-02-27 11:51:37 +08:00
Folyd	36dd233e88	Build crates index way from the dumped database file, discard old API crawl way due to speed and accuracy issues	2020-02-27 01:09:57 +08:00
Folyd	72bb099911	Check char boundary to prevent panic	2020-02-25 15:15:52 +08:00
Folyd	cf5894e129	Minifier letter change to prefix and suffix combination	2020-02-24 00:30:04 +08:00
Folyd	61984354b6	Fix try truncate unwrap panic	2020-02-23 21:24:28 +08:00
Folyd	3722218f62	Implement TryTruncate trait for String	2020-02-23 19:45:54 +08:00
Folyd	4f4035f4eb	Try bump crates index number to 20K	2020-02-23 14:46:39 +08:00
Folyd	8d1d148b84	Cargo clippy	2020-02-22 17:05:28 +08:00
Folyd	28dcf4c822	Minify books index file	2020-02-22 16:22:41 +08:00
Folyd	0e83e7efc6	Implement Serialize for Page struct	2020-02-22 16:11:01 +08:00
Folyd	4c8764e5ad	Fix parse book chapters issues	2020-02-22 15:56:22 +08:00
Folyd	8f26302044	Parse book chapters recursively	2020-02-22 15:44:35 +08:00
Folyd	e0a35b8c47	Add basic book search feature	2020-02-17 11:54:27 +08:00
Folyd	d74efaffe0	Optimize index javascript file structure	2020-02-12 15:12:30 +08:00
Folyd	bfa6c91ef5	Adjust rust directory structure, add book.rs to fetch rust book chapters	2020-02-11 16:05:43 +08:00
Folyd	7d7f2e9650	Minify crates-index's crate id	2020-02-08 15:39:25 +08:00
Folyd	7d8a6efb85	Increase timeout to 15 seconds, add retry fetch crates strategy	2020-02-08 12:25:56 +08:00
Folyd	972a5c00c7	Add 10 seconds timeout, crawl up to top 15K crates	2020-02-07 20:55:57 +08:00
Folyd	986ac2c7b5	Remove crates-index's documentation field, rename crates-index.js to index.js	2020-02-07 12:04:39 +08:00
Folyd	4d680cd86a	Add minify letters	2020-01-13 19:27:39 +08:00
Folyd	da0206d1cb	Change MINIFY_LETTERS to alphanumeric letters	2020-01-12 15:29:46 +08:00
Folyd	cee3a0231d	Sort frequency word by score	2020-01-12 14:47:29 +08:00
Folyd	676d2acf22	Apply cargo clippy suggestions	2020-01-10 19:42:58 +08:00
Folyd	187ea4464a	Move minify function to Minifer struct	2020-01-10 19:36:56 +08:00
Folyd	9c0e254b7e	Write mapping into crates-index.js	2020-01-10 12:11:42 +08:00
Folyd	8e63b147eb	Use MappingMinifier to minify crate's description	2020-01-10 11:54:35 +08:00
Folyd	57df7feb0e	Generate crates description frequency words mapping	2020-01-09 21:42:33 +08:00
Folyd	9db33fac09	Add User-Agent and limit the request rate to comply crates.io crawler policy	2020-01-07 23:02:21 +08:00
Folyd	b031a2aec6	Fix update crates index failed bug, upgrade reqwest and tokio	2019-12-31 11:30:00 +08:00
Folyd	a8762d48b4	Support specify crates index path in the args	2019-12-22 13:57:48 +08:00
Folyd	2214017845	Minify description to reduct crates-index file size	2019-12-10 00:37:06 +08:00
Folyd	bbdcd4f37f	Aggregate crates index common string into array to reduce file size	2019-11-20 17:30:57 +08:00
Folyd	ed9892a20d	Cargo fmt	2019-11-20 15:57:18 +08:00
Folyd	6787ecfb3d	Use minifier crate to replace null keyword	2019-11-20 15:50:00 +08:00
Folyd	47b4a69ff9	Cargo clippy	2019-11-19 21:20:40 +08:00
Folyd	c562716a5f	Fix mistake comment and some stuffs	2019-11-19 20:05:41 +08:00
Folyd	92ce32ae80	Add crates-index.js, complete basic crate search feature	2019-11-19 19:13:23 +08:00
Folyd	ec803c3bd0	Optimize format to reduce crates-index.js file size	2019-11-19 18:59:07 +08:00
Folyd	ce091b1b32	Remove crate homepage/repository/downloads fields	2019-11-19 17:54:17 +08:00
Folyd	591640129d	Lowercase all crate ids	2019-11-18 18:26:48 +08:00
Folyd	ff35037242	Convert crate vec to HashMap	2019-11-18 18:16:55 +08:00
Folyd	da6d203b25	Minify javascript null to variable N	2019-11-17 00:38:50 +08:00
Folyd	a71ca571ff	Minify crates homepage, documentaion, repository url	2019-11-17 00:16:42 +08:00
Folyd	0b9a0e5c24	Minify crate fields to reduce the crate-index.js file size	2019-11-15 18:41:46 +08:00
Folyd	e95b31827a	Generate basic crate-ids.js file	2019-11-15 18:03:22 +08:00
Folyd	7a7af518ad	Add rust directory	2019-11-13 16:48:26 +08:00