173 Commits

Author	SHA1	Message	Date
Harsh Gupta (aider)	d15b721bfa	refactor: Remove rate limiting from crawler.ts	2024-08-14 13:43:04 +05:30
Harsh Gupta	aa862d4247	fix: Refactor crawler.ts by removing unused imports and code	2024-08-14 13:43:03 +05:30
Yanlong Wang	df58fcb3fa	fix: alleviate search performance issue	2024-08-09 15:03:24 +08:00
Yanlong Wang	eb74e9c6f8	fix: remove select element from markdown to walk around turndown performance issue	2024-08-09 10:55:36 +08:00
Yanlong Wang	e4ef6cb0f9	chore: reduce fetch count in search	2024-08-09 10:29:50 +08:00
Yanlong Wang	e529369ba6	fix: search with failed pages	2024-08-08 15:49:23 +08:00
Yanlong Wang	0dd05b5dab	chore: tweak concurrency	2024-08-06 17:58:27 +08:00
Yanlong Wang	7af2bde01f	fix: html rebasing with <base> tag	2024-08-06 13:15:10 +08:00
Yanlong Wang	40e91853e2	fix	2024-08-02 20:10:17 +08:00
Yanlong Wang	cda0f371e1	feat: updated rate policy	2024-08-02 19:39:51 +08:00
Yanlong Wang	0a2c0932fd	fix	2024-08-02 17:13:50 +08:00
Yanlong Wang	ee632199df	fix	2024-08-02 17:12:10 +08:00
Yanlong Wang	0a33207f8f	fix: another approach to suspected DoS abuse	2024-08-02 17:04:13 +08:00
yanlong.wang	e658e8102c	fix	2024-08-01 20:07:39 +08:00
yanlong.wang	f4f189c8e6	fix	2024-08-01 19:51:53 +08:00
yanlong.wang	54fa5feb7f	fix	2024-08-01 19:49:40 +08:00
yanlong.wang	d0a922144d	fix	2024-08-01 19:48:00 +08:00
yanlong.wang	6fb5df97cc	fix: abuse of flooding elements	2024-08-01 19:34:39 +08:00
yanlong.wang	8b7af6d076	fix: ignore match all target selectors for performance	2024-07-31 14:06:22 +08:00
yanlong.wang	a08218506e	fix: truncate svg in jsdom	2024-07-31 13:12:57 +08:00
Yanlong Wang	4e5aff3332	debug: log jsdom and turndown operations	2024-07-31 11:12:12 +08:00
Yanlong Wang	0f239793d2	fix: also recover screenshot pricing	2024-07-30 20:11:55 +08:00
Yanlong Wang	d3f3a8502a	fix: revert screenshot behavior and introduce pageshot	2024-07-30 20:09:06 +08:00
yanlong.wang	57cbae864e	fix: jsdom, cache tolerance, screenshot pricing	2024-07-30 17:47:26 +08:00
yanlong.wang	77c8480ca6	feat: with-iframe and full-page screenshot	2024-07-30 15:08:09 +08:00
Yanlong Wang	e4d46e7acb	fix: count parameter	2024-07-25 19:46:28 +08:00
Yanlong Wang	1c4b64fe04	feat: bring your own html	2024-07-25 16:54:28 +08:00
Yanlong Wang	78ea13b101	fix	2024-07-25 15:14:36 +08:00
Yanlong Wang	9bcde30f11	fix	2024-07-25 14:56:19 +08:00
Yanlong Wang	09dbbd3b0f	fix: retry for brave search	2024-07-25 08:56:02 +08:00
Yanlong Wang	ec7c2ab52c	fix: scrap timing	2024-07-24 22:59:00 +08:00
yanlong.wang	873994397a	fix: allow POST with url	2024-07-24 15:08:32 +08:00
yanlong.wang	786b1828b7	fix: brave search operators in headers	2024-07-23 15:50:28 +08:00
yanlong.wang	efe7a61e3b	fix: timeout parameter	2024-07-18 12:33:19 +08:00
yanlong.wang	61ff011c13	fix: stop early return when timeout is explicitly defined	2024-07-18 11:41:58 +08:00
yanlong.wang	d0e2920163	feat: expose brave search operators explicitly	2024-07-15 18:11:58 +08:00
Yanlong Wang	336931b5e8	chore: tweak deployment	2024-07-12 15:10:01 +08:00
yanlong.wang	c6634298c4	fix: provide empty content field	2024-07-09 18:14:19 +08:00
Yanlong Wang	f6a183f87a	fix	2024-07-06 01:00:37 +08:00
Yanlong Wang	c69ec77c60	fix	2024-07-06 00:59:37 +08:00
Yanlong Wang	c9ef602a94	fix: potential firebase blockade	2024-07-06 00:25:48 +08:00
yanlong.wang	1bcb5a742e	fix: dos abuse	2024-07-01 18:40:14 +08:00
Yanlong Wang	0a09aeb967	fix	2024-07-01 11:06:30 +08:00
Yanlong Wang	8951578ef1	chore	2024-07-01 00:29:12 +08:00
Yanlong Wang	858e52fe2e	chore: tweak cpu	2024-06-30 22:19:03 +08:00
yanlong.wang	62fb6cff94	feat: keepImgDataUrl	2024-06-24 15:18:53 +08:00
yanlong.wang	1084b16c84	fix: typo	2024-06-20 18:31:56 +08:00
yanlong.wang	579f259cb9	fix: detect when readability does not work	2024-06-20 18:20:13 +08:00
yanlong.wang	eaa06781e3	fix: normalize-url pollution	2024-06-20 14:53:25 +08:00
yanlong.wang	6f37e5d3b4	feat: x-remove-selector	2024-06-18 18:07:38 +08:00