858 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
rafaelsideguide	009df6c930	Added crawl limit unit test ... I think this test is over relying on mocks but I have no idea on how to fix this without changing the code arch structure	2024-06-26 09:54:25 -03:00
Nicolas	45f2765601	Merge pull request #316 from snippet/types-webscraper ... add some types	2024-06-25 22:03:21 -03:00
Nicolas	768a131b5c	Merge pull request #318 from mendableai/bug/fix-custom-scrape-pdf-google-drive ... [Bug] Fixed the regex test for google drive pdf files	2024-06-25 18:27:11 -03:00
rafaelsideguide	5f69fc7677	Fixed the regex test	2024-06-25 18:24:01 -03:00
Nicolas	dbb22c8f0d	Merge pull request #317 from mendableai/bug/fix-clean-jobs ... [Bug] Fixed clean jobs	2024-06-25 17:50:55 -03:00
rafaelsideguide	d02829d335	fixed clean jobs	2024-06-25 17:49:29 -03:00
Jeff Pereira	199cbe8bcb	add some types	2024-06-25 12:20:25 -07:00
Nicolas	749b0c05dc	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-25 15:21:15 -03:00
Nicolas	e7be17db92	Nick: metadata fixes and lock duration for bull decreased to 2 hrs	2024-06-25 15:21:14 -03:00
Nicolas	f84fb4b331	Merge pull request #313 from snippet/google-search-term-fix ... fix multi-word search term issue: /search (w/o Serp)	2024-06-24 19:24:58 -03:00
Jeff Pereira	6ddf3a58a1	fix multi-word search term issue: /search (w/o Serp)	2024-06-24 14:21:52 -07:00
Nicolas	e5314ee8e7	Merge pull request #312 from mendableai/rafa/investigating-crawl-bugs ... [Bug] Fixed axios bug that were making jobs stuck on active queue	2024-06-24 16:52:34 -03:00
Nicolas	90b7fff366	Update crawler.ts	2024-06-24 16:52:01 -03:00
Nicolas	08c1fa799b	Update queue-worker.ts	2024-06-24 16:51:32 -03:00
rafaelsideguide	3ebdf93342	removed console.logs	2024-06-24 16:43:12 -03:00
Nicolas	56d42d9c9b	Nick:	2024-06-24 16:33:07 -03:00
rafaelsideguide	21d29de819	testing crawl with new.abb.com case ... many unnecessary console.logs for tracing the code execution	2024-06-24 16:25:07 -03:00
Nicolas	3c7b7e7242	NIck: fixes fallback	2024-06-23 18:59:08 -03:00
Nicolas	b394e64684	Merge pull request #308 from 100gle/fix-typo	2024-06-23 11:14:26 -04:00
Xiaoyue Lin	3624ed20f9	docs: Fix pydanti to pydantic	2024-06-23 22:27:48 +08:00
Eric Ciarla	22541362d7	Reduce web example bloat	2024-06-22 08:40:26 -04:00
Eric Ciarla	8e39083d8c	Update examples section	2024-06-21 15:40:46 -04:00
rafaelsideguide	5cf2beff92	Update clean-before-24h-complete-jobs.yml	2024-06-20 11:18:53 -03:00
Nicolas	3746b6207a	Merge pull request #303 from Lakr233/patch-1 ... Fix Broken Link	2024-06-19 11:13:25 -04:00
Lakr	3d1766ba7b	Fix Broken Link	2024-06-19 20:38:42 +08:00
Nicolas	c4252b6170	Merge pull request #302 from mendableai/cjp/email-to-posthog-logging ... Cjp/email to posthog logging	2024-06-18 21:30:42 -04:00
Caleb Peffer	e59ba758f5	Caleb: changed posthog logging so that It associates jobs with a group. No	2024-06-18 17:42:21 -07:00
Caleb Peffer	5a91d8425f	Caleb: solve for typechecking on idempotencyKey on my machine	2024-06-18 17:07:38 -07:00
Nicolas	32dde257a5	Merge pull request #301 from mendableai/bugfix/issue-291 ... [Bug] Fixed includeHTML to use cleanedHtml as response	2024-06-18 16:26:55 -04:00
rafaelsideguide	9c539e9113	Fixed includeHTML to use cleanedHtml as response	2024-06-18 16:26:54 -03:00
Nicolas	1c5a1dd487	Merge pull request #297 from AndyMik90/feat/removeTags-regex ... [Feat] Added support for RegEx in removeTags	2024-06-18 14:03:41 -04:00
Rafael Miller	f5a9acc4c6	Merge branch 'main' into feat/removeTags-regex	2024-06-18 14:39:59 -03:00
rafaelsideguide	9f7afd1e88	fix for some complex cases	2024-06-18 14:36:51 -03:00
Nicolas	8db8997daf	Nick: test suite + fly	2024-06-18 13:34:44 -04:00
Nicolas	d0c05accf6	Nick:	2024-06-18 13:21:50 -04:00
Nicolas	818751a256	Merge pull request #294 from mendableai/tests/e2e-to-unit ... [Test] Transcribed from e2e to unit tests for many cases	2024-06-18 13:09:22 -04:00
Nicolas	a3115004af	Merge pull request #296 from NeevJewalkar/sdk-update ... Added local host support for the javascript SDK	2024-06-18 13:03:37 -04:00
Nicolas	754c9fa08d	Update package.json	2024-06-18 12:58:57 -04:00
Nicolas	90a807c547	Update index.ts	2024-06-18 12:56:13 -04:00
Nicolas	26e8bfc23a	Merge branch 'main' into pr/296	2024-06-18 12:55:45 -04:00
Nicolas	b53ba58bc0	Merge pull request #282 from mendableai/nsc/rate-limiter-tests ... test: Rate Limit Unit Tests	2024-06-18 11:01:28 -04:00
Nicolas	3b6c9a8557	Merge pull request #298 from mendableai/feat/type-improvements ... [Feat] Improvements on response document types	2024-06-18 11:00:02 -04:00
rafaelsideguide	727e5de8c5	Update index.test.ts	2024-06-18 11:54:10 -03:00
rafaelsideguide	c54e797eb1	(╯°□°)╯︵ ┻━┻	2024-06-18 11:51:28 -03:00
rafaelsideguide	6e32522fa2	Improvements on response document types	2024-06-18 11:43:06 -03:00
rafaelsideguide	3c1af0aa33	Update ci.yml	2024-06-18 11:03:57 -03:00
rafaelsideguide	20f14bcf7f	Added some types	2024-06-18 10:55:07 -03:00
rafaelsideguide	c2fc69af1c	removed some e2e tests that are making the ci get stuck	2024-06-18 09:57:05 -03:00
rafaelsideguide	6c726a02eb	Moved to utils/removeUnwantedElements, added unit tests	2024-06-18 09:46:42 -03:00
AndyMik90	8b3c3aae91	Added support for RegEx in removeTags	2024-06-18 07:31:46 +02:00