852 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Nicolas	11b6d5afa5	Update fly.toml	2024-06-12 18:00:22 -07:00
Nicolas	67dc46b454	Nick: clusters	2024-06-12 17:53:04 -07:00
Nicolas	bad0f57134	Merge pull request #272 from mendableai/bug/issue-258 ... [Feat] Added jobId to webhook data	2024-06-12 12:29:50 -07:00
rafaelsideguide	d20af257ba	Added jobId to webhook data	2024-06-12 15:38:41 -03:00
rafaelsideguide	e37d151404	added parsePDF option to pageOptions ... user can decide if they are going to let us take care of the parse or they are going to parse the pdf by themselves	2024-06-12 15:06:47 -03:00
Rafael Miller	48f6c19a05	Merge pull request #269 from mendableai/feat/allowbackwardcrawling-option ... [Feat] Added allowBackwardCrawling option	2024-06-12 11:34:39 -03:00
rafaelsideguide	01c9f071fa	fixed	2024-06-12 11:27:06 -03:00
rafaelsideguide	dc6acbf1f0	Merge remote-tracking branch 'origin/main' into feat/allowbackwardcrawling-option	2024-06-12 11:01:05 -03:00
Nicolas	f93231499f	Merge pull request #265 from mendableai/feat/issue-264 ... [Feat] Added route to clean completed jobs and a github action cron that triggers every 24h	2024-06-11 21:33:52 -07:00
Nicolas	705f5ebe78	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-11 15:49:35 -07:00
Nicolas	d4df6f049d	Nick:	2024-06-11 15:49:30 -07:00
Nicolas	45dee63943	Merge pull request #262 from mendableai/nsc/webhook-self-host-fix ... Only fetch webhook from db if self host webhook not set and using db auth	2024-06-11 15:46:57 -07:00
rafaelsideguide	157fbe4a1e	added bull auth key	2024-06-11 17:52:01 -03:00
rafaelsideguide	df3a678cf4	getting back the cancel test, this should work	2024-06-11 17:46:56 -03:00
rafaelsideguide	def2ba9987	added tests	2024-06-11 17:46:25 -03:00
Nicolas	c08db830df	Merge pull request #268 from mendableai/nsc/abs-path-fix ... Fixed bugs associated with absolute path replacements	2024-06-11 13:20:42 -07:00
Nicolas	1e3e06a1d5	Update replacePaths.test.ts	2024-06-11 13:02:39 -07:00
Nicolas	2239e03269	Update replacePaths.test.ts	2024-06-11 12:54:02 -07:00
Nicolas	520739c9f4	Nick: fixed bugs associated with absolute path replacements	2024-06-11 12:43:16 -07:00
Nicolas	788abdce6e	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-11 12:08:51 -07:00
Nicolas	b87725c683	Update openapi.json	2024-06-11 12:08:49 -07:00
rafaelsideguide	ee282c3d55	Added allowBackwardCrawling option	2024-06-11 15:24:39 -03:00
rafaelsideguide	a9f93c2f1e	Added route to clean completed jobs and a github action cron that triggers every 24h	2024-06-11 14:18:05 -03:00
Eric Ciarla	06b0d01fd4	Update examples	2024-06-11 12:23:36 -04:00
Eric Ciarla	00c23855b1	Update examples	2024-06-11 11:46:35 -04:00
Nicolas	da38dad9a7	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-10 18:26:31 -07:00
Nicolas	9390816c1b	Update openapi.json	2024-06-10 18:26:25 -07:00
Nicolas	15e791ffb1	Merge pull request #263 from mendableai/nsc/pageoptions-crawler ... ignoreSitemap feature, pageOptions now respected in the initial crawl as well	2024-06-10 18:22:51 -07:00
Nicolas	f6b06ac27a	Nick: ignoreSitemap, better crawling algo	2024-06-10 18:12:41 -07:00
Nicolas	1bd0327e1a	Merge branch 'main' into nsc/pageoptions-crawler	2024-06-10 17:15:10 -07:00
Nicolas	149d79a529	Merge pull request #260 from mendableai/nsc/fix-deadlocks ... Moving from fetch to axios and preventing deadlocks by setting timeouts on fallbacks	2024-06-10 17:06:33 -07:00
Nicolas	99f2ffd6d5	Update webhook.ts	2024-06-10 17:03:10 -07:00
Nicolas	7ae9778642	Update single_url.ts	2024-06-10 16:57:31 -07:00
Nicolas	913c1dd568	Nick: fetch -> axios and fix timeouts	2024-06-10 16:49:03 -07:00
Nicolas	3091f0134c	Nick:	2024-06-10 16:27:10 -07:00
Matt Joyce	827354a116	Added logging to python sdk FIRECRAWL_LOGGING_LEVEL ... Instantiates the logger early and depends on env to set.	2024-06-10 21:21:23 +10:00
Nicolas	aafd23fa8a	Merge pull request #252 from mattjoyce/fix-208-py-sdk-interval-poll-name ... Fix 208 py sdk interval poll name	2024-06-08 21:33:17 -07:00
Matt Joyce	6fd9ce1c89	type hints and linting	2024-06-08 11:46:52 +10:00
Matt Joyce	7477c5e5bd	Use error handler consistently	2024-06-08 11:28:51 +10:00
Matt Joyce	9f306736af	More detailed error handling	2024-06-08 11:18:30 +10:00
Matt Joyce	c71ea7a795	Prepare headers consistently	2024-06-08 11:08:26 +10:00
Matt Joyce	8f9a165c2f	Lint - whitespace	2024-06-08 08:03:02 +10:00
Matt Joyce	5f0df596ec	Align param name with JS SDK ... timeout becomes poll_interval	2024-06-08 07:37:08 +10:00
Nicolas	f24ca76618	Nick: removing rate limit emails for now	2024-06-07 10:39:11 -07:00
rafaelsideguide	556c57648e	Update fly.yml	2024-06-07 09:40:40 -03:00
rafaelsideguide	0e89f8b9a3	fixing workflow	2024-06-07 09:35:56 -03:00
Nicolas	98d82c4cec	Update search.ts	2024-06-06 20:02:21 -07:00
Nicolas	5e80f8af87	Nick: llm extract 50	2024-06-06 18:35:44 -07:00
rafaelsideguide	7b7a6f8a39	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-06-06 17:51:28 -03:00
rafaelsideguide	f2695df215	updated sdk versions	2024-06-06 17:51:12 -03:00