Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 11:42:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
1,314 Commits 273 Branches 1 Tag 48 MiB
fe2e8c0b7a
Commit Graph

995 Commits

Author SHA1 Message Date
Gergő Móricz
5a6570cba2 fix(webhooks): call back with parent crawl ID 2024-08-16 17:42:42 +02:00
rafaelsideguide
7d324bd2c8 Create checkCredits.ts 2024-08-16 11:21:52 -03:00
Nicolas
ec361609d2 Nick: added growth-2x plan
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-15 18:37:19 -04:00
Gergő Móricz
8b7569f8f3 add zod, create middleware, update openapi declaration, add crawl logic 2024-08-15 23:30:33 +02:00
Gergő Móricz
4165de1773 v1 restructure 2024-08-15 21:51:59 +02:00
Gergő Móricz
af08ab0b1a fix bad module resolution 2024-08-15 21:51:52 +02:00
Nicolas
c917c8fbcd Merge branch 'main' into v1-webscraper 2024-08-15 15:14:29 -04:00
Nicolas
32c6b1f136 Nick: remove active job alerts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-15 14:50:30 -04:00
Gergő Móricz
0c14366720 fix: add checkandupdateurl to crawlPreview 2024-08-15 20:30:25 +02:00
Nicolas
81b2479db3
Merge pull request #459 from mendableai/feat/queue-scrapes 
feat: Move scraper to queue
 2024-08-15 14:19:55 -04:00
Gergő Móricz
fc08ff450d search port 2024-08-15 20:10:59 +02:00
Nicolas
86326f34e9 Update single_url.test.ts 2024-08-15 13:48:42 -04:00
Gergő Móricz
129a882bcc fix(scrape): give scrapes their real job id 2024-08-15 19:29:47 +02:00
Gergő Móricz
965a5817d1 fix(queue-worker): log jobs correctly 2024-08-15 19:27:15 +02:00
Gergő Móricz
dad9d353d9 use thomas's url validation 2024-08-15 19:19:02 +02:00
Gergő Móricz
e3279274f1 fix: make playground crawl work 2024-08-15 19:14:32 +02:00
Gergő Móricz
c5597bc722 fix: robots.txt laoding 2024-08-15 19:11:07 +02:00
Gergő Móricz
29f0d9ec94 propagate priority to fire-engine 2024-08-15 19:04:46 +02:00
Gergő Móricz
b79d3d1754 fix 2024-08-15 19:02:05 +02:00
Gergő Móricz
57730f6a35 priority changes 2024-08-15 18:58:07 +02:00
Gergő Móricz
846610681b fix: fix posthog, add dummy crawl DB items 2024-08-15 18:55:18 +02:00
Nicolas
6e1074cdd1 Update website_params.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-14 17:39:54 -04:00
Thomas Kosmas
6410e1a81d Update params 2024-08-15 00:10:14 +03:00
Gergő Móricz
8a5cad72f6 fix(queue-worker): variable name collision 2024-08-14 22:02:05 +02:00
Gergő Móricz
b8ec40dd72 fix(crawl): submit sitemapped jobs in bulk 2024-08-14 20:34:19 +02:00
Gergő Móricz
2ca1017fc3 fix(crawl): make request 0 of crawl jobs higher priority 2024-08-14 19:34:18 +02:00
Gergő Móricz
f4466f6bb0 fix(test-suite): add artillery 2024-08-14 19:33:09 +02:00
Gergő Móricz
cfad067a63 fix(fly): change proxy limits 2024-08-14 18:52:40 +02:00
Gergő Móricz
a6c81f9d62 fix: return all data when calling webhook 2024-08-14 17:53:47 +02:00
rafaelsideguide
f86d2bb291 added go-sdk as submodule 2024-08-13 18:17:35 -03:00
Nicolas
e2472b9b0d Merge remote-tracking branch 'origin/v1/mockup-controllers' into v1-webscraper 2024-08-13 16:34:57 -04:00
Gergo Moricz
2e5e480cc2 fix(crawl): call webhooks 2024-08-13 22:10:17 +02:00
Gergo Moricz
a33596de3c fix(log_job): add crawl_id 2024-08-13 22:03:46 +02:00
Gergo Moricz
9252940b52 fix(crawl-status): sort data 2024-08-13 21:55:13 +02:00
Gergo Moricz
8dbac0268c feat: offload crawl results to the DB 2024-08-13 21:40:59 +02:00
Gergo Moricz
4bbc9db1df fix: prioritize scrape jobs over crawl jobs 2024-08-13 21:31:34 +02:00
Gergo Moricz
5f2af37880 fix(scrape): remove scrape job from queue after the job is done 2024-08-13 21:26:41 +02:00
Gergo Moricz
2413e33359 fix(queue-worker): remove console.log 2024-08-13 21:07:36 +02:00
Gergo Moricz
d7549d4dc5 feat: remove webScraperQueue 2024-08-13 21:03:24 +02:00
Gergő Móricz
4a2c37dcf5
Merge branch 'main' into feat/queue-scrapes 2024-08-13 20:53:49 +02:00
Gergo Moricz
86e136beca feat: crawl to scrape conversion 2024-08-13 20:51:43 +02:00
rafaelsideguide
a4be95ac27 fixed tests 2024-08-13 13:42:26 -03:00
KentHsu
fd060c7ef1 fix: go-sdk module name 2024-08-13 10:10:45 +08:00
Nicolas
09ca165d2e
Merge pull request #531 from kevinswiber/fix/respect-docker-env-file-comments 
Self-host fix: Moving comments of .env.example values from end-of-line to above-line.
 2024-08-12 16:54:56 -04:00
Nicolas
d06f40810c
Merge pull request #515 from wahpiangle/main 
Update redis urls in example .env
 2024-08-12 16:51:59 -04:00
Nicolas
6810338271 Update search.ts 2024-08-12 16:51:43 -04:00
Thomas Kosmas
98be29c963 Update parameters for platform.openai.com
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-12 22:49:28 +03:00
Kevin Swiber
33aa5cf0de
Moving comments of .env.example values from end-of-line to above-line. Self-host docs suggest using .env.example as a base. However, Docker doesn't respect end-of-line comments. It sets the comment as the actual value of the variable. This fix prevents that. 2024-08-12 12:24:46 -07:00
Nicolas
74a5125185 Nick: removed redlock 2024-08-12 15:07:30 -04:00
Nicolas
0bd1a820ee Update auth.ts 2024-08-12 13:42:09 -04:00
Nicolas
25a899eae3 Nick: redlock cache in auth 2024-08-12 13:37:47 -04:00
Rafael Miller
36e4b2cf49
Update .env.example 2024-08-12 10:37:00 -03:00
Quan Ming
a96ad4b0e2 Update redis url to use comment 2024-08-10 12:33:26 +08:00
Nicolas
e28c415cf4 Nick: 2024-08-09 14:07:46 -04:00
Gergo Moricz
5a778f2c22 fix(js-sdk): add type metadata to exports
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Details
Fly Deploy / Test Suite (push) Has been cancelled
Details
Fly Deploy / Python SDK Tests (push) Has been cancelled
Details
Fly Deploy / JavaScript SDK Tests (push) Has been cancelled
Details
Fly Deploy / Go SDK Tests (push) Has been cancelled
Details
Fly Deploy / Deploy app (push) Has been cancelled
Details
Fly Deploy / Build and publish Python SDK (push) Has been cancelled
Details
Fly Deploy / Build and publish JavaScript SDK (push) Has been cancelled
Details
2024-08-09 20:05:36 +02:00
Rafael Miller
6a78f6fe78
Merge pull request #497 from KentHsu/feat/add-go-sdk 
[Feat] Add Go SDK implementation
 2024-08-09 14:58:20 -03:00
rafaelsideguide
0591000b64 bugfix includes excludes 2024-08-09 14:30:41 -03:00
Kent (Chia-Hao), Hsu
1fda882983
Merge branch 'mendableai:main' into feat/add-go-sdk 2024-08-10 00:46:15 +08:00
Quan Ming
0221872a70 Update redis urls in example .env 2024-08-10 00:40:11 +08:00
rafaelsideguide
b802ea02a1 small improvements 
- wait for getting results on crawl: sometimes crawl takes some a second to save the data on the db and this causes response.data to be empty
- added timeout value to test script
- increased http client timeout (llm extract was failing on e2e tests)
- fixed env path on test script
 2024-08-09 11:13:14 -03:00
Nicolas
f1f5605010 Update website_params.ts 2024-08-08 12:31:58 -04:00
Nicolas
b0abad07da
Merge pull request #496 from tak-s/improve-logging-level
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details 
Improve logs
 2024-08-07 22:01:12 -04:00
Gergo Moricz
920b7f2f44 fix(runWebScraper): don't filter empty docs 2024-08-07 21:00:22 +02:00
Gergo Moricz
55ec96c23f fix(queue-worker): bad job lock extension time 2024-08-07 20:24:16 +02:00
Gergo Moricz
ab7a35c581 fix(queue-worker): log lock extensions 2024-08-07 19:49:48 +02:00
Gergo Moricz
a1c2ee5aa9 fix: always complete job, no try 2024-08-07 19:39:09 +02:00
Gergo Moricz
191dfbd9ca fix: move to completed in one place 2024-08-07 18:49:58 +02:00
Nicolas
457c082ba1 Nick: fixed tests 2024-08-07 11:08:53 -04:00
Nicolas
8a992b1596 Merge branch 'main' of https://github.com/mendableai/firecrawl
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-07 10:40:06 -04:00
Nicolas
b12e1157cc Nick: v35 bump 2024-08-07 10:40:00 -04:00
Gergő Móricz
5fc7fcb77c
Merge branch 'main' into feat/queue-scrapes 2024-08-07 16:35:44 +02:00
Gergo Moricz
fe9fdb578b revert bad hotfixes 2024-08-07 16:34:25 +02:00
Gergo Moricz
b7c01dcb9b fix(webScraperQueue): reduce retries to 2 2024-08-07 16:31:50 +02:00
Gergo Moricz
cdf7bad5b4 fix(runWebScraper): don't move to completed 2024-08-07 15:20:56 +02:00
Gergo Moricz
9df8719efa fix(queue-worker): raise queue log level to info 2024-08-07 14:56:04 +02:00
Gergo Moricz
7bb922071c fix(queue-worker): manually renew lock (testing) 2024-08-07 14:35:20 +02:00
Gergo Moricz
8216266d16 fix(scrape_log): display error properly 2024-08-07 14:19:20 +02:00
Gergo Moricz
2e2e80d679 fix(scrape-events): updateScrapeResult fix 2024-08-07 14:17:50 +02:00
Gergo Moricz
b5ec47fd96 fix(runWebScraper): don't fetch next job 2024-08-07 13:53:04 +02:00
Gergo Moricz
020a5efdb7 Revert "Revert "Merge pull request #432 from mendableai/mog/js-sdk-cjs"" 
This reverts commit 5da4472842.
 2024-08-07 01:27:26 +02:00
Gergő Móricz
7380d7799f
Merge branch 'main' into mog/js-sdk-cjs 2024-08-07 01:12:36 +02:00
Gergo Moricz
5f7724205f fix(js-sdk): re-add types 2024-08-07 01:06:21 +02:00
Nicolas
f294d3922c Nick: revert 2024-08-06 18:44:45 -04:00
Nicolas
5da4472842 Revert "Merge pull request #432 from mendableai/mog/js-sdk-cjs" 
This reverts commit bb90e03dea, reversing
changes made to 3321ca9398.
 2024-08-06 18:41:06 -04:00
Nicolas
a67a5c04c9 Revert "Merge pull request #432 from mendableai/mog/js-sdk-cjs" 
This reverts commit bb90e03dea, reversing
changes made to 3321ca9398.
 2024-08-06 18:02:56 -04:00
Nicolas
bb90e03dea
Merge pull request #432 from mendableai/mog/js-sdk-cjs 
fix(js-sdk): build both CommonJS and ESM versions
 2024-08-06 17:38:57 -04:00
rafaelsideguide
6cdf4c68ec wip: map, crawl, scrape mockups 2024-08-06 15:24:45 -03:00
Nicolas
3321ca9398
Merge pull request #504 from mendableai/feat/fullpage-screenshot 
[Feat] Added fullpagescreenshot capabilities
 2024-08-06 13:52:29 -04:00
Gergo Moricz
b60ee30dba fix(single_url): accept 500 2024-08-06 18:00:56 +02:00
Gergo Moricz
06751a8e21 fix(crawl-status): missing partial data after cancel 2024-08-06 17:31:20 +02:00
Gergo Moricz
810b98ec38 fix(scrape): fix timeout error code 2024-08-06 17:30:01 +02:00
Gergo Moricz
3ae95a2740 fix(scrape): consider timeout property 2024-08-06 17:25:58 +02:00
Gergo Moricz
8566ece700 fix(scrape): pass extractorOptions 2024-08-06 17:15:19 +02:00
Gergo Moricz
8e0aa69603 fix(crawl-status): partial_data 2024-08-06 17:06:21 +02:00
Gergo Moricz
1ab119c874 fix(scrape): don't double-bill for scrape 2024-08-06 16:57:23 +02:00
Gergo Moricz
7c5cda7b45 fix(queue-worker): concurrency 2024-08-06 16:57:00 +02:00
Gergo Moricz
d7d63790e5 fix(crawl-status): isCancelled should be status failed 2024-08-06 16:35:55 +02:00
Gergo Moricz
03c84a9372 cleanup and fix cancelling 2024-08-06 16:26:46 +02:00
rafaelsideguide
4d24a99d50 fix params 2024-08-06 09:34:43 -03:00
rafaelsideguide
3edc3a3d15 added fullpagescreenshot capabilities, wip on fire-engine side 2024-08-05 18:17:37 -03:00