Commit Graph

1168 Commits

Author SHA1 Message Date
Nicolas
e1171ade1b Merge branch 'main' into mog/fix-status-job-get 2024-09-12 11:35:36 -04:00
Nicolas
4cd1065ae2 Update rate-limiter.ts 2024-09-11 14:03:34 -04:00
Gergő Móricz
97ffabff3a fix(v1): converting bad docs always gives null 2024-09-10 21:21:20 +02:00
Gergő Móricz
83a165db0f fix(v0/scrape): ensure url is string 2024-09-10 21:18:53 +02:00
Gergő Móricz
ad1a6fbc74 fix(v1/map): handle invalid URLs gracefully 2024-09-10 19:41:01 +02:00
Gergő Móricz
4ebc35c9dd fix(crawl-status): add success: true 2024-09-10 18:59:09 +02:00
Nicolas
a7406031fa Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-10 06:53:30 -03:00
Nicolas
a17e1cac92 Rate bump 2024-09-10 06:53:24 -03:00
Gergo Moricz
b4dbf75537 fix(v1): check if url is string in blocklistMiddleware
Fixes FIRECRAWL-SCRAPER-JS-9Z
2024-09-10 10:25:14 +02:00
Gergo Moricz
26f2095de6 fix(v1): proper Invalid URL handling 2024-09-10 09:24:23 +02:00
Gergo Moricz
f8fbc71f91 fix(supabase-jobs): do not use RPCs
RPCs are more failure-prone for this use case than regular queries are.
2024-09-10 09:20:18 +02:00
Gergo Moricz
a6bcf7b438 fix(v0/crawl-status): don't crash on big crawls when requesting jobs from supabase 2024-09-10 08:53:40 +02:00
Nicolas
17e419a7fb Nick: 2024-09-09 21:06:23 -03:00
Nicolas
ca9a781eb7 Update index.ts 2024-09-09 12:27:55 -03:00
Nicolas
22a5e85899 Update index.ts 2024-09-09 12:26:55 -03:00
Nicolas
60a15d00eb Update types.ts 2024-09-08 16:39:12 -03:00
Nicolas
fbdfa1256b Update credit_billing.ts 2024-09-08 13:07:10 -03:00
Nicolas
48c665519e Update credit_billing.ts 2024-09-07 13:42:45 -03:00
Nicolas
32097fa29d Merge branch 'main' of https://github.com/mendableai/firecrawl
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Clean Every 30 Minutes Before 24h Completed Jobs / clean-jobs (push) Has been cancelled
Fly Deploy Direct / Deploy app (push) Has been cancelled
Check Queues / clean-jobs (push) Has been cancelled
Fly Deploy / Test Suite (push) Has been cancelled
Fly Deploy / Python SDK Tests (push) Has been cancelled
Fly Deploy / JavaScript SDK Tests (push) Has been cancelled
Fly Deploy / Go SDK Tests (push) Has been cancelled
Fly Deploy / Rust SDK Tests (push) Has been cancelled
Fly Deploy / Deploy app (push) Has been cancelled
Fly Deploy / Build and publish Python SDK (push) Has been cancelled
Fly Deploy / Build and publish JavaScript SDK (push) Has been cancelled
Fly Deploy / Build and publish Rust SDK (push) Has been cancelled
2024-09-07 13:12:52 -03:00
Nicolas
5758af3291 Update website_params.ts 2024-09-07 13:12:46 -03:00
Nicolas
79870e7305 Update excludeTags.ts 2024-09-06 20:15:26 -03:00
Nicolas
a0f9ab2be7 Update map.ts 2024-09-06 20:14:47 -03:00
Nicolas
f5b84e15e1 Update sitemap.ts 2024-09-05 17:52:27 -03:00
rafaelsideguide
82d6bf4ec8 feat(go-parser): singleton 2024-09-05 16:14:21 -03:00
Nicolas
cb630bfc34 Update fireEngine.ts 2024-09-05 14:24:10 -03:00
rafaelsideguide
8c1097e9e1 fix: pageOptions 2024-09-05 14:16:31 -03:00
rafaelsideguide
b301ffc922 added missing variables 2024-09-05 13:57:26 -03:00
Nicolas
c6f1d80992 Update crawl.ts 2024-09-05 13:03:43 -03:00
Nicolas
eb03a81152 Update crawl-status.ts 2024-09-05 12:55:04 -03:00
Nicolas
34adf43200 Merge branch 'main' into feat/go-html-to-md-parser 2024-09-05 12:13:14 -03:00
Nicolas
82cb80c817 Update map.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
2024-09-04 23:46:18 -03:00
Nicolas
a0113dac37 Update credit_billing.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
2024-09-04 16:54:20 -03:00
Nicolas
b7b99b53da Merge branch 'main' into nsc/check-credits-optimization 2024-09-04 16:39:18 -03:00
rafaelsideguide
85b824e122 test: what about false false? 2024-09-04 16:35:32 -03:00
rafaelsideguide
78edf13ec6 test: usedbauth envs wth 2024-09-04 16:31:42 -03:00
rafaelsideguide
cb8571abad fix: enforced dotenv config 2024-09-04 15:57:57 -03:00
Nicolas
5ecb243693 Nick: 2024-09-04 15:19:45 -03:00
rafaelsideguide
3f462eabe9 fix(cicd): 2024-09-04 12:27:46 -03:00
rafaelsideguide
364ba9f901 fix(cicd): mendable->firecrawl and waitfor 2024-09-04 12:05:02 -03:00
rafaelsideguide
ad950a6c9d fixed controller res and tests 2024-09-04 11:29:32 -03:00
rafaelsideguide
74ac8915cd details 2024-09-04 09:41:38 -03:00
rafaelsideguide
d836ba6782 added log to check response on cicd 2024-09-04 09:35:56 -03:00
Nicolas
049a11187d Nick: 2024-09-03 21:09:32 -03:00
Nicolas
3072d4a333 Nick: fixed .sort coupons and sentry to withAuth 2024-09-03 21:02:41 -03:00
Nicolas
7561fd279f Nick: debug the billing email system for free credits 2024-09-03 17:08:12 -03:00
rafaelsideguide
c5e1d77a82 added invalid html tests 2024-09-03 15:21:45 -03:00
rafaelsideguide
d60fa6e084 fixed dockerfile and function name. it's working 2024-09-03 14:08:07 -03:00
rafaelsideguide
ebf4035484 added log so we can check 2024-09-03 13:15:21 -03:00
rafaelsideguide
291d9e375b now using compiled go/C lib with koffi 2024-09-03 10:56:07 -03:00
Nicolas
08a9cb8db4 Merge branch 'main' into pr/516 2024-09-02 23:32:23 -03:00
Nicolas
3244454826 Nick: 2024-09-02 19:36:36 -03:00
Nicolas
2444f7c450 Update scrape.ts 2024-09-02 18:51:18 -03:00
Nicolas
00dacaf935 Nick: scrape id when origin is website for report system 2024-09-02 18:43:52 -03:00
rafaelsideguide
48056ea1bd feat: added go html to md parser 2024-09-02 14:15:56 -03:00
rafaelsideguide
036eb925a2 fix(v1): maxDepth 2024-09-02 09:55:35 -03:00
Nicolas
86b0477708 Update map.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
2024-09-01 19:54:15 -03:00
Nicolas
304b3e752e Update map.ts 2024-09-01 19:52:21 -03:00
Nicolas
cb2dfe29be Nick: 2024-09-01 15:06:36 -03:00
Gergő Móricz
980293652d fix(queue-worker): new getJobs, log on v0 2024-09-01 19:29:35 +02:00
Nicolas
44fe741c35 Update queue-worker.ts 2024-09-01 14:19:43 -03:00
Nicolas
758f729ae2 Update queue-worker.ts 2024-09-01 14:10:47 -03:00
Nicolas
5c05bb12a7 Update webhook.ts 2024-09-01 13:58:50 -03:00
Nicolas
ae90370543 Update webhook.ts 2024-09-01 13:58:07 -03:00
Nicolas
b68a50fea3 Nick: 2024-09-01 13:52:34 -03:00
Nicolas
979697df1c Update queue-worker.ts 2024-09-01 13:47:51 -03:00
Nicolas
95b9dc915d Nick: webhooks v1 working great 2024-09-01 13:44:36 -03:00
Nicolas
faae98ecb8 Merge branch 'main' into v1/webhooks 2024-09-01 12:56:11 -03:00
Nicolas
fe6abe8f7d Nick: 2024-09-01 12:55:02 -03:00
Nicolas
2ef43d5fc2 Update rate-limiter.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
2024-08-31 14:26:16 -03:00
Nicolas
8431be5826 Nick: 2024-08-31 14:23:55 -03:00
Nicolas
af5cc5f16b Nick: 1.2 - v1 llm extract 2024-08-31 13:25:48 -03:00
Nicolas
0c59564393 Merge branch 'main' into v1/llm-extract 2024-08-30 21:31:49 -03:00
Gergő Móricz
693dc14d9b remove invalid keys
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
2024-08-31 00:08:24 +02:00
Nicolas
6d1da2e22a Update index.ts 2024-08-30 18:45:02 -03:00
Nicolas
d347160ff9 Nick: 2024-08-30 17:32:41 -03:00
Nicolas
522d256b11 Merge branch 'main' into v1/llm-extract 2024-08-30 17:16:21 -03:00
Nicolas
ef5c8931eb Update scrape.ts 2024-08-30 17:09:44 -03:00
Nicolas
050cac51ec Update types.ts 2024-08-30 17:02:29 -03:00
Nicolas
012bc74e14 Revert "Update types.ts"
This reverts commit 77766cf00d.
2024-08-30 17:02:05 -03:00
Nicolas
77766cf00d Update types.ts 2024-08-30 17:01:54 -03:00
rafaelsideguide
87e61f2d51 v0 working 2024-08-30 16:38:55 -03:00
rafaelsideguide
dc189e1e9d feat: webhooks config on v1 2024-08-30 16:22:59 -03:00
Nicolas
0732997a13 Merge branch 'main' into v1/llm-extract 2024-08-30 15:42:45 -03:00
Gergo Moricz
c8e0bb93ea fix(crawl-status): handle null data when purging rawHtml 2024-08-30 20:40:57 +02:00
Gergo Moricz
6a6b487474 fix(v1): don't fail on doc = null 2024-08-30 20:28:18 +02:00
Nicolas
71dab56e36 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-08-30 15:21:29 -03:00
Nicolas
d7dbc2536d Update crawl.ts 2024-08-30 15:21:22 -03:00
Nicolas
9de39cfa2c
Merge pull request #591 from mendableai/mog/websocket-sdk
[v1] Websockets SDKs
2024-08-30 15:06:06 -03:00
Nicolas
234c6daee8 Update supabase-jobs.ts 2024-08-30 14:52:59 -03:00
Nicolas
7c8ca0c889 Update types.ts 2024-08-30 13:54:04 -03:00
Nicolas
f4a49bc1d1 Update types.ts 2024-08-30 13:49:16 -03:00
Nicolas
bb4808443c Merge branch 'main' into v1/llm-extract 2024-08-30 12:48:26 -03:00
Nicolas
2fd9c285bd
Merge pull request #582 from mendableai/fix--rawHtml-return-on-crawl
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Fly Deploy / Build and publish Rust SDK (push) Waiting to run
[Bug] Moved delete rawHtml to end of controller
2024-08-30 12:40:06 -03:00
Nicolas
b8920d6f4a Nick: 2024-08-30 12:37:45 -03:00
Nicolas
552328d168 Merge branch 'main' into v1/llm-extract 2024-08-30 12:34:50 -03:00
Nicolas
ee3e5dc69c Nick: 2024-08-30 12:34:45 -03:00
Gergő Móricz
26d87bd099
fix(api): handle zoderrors earlier 2024-08-30 17:33:42 +02:00
Nicolas
1805d901a9 Update credit_billing.ts 2024-08-30 12:05:37 -03:00
Nicolas
e5ca4364ba Nick: improvements to llm extract error handling 2024-08-30 11:57:55 -03:00
Nicolas
52ac132328
Update auth.ts 2024-08-30 11:10:48 -03:00
Nicolas
41eb620959 Nick: prompt option, still need to convert to new structured outputs 2024-08-29 21:00:57 -03:00
Nicolas
49e1cb7ca0 Nick: 2024-08-29 20:08:06 -03:00
Gergő Móricz
d4001e4528 fix(api/websocket): fix auth and termination 2024-08-29 20:01:00 +02:00
Nicolas
1174bc3c82 Nick: blocked sw/ra
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-29 13:55:54 -03:00
rafaelsideguide
7eefaffe48 fix: rawHtml for v0 2024-08-29 12:24:14 -03:00
rafaelsideguide
f8323f9b11 fix: moved delete rawHtml to end of controller 2024-08-29 10:02:05 -03:00
Nicolas
02410aca36 Update runWebScraper.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-28 22:39:50 -03:00
Nicolas
67229c6b3a Nick: fixed credits issue
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-28 18:32:45 -03:00
Nicolas
70bff7f8fb Nick: 2024-08-28 15:59:20 -03:00
Nicolas
8be75accb8 Nick: 2024-08-28 15:40:30 -03:00
Nicolas
547da88ccd Update rate-limiter.test.ts 2024-08-28 14:53:02 -03:00
Nicolas
4edd9a34c7 Nick: totalCount -> total, completedCount -> completed 2024-08-28 14:46:53 -03:00
Nicolas
2a56f89f50 Update index.test.ts 2024-08-28 14:24:26 -03:00
Nicolas
59547d7129 Update index.test.ts 2024-08-28 14:22:26 -03:00
rafaelsideguide
b7f62f93c4 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-28 14:18:07 -03:00
rafaelsideguide
1c35534e2a fix(v1): check-status for preview 2024-08-28 14:18:05 -03:00
Nicolas
af67b14c9d Nick: 2024-08-28 14:17:59 -03:00
Nicolas
8fbff28009 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-28 14:09:20 -03:00
Nicolas
08387c06f8 Update auth.ts 2024-08-28 14:09:12 -03:00
rafaelsideguide
ef2d8d012b Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-28 14:07:31 -03:00
rafaelsideguide
5cbf0dcaf5 fix(v1): includeTags 2024-08-28 14:07:28 -03:00
Nicolas
7ba3f3bd1a Nick: 2024-08-28 13:54:39 -03:00
Nicolas
9a13034095 Nick: 2024-08-28 13:46:39 -03:00
Nicolas
ecd07be49e Nick: fixed issues 2024-08-28 13:17:22 -03:00
Nicolas
4003d37fbc Nick: 2024-08-28 12:46:59 -03:00
Nicolas
d872bf0c4c Merge branch 'main' into v1-webscraper 2024-08-28 12:42:23 -03:00
Nicolas
7dff5cdf49 Nick: fixed sdk types and map preview 2024-08-27 20:02:39 -03:00
Nicolas
e7d283c4ef Update job-priority.ts 2024-08-27 17:04:04 -03:00
Nicolas
ca34f1203b Nick: bucket limit increase 2024-08-27 17:03:46 -03:00
Nicolas
f0dfcd6a49 Update job-priority.ts 2024-08-27 16:58:28 -03:00
Nicolas
080240e040 Merge branch 'main' into nsc/job-priority 2024-08-27 16:18:02 -03:00
Nicolas
88239d7211 Merge branch 'main' into v1-webscraper 2024-08-27 16:16:40 -03:00
Nicolas
8c8d0602bb Update rate-limiter.test.ts 2024-08-27 15:38:01 -03:00
Nicolas
bd84e49a77 Update rate-limiter.ts 2024-08-27 15:37:36 -03:00
Nicolas
a0f9a81ee6 Nick: 2024-08-27 15:36:46 -03:00
Nicolas
c009013ff6 Nick: expire tests 2024-08-27 15:26:43 -03:00
Nicolas
8b53285a93 Update job-priority.ts 2024-08-27 15:06:13 -03:00
Nicolas
06b70a47e0 Update job-priority.ts 2024-08-27 15:04:10 -03:00
Nicolas
1e08e6d317 Merge branch 'main' into nsc/job-priority 2024-08-27 15:01:58 -03:00
Nicolas
861e2ebdf1 Nick: 2x rate limits
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-08-27 12:08:50 -03:00
Nicolas
170a8ebfe5 Nick: 2024-08-27 11:58:42 -03:00
Nicolas
3850b000f0 Nick: removing credit notification for now 2024-08-27 11:14:04 -03:00
Nicolas
d30119707f Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-27 11:12:00 -03:00
Nicolas
463d38f77e Update map.ts 2024-08-27 11:11:59 -03:00
rafaelsideguide
1ef41b92a0 feat: cancel
v0 implementation + e2e test
2024-08-27 09:42:55 -03:00
Nicolas
fb553a020d Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-26 19:57:28 -03:00
Nicolas
6ab6ef9004 Update auth.ts 2024-08-26 19:57:27 -03:00
rafaelsideguide
adc3e4233d Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-26 19:22:05 -03:00
rafaelsideguide
65faa3e163 tests/feat: url validation 2024-08-26 19:22:03 -03:00
Nicolas
558acffb33 Nick: @rafaelsideguide isarray for includes/excludes 2024-08-26 19:07:14 -03:00