Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 03:32:22 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
2,008 Commits 273 Branches 1 Tag 48 MiB
3afaab13d9
Commit Graph

2008 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gergo Moricz
b4dbf75537 fix(v1): check if url is string in blocklistMiddleware 
Fixes FIRECRAWL-SCRAPER-JS-9Z
 2024-09-10 10:25:14 +02:00
Gergo Moricz
26f2095de6 fix(v1): proper Invalid URL handling 2024-09-10 09:24:23 +02:00
Gergo Moricz
f8fbc71f91 fix(supabase-jobs): do not use RPCs 
RPCs are more failure-prone for this use case than regular queries are.
 2024-09-10 09:20:18 +02:00
Gergo Moricz
a6bcf7b438 fix(v0/crawl-status): don't crash on big crawls when requesting jobs from supabase 2024-09-10 08:53:40 +02:00
Nicolas
2cbc4c59ce
Merge pull request #649 from mendableai/nsc/chrome-cdp-for-params 
Better engine for wait + other params
 2024-09-09 21:12:46 -03:00
Nicolas
17e419a7fb Nick: 2024-09-09 21:06:23 -03:00
Nicolas
31e973d915
Merge pull request #648 from mendableai/nsc/fix-json-error-handler 
Fixed 500 errors when JSON is badly formatted
 2024-09-09 12:28:18 -03:00
Nicolas
ca9a781eb7 Update index.ts 2024-09-09 12:27:55 -03:00
Nicolas
22a5e85899 Update index.ts 2024-09-09 12:26:55 -03:00
y5n
4278fae51e
Update README.md 2024-09-09 10:55:31 +08:00
Nicolas
60a15d00eb Update types.ts 2024-09-08 16:39:12 -03:00
Nicolas
84fb00ec3f
Merge pull request #640 from mendableai/nsc/improved-billing-notifications 
Improved team credits check and billing notifications
 2024-09-08 13:07:45 -03:00
Nicolas
fbdfa1256b Update credit_billing.ts 2024-09-08 13:07:10 -03:00
Nicolas
48c665519e Update credit_billing.ts 2024-09-07 13:42:45 -03:00
Nicolas
32097fa29d Merge branch 'main' of https://github.com/mendableai/firecrawl
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Details
Clean Every 30 Minutes Before 24h Completed Jobs / clean-jobs (push) Has been cancelled
Details
Fly Deploy Direct / Deploy app (push) Has been cancelled
Details
Check Queues / clean-jobs (push) Has been cancelled
Details
Fly Deploy / Test Suite (push) Has been cancelled
Details
Fly Deploy / Python SDK Tests (push) Has been cancelled
Details
Fly Deploy / JavaScript SDK Tests (push) Has been cancelled
Details
Fly Deploy / Go SDK Tests (push) Has been cancelled
Details
Fly Deploy / Rust SDK Tests (push) Has been cancelled
Details
Fly Deploy / Deploy app (push) Has been cancelled
Details
Fly Deploy / Build and publish Python SDK (push) Has been cancelled
Details
Fly Deploy / Build and publish JavaScript SDK (push) Has been cancelled
Details
Fly Deploy / Build and publish Rust SDK (push) Has been cancelled
Details
2024-09-07 13:12:52 -03:00
Nicolas
5758af3291 Update website_params.ts 2024-09-07 13:12:46 -03:00
Nicolas
b3e2ca9a11
Merge pull request #638 from mendableai/nsc/removal-of-tags-in-include-main-content 
Removal of generic classnames/ids from onlyMainContent cleaning
 2024-09-07 13:06:43 -03:00
y5n
1ea9131e63 feat: Update redis deployment to run redis with password if REDIS_PASSWORD is configured 2024-09-07 16:00:32 +08:00
y5n
7685853d8a [Fix] fix SELF_HOST.md kubernetes cluster-install link 2024-09-07 13:50:47 +08:00
Nicolas
79870e7305 Update excludeTags.ts 2024-09-06 20:15:26 -03:00
Nicolas
a0f9ab2be7 Update map.ts 2024-09-06 20:14:47 -03:00
Nicolas
0731b3127d
Merge pull request #635 from shige/doc/update-self-host-v1.0.0
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
Details 
[Docs] upgraded the path of the self-hosted documentation URL to `/v1`.
 2024-09-06 18:24:44 -03:00
Eric Ciarla
e89e6d89a8
Merge pull request #637 from mendableai/ETL-unicorn-example-app 
Add Docs to API Spec Example
 2024-09-06 15:28:41 -04:00
Eric Ciarla
2d245a35f2 Delete combined_api_spec.json 2024-09-06 15:27:58 -04:00
Eric Ciarla
2044e71fcf Docs to API Spec 2024-09-06 15:26:33 -04:00
Tadashi Shigeoka
aa2cf686f4 [Docs] upgraded the path of the self-hosted documentation URL to /v1. 2024-09-06 21:41:31 +09:00
Nicolas
f5b84e15e1 Update sitemap.ts 2024-09-05 17:52:27 -03:00
Nicolas
554a05068c
Merge pull request #629 from mendableai/go-parser-singleton
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Details
Fly Deploy / Test Suite (push) Has been cancelled
Details
Fly Deploy / Python SDK Tests (push) Has been cancelled
Details
Fly Deploy / JavaScript SDK Tests (push) Has been cancelled
Details
Fly Deploy / Go SDK Tests (push) Has been cancelled
Details
Fly Deploy / Rust SDK Tests (push) Has been cancelled
Details
Fly Deploy / Deploy app (push) Has been cancelled
Details
Fly Deploy / Build and publish Python SDK (push) Has been cancelled
Details
Fly Deploy / Build and publish JavaScript SDK (push) Has been cancelled
Details
Fly Deploy / Build and publish Rust SDK (push) Has been cancelled
Details 
Feat: parser singleton
 2024-09-05 16:57:13 -03:00
Nicolas
4fa917f2b3
Update README.md 2024-09-05 16:45:23 -03:00
rafaelsideguide
82d6bf4ec8 feat(go-parser): singleton 2024-09-05 16:14:21 -03:00
Nicolas
b3f21d437b
Update README.md 2024-09-05 15:30:10 -03:00
Nicolas
cb630bfc34 Update fireEngine.ts 2024-09-05 14:24:10 -03:00
rafaelsideguide
8c1097e9e1 fix: pageOptions 2024-09-05 14:16:31 -03:00
rafaelsideguide
b301ffc922 added missing variables 2024-09-05 13:57:26 -03:00
Nicolas
a37ec6d17c Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-05 13:03:49 -03:00
Nicolas
c6f1d80992 Update crawl.ts 2024-09-05 13:03:43 -03:00
rafaelsideguide
7561bfe173 added envs to github action workflows 2024-09-05 12:59:32 -03:00
Nicolas
eb03a81152 Update crawl-status.ts 2024-09-05 12:55:04 -03:00
Nicolas
f7461dac01
Merge pull request #608 from mendableai/feat/go-html-to-md-parser 
Feat: Added go html to md parser
 2024-09-05 12:32:03 -03:00
Nicolas
9ee7b19ca1 Merge branch 'feat/go-html-to-md-parser' of https://github.com/mendableai/firecrawl into feat/go-html-to-md-parser 2024-09-05 12:13:20 -03:00
Nicolas
34adf43200 Merge branch 'main' into feat/go-html-to-md-parser 2024-09-05 12:13:14 -03:00
Nicolas
82cb80c817 Update map.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
Details
2024-09-04 23:46:18 -03:00
rafaelsideguide
de9b9ef901 Merge remote-tracking branch 'origin/main' into feat/go-html-to-md-parser 2024-09-04 17:06:46 -03:00
Nicolas
a0113dac37 Update credit_billing.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
Details
2024-09-04 16:54:20 -03:00
Nicolas
b8e9c445f9
Merge pull request #624 from mendableai/nsc/check-credits-optimization 
Optimize check credits func w/ caching
 2024-09-04 16:51:44 -03:00
Nicolas
28c5635502 Update ci.yml 2024-09-04 16:45:56 -03:00
Nicolas
b7b99b53da Merge branch 'main' into nsc/check-credits-optimization 2024-09-04 16:39:18 -03:00
rafaelsideguide
85b824e122 test: what about false false? 2024-09-04 16:35:32 -03:00
rafaelsideguide
78edf13ec6 test: usedbauth envs wth 2024-09-04 16:31:42 -03:00
rafaelsideguide
cb8571abad fix: enforced dotenv config 2024-09-04 15:57:57 -03:00