Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 19:58:08 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
1,635 Commits 273 Branches 1 Tag 48 MiB
d836ba6782
Commit Graph

1022 Commits

Author SHA1 Message Date
Nicolas
6ab6ef9004 Update auth.ts 2024-08-26 19:57:27 -03:00
rafaelsideguide
adc3e4233d Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-26 19:22:05 -03:00
rafaelsideguide
65faa3e163 tests/feat: url validation 2024-08-26 19:22:03 -03:00
Nicolas
558acffb33 Nick: @rafaelsideguide isarray for includes/excludes 2024-08-26 19:07:14 -03:00
Nicolas
7d93eab0f8 Nick: 2024-08-26 18:48:00 -03:00
rafaelsideguide
72454de18d Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-26 18:21:54 -03:00
rafaelsideguide
04556ded40 tests: e2e for crawl and crawl status 2024-08-26 18:21:52 -03:00
Nicolas
8c37ea6d96 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-26 18:17:17 -03:00
Nicolas
f277a0e2bb Update package.json 2024-08-26 18:17:09 -03:00
rafaelsideguide
f2f6f78dcf fix(url validation): sub paths 2024-08-26 18:12:03 -03:00
Nicolas
0bbb8bb24e Nick: 2024-08-26 17:17:12 -03:00
Nicolas
98a770f38f Nick: rm wip 2024-08-26 17:16:44 -03:00
Nicolas
6f68678b5d Nick: 2024-08-26 17:13:00 -03:00
Nicolas
b0bd71a3a9 Merge branch 'main' into v1-webscraper 2024-08-26 16:58:00 -03:00
Nicolas
2d78c20d68 Nick: 2024-08-26 16:56:27 -03:00
Nicolas
fa7dc5b10b Update rate-limiter.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-26 16:33:34 -03:00
Nicolas
4d0acc9722 Merge branch 'main' into v1-webscraper 2024-08-26 16:22:05 -03:00
Nicolas
5606fe5870 Nick: 2024-08-26 16:05:11 -03:00
rafaelsideguide
1baba3ce0a fix(go-sdk): submodules 2024-08-26 11:11:34 -03:00
Gergo Moricz
d591e0f51c block corterix.com for performance issues
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-25 20:06:12 +02:00
rafaelsideguide
6f9a2687ae fixed turndown bug 2024-08-25 15:04:32 -03:00
Gergo Moricz
96e91ab9ec convert webhook call to v1 2024-08-25 14:05:46 +02:00
Nicolas
1f99bfd3c8 Update queue.ts
Some checks failed
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
Simple Autoscaler / scale (push) Has been cancelled
Details
2024-08-23 22:47:12 -03:00
Nicolas
b80277d4de Update queue.ts 2024-08-23 22:46:44 -03:00
Nicolas
d87b62fed9 Nick: 2024-08-23 22:33:17 -03:00
Nicolas
b9e06e27f4 Update queue.ts 2024-08-23 22:17:27 -03:00
Nicolas
8e78511ed4 Update queue.ts 2024-08-23 22:15:47 -03:00
Nicolas
28d7a637c2 Update queue.ts 2024-08-23 22:07:49 -03:00
Nicolas
173f4ee1bf Nick: chrome cdp main | simple autoscaler 2024-08-23 20:09:59 -03:00
Gergő Móricz
064ebfc54d fix websocket 2024-08-23 19:55:41 +02:00
Gergő Móricz
05c250d3b8 Merge branch 'main' into v1-webscraper 2024-08-23 19:38:57 +02:00
Gergő Móricz
2ab0dd2e15 fix(scrape): add further llm extraction catch 2024-08-23 19:20:17 +02:00
Gergő Móricz
1054a1397b Merge branch 'main' into v1-webscraper 2024-08-23 19:14:49 +02:00
Nicolas
3d53f4e213 Nick: unblocking pin 2024-08-23 13:56:05 -03:00
Gergő Móricz
5ef3926d2a fix(scrape,search): handle failed jobs 2024-08-23 18:47:56 +02:00
Gergő Móricz
866e71910c further fixes 2024-08-23 18:27:00 +02:00
Gergő Móricz
eea530e0ad feat(v1): update for sentry 2024-08-23 17:29:42 +02:00
Gergő Móricz
e7f267b6fe Merge branch 'main' into v1-webscraper 2024-08-23 17:21:54 +02:00
Gergő Móricz
8d9ff90bcb feat(fire-engine): propagate sentry trace 2024-08-22 23:38:04 +02:00
rafaelsideguide
74ea820bc6 fix: url and check for metadata 2024-08-22 18:32:19 -03:00
Nicolas
1f0abacadf Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-08-22 18:30:54 -03:00
Nicolas
1f779e261a Update rate-limiter.ts 2024-08-22 18:30:45 -03:00
Gergő Móricz
8e3c2b2855 fix(crawler): verify URL 2024-08-22 23:30:19 +02:00
Gergő Móricz
e690a6fda7 fix: remove QueueEvents 2024-08-22 22:38:39 +02:00
Gergő Móricz
76c8e9f996 fix
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-22 22:24:24 +02:00
Gergő Móricz
ad82175fb8 fix(scrape): poll 2024-08-22 22:12:02 +02:00
rafaelsideguide
5f60a55967 workflow and npm now running v1 tests 2024-08-22 15:28:49 -03:00
rafaelsideguide
30e809966f Merge remote-tracking branch 'origin/v1/python-sdk' into v1-webscraper 2024-08-22 15:18:05 -03:00
rafaelsideguide
a37681bdff fix: replace jest, removed map for v0 2024-08-22 15:16:46 -03:00
rafaelsideguide
7473b74021 fix: html and rawlhtmls for pdfs 2024-08-22 15:15:45 -03:00
Gergő Móricz
dd737f1235 feat(sentry): add queue instrumentation to 2024-08-22 19:17:51 +02:00
Nicolas
d2521612b4 Update .gitignore 2024-08-22 14:15:19 -03:00
Gergő Móricz
7265ab7c67 fix(search): filter docs properly 2024-08-22 18:46:56 +02:00
rafaelsideguide
b1d61d8557 Merge remote-tracking branch 'origin/v1-webscraper' into v1/python-sdk 2024-08-22 13:39:09 -03:00
Gergő Móricz
d036738da0 fix(bullmq): duplicate redis connection for QueueEvents 2024-08-22 18:04:09 +02:00
Gergő Móricz
6d48dbcd38 feat(sentry): add trace continuity for queue 2024-08-22 16:47:38 +02:00
Gergő Móricz
6d92b8524d feat(scrape): record job result in span 2024-08-22 16:00:13 +02:00
Gergő Móricz
5ca36fe9fc feat(api): add more captureExceptions 2024-08-22 15:49:16 +02:00
Gergő Móricz
0e8fd6ce70 fix(scrape): ensure extractionSchema is an object if llm-extraction is specified 2024-08-22 14:50:51 +02:00
Gergő Móricz
4bd2ff26d3 fix(llm-extract): pass stacktrace properly 2024-08-22 14:37:09 +02:00
Gergő Móricz
e4adbaa88e fix(llm-extract): handle llm-extract if scrape failed
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-22 14:12:52 +02:00
Gergő Móricz
670d253a8c fix(auth): fix error reporting 2024-08-22 14:08:09 +02:00
Gergő Móricz
7d9f5bf8b1 fix(crawl): don't use sitemap if it's empty 
Fixes FIRECRAWL-SCRAPER-JS-11
 2024-08-22 13:41:33 +02:00
Gergő Móricz
1f580deefc fix(crawl): validate includes.excludes regexes 2024-08-22 13:29:11 +02:00
Gergő Móricz
fbbc3878f1 fix(crawler): make sure includes/excludes is an array 2024-08-22 13:18:26 +02:00
Gergő Móricz
508568f943 fix(search): handle scrape timeouts on search 
Fixes FIRECRAWL-SCRAPER-JS-15
 2024-08-22 13:10:58 +02:00
Gergő Móricz
14fa75cae6 fix(crawl): send error if url is not a string 
Fixes FIRECRAWL-SCRAPER-JS-1E and FIRECRAWL-SCRAPER-JS-Z
 2024-08-22 13:09:08 +02:00
Nicolas
8a778278a9 Merge branch 'main' into nsc/job-priority 2024-08-21 22:57:55 -03:00
Gergo Moricz
0cdf41587e feat(sentry): add error handles to try-catch blocks
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-22 03:55:40 +02:00
Nicolas
53ca704620 Update index.ts 2024-08-21 22:55:39 -03:00
Nicolas
477c3257dc Nick: 2024-08-21 22:53:33 -03:00
Nicolas
c7bfe4ffe8 Nick: 2024-08-21 22:20:40 -03:00
Nicolas
6bdb1d045d Merge branch 'main' into nsc/job-priority 2024-08-21 21:52:05 -03:00
Nicolas
e78d2af1f0 Nick: 2024-08-21 21:51:54 -03:00
Nicolas
e64d3815ea Merge branch 'main' into nsc/job-priority 2024-08-21 20:54:57 -03:00
Nicolas
0ea0a5db46 Nick: wip 2024-08-21 20:54:39 -03:00
rafaelsideguide
fe2e8c0b7a includehtml fix 2024-08-21 15:54:00 -03:00
Gergő Móricz
629da74a5c fix(sentry): decrease tracesSampleRate
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-21 20:51:35 +02:00
Gergő Móricz
55009e51f5 fix: filter out invalid URLs from crawl links 2024-08-21 20:49:25 +02:00
Gergő Móricz
dae1408e66 fix(Dockerfile): retain sentry auth token properly 2024-08-21 20:40:42 +02:00
Gergő Móricz
ac9783ed2f fix(sentry): adjust profiles sample rate to be even lower 2024-08-21 20:21:16 +02:00
Gergő Móricz
9579f03c4b fix: import resolution 2024-08-21 20:16:06 +02:00
Gergő Móricz
6104d74213 fix(sentry): drop profiling sample rate 2024-08-21 20:12:47 +02:00
Gergő Móricz
3d5dc9d90a feat(sentry): add log + server name 2024-08-21 19:39:10 +02:00
Gergő Móricz
85ff0c311e Add worker ID to job attribute 2024-08-21 19:21:29 +02:00
Gergő Móricz
920702cdde Update builder to handle uploading sourcemaps 2024-08-21 19:08:03 +02:00
Gergő Móricz
86942728e3 Add metadata for queue-worker and Express 2024-08-21 17:58:27 +02:00
Nicolas
35decb1af2 Nick: 2024-08-21 12:35:03 -03:00
rafaelsideguide
52abec41c2 fixing delete 2024-08-21 10:35:50 -03:00
Nicolas
db8c84ff0f Update requests.http 2024-08-21 10:19:37 -03:00
rafaelsideguide
b66553867e reverting delete, fixed express bug on checkCredits 2024-08-21 09:28:20 -03:00
rafaelsideguide
138437d616 commenting out delete, crashing on fire-engine
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-21 08:11:24 -03:00
rafaelsideguide
5e48bec1fd commenting out delete, crashing on fire-engine 2024-08-21 08:10:46 -03:00
Nicolas
90b32f16c8 Nick: fixes 2024-08-20 21:38:11 -03:00
Nicolas
819ad50af3 Update fireEngine.ts 2024-08-20 21:16:33 -03:00
rafaelsideguide
e9d6ca197e tests passing now 2024-08-20 20:00:41 -03:00
Nicolas
1b3ad60a2c Reapply "Merge pull request #561 from mendableai/bug/dealing-with-dns-error" 
This reverts commit ffe11a5bf7.
 2024-08-20 19:22:09 -03:00
Nicolas
441628998f Reapply "Merge pull request #561 from mendableai/bug/dealing-with-dns-error" 
This reverts commit ffe11a5bf7.
 2024-08-20 19:16:48 -03:00
Nicolas
ffe11a5bf7 Revert "Merge pull request #561 from mendableai/bug/dealing-with-dns-error"
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details 
This reverts commit 2030ec6031, reversing
changes made to f494d2b707.
 2024-08-20 18:16:11 -03:00
Gergő Móricz
5818236659 fix: remove rawHtml properly 2024-08-20 22:51:12 +02:00
Gergő Móricz
1368f9a87f fix: treat existing screenshot as a scraper success condition 2024-08-20 22:24:18 +02:00
rafaelsideguide
f98be7d94e Update fireEngine.ts 2024-08-20 16:53:01 -03:00
Nicolas
0c48c8a436 Nick: billing for map 2024-08-20 16:43:46 -03:00
Gergő Móricz
f494d2b707 prioritize search lower 2024-08-20 21:18:31 +02:00
Nicolas
59eb552df5
Merge pull request #560 from mendableai/feat/cancel-fire-engine 
[Feat] Added check job and cancel to fire-engine requests
 2024-08-20 15:43:31 -03:00
rafaelsideguide
1f27182a13 added try catch 2024-08-20 15:42:39 -03:00
Gergő Móricz
efb91f9ca5 fix(search): delete jobs after done 2024-08-20 20:29:08 +02:00
Nicolas
39388cdc35 Update crawl.ts 2024-08-20 14:41:43 -03:00
Nicolas
674adee144 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-20 14:41:05 -03:00
Nicolas
b36faeaf54 Nick: 2024-08-20 14:39:52 -03:00
Gergő Móricz
cf32893c2e add strict enforcement + move crawlerOptions to top-level in /crawl 2024-08-20 19:31:26 +02:00
rafaelsideguide
e326249a57 added check job and cancel to fire-engine requests 2024-08-20 14:26:42 -03:00
Gergő Móricz
70d50b3640 fix(queue-worker): move dotenv config up 2024-08-20 19:25:19 +02:00
Nicolas
c5ad4dedeb Update crawl.ts 2024-08-20 14:19:20 -03:00
Nicolas
de0dc20a02 Update credit_billing.ts 2024-08-20 14:18:14 -03:00
Nicolas
5abd26a267 Nick: set the crawl limit to the remaining credits 2024-08-20 14:16:54 -03:00
Nicolas
e200ec9e12 Nick: 2024-08-20 12:24:14 -03:00
Nicolas
55dad82df1 Nick: fixed map search 2024-08-20 12:17:53 -03:00
Nicolas
27903247b6 Nick: map tests and fixes 2024-08-20 12:04:08 -03:00
rafaelsideguide
e1c9cbf709 bug fixed. crawl should not stop if sitemap url is invalid
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-20 09:11:58 -03:00
rafaelsideguide
ecd472356b added variables to beta customers 2024-08-19 16:41:54 -03:00
Nicolas
3dc298be54 Nick: 2x rate limits for standard and growth for /scrape 2024-08-19 13:52:54 -03:00
rafaelsideguide
72461ce9a6 Update index.test.ts 2024-08-19 13:29:52 -03:00
rafaelsideguide
fd7fdc1d52 added blocklist middleware 2024-08-19 13:28:54 -03:00
Nicolas
8e4ca86463 Update crawl.ts 2024-08-19 11:02:24 -03:00
Nicolas
36b35dbc67 Update crawl.ts 2024-08-19 11:01:26 -03:00
rafaelsideguide
4ffc60596a Update queue-worker.ts
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-19 09:29:23 -03:00
rafaelsideguide
b8170aaa47 Update blocklist.ts 2024-08-19 08:51:48 -03:00
Nicolas
3fe82b4f12 Update queue-worker.ts
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Details
Fly Deploy / Test Suite (push) Has been cancelled
Details
Fly Deploy / Python SDK Tests (push) Has been cancelled
Details
Fly Deploy / JavaScript SDK Tests (push) Has been cancelled
Details
Fly Deploy / Go SDK Tests (push) Has been cancelled
Details
Fly Deploy / Deploy app (push) Has been cancelled
Details
Fly Deploy / Build and publish Python SDK (push) Has been cancelled
Details
Fly Deploy / Build and publish JavaScript SDK (push) Has been cancelled
Details
2024-08-17 03:09:31 -04:00
Nicolas
f797380112 Nick:
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-16 22:17:38 -04:00
Nicolas
47123be783 Nick: weird activity block 2024-08-16 22:01:56 -04:00
Nicolas
ff84f1fe5e Update map.ts 2024-08-16 20:42:36 -04:00
Nicolas
4314313477 Update map.ts 2024-08-16 19:56:18 -04:00
Nicolas
af9a0a6f0b Update map.ts 2024-08-16 19:56:03 -04:00
Nicolas
ba5279eafc Nick: all tests passing 2024-08-16 19:55:44 -04:00
Nicolas
5205c5f005 Update map.ts 2024-08-16 19:37:00 -04:00
Nicolas
0c05d096a9 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 19:33:58 -04:00
Nicolas
ab48353226 Nick: /map almost good 2024-08-16 19:33:57 -04:00
Gergő Móricz
eb84673b06 feat: crawl status websocket WIP 2024-08-17 01:04:14 +02:00
Gergő Móricz
e2a6ef26d3 mount v1Router under v1 path 2024-08-16 23:48:50 +02:00
Gergő Móricz
4c1b74dab3 fix(map): remove robots.txt 2024-08-16 23:46:10 +02:00
Gergő Móricz
c281fe62c0 fix(crawl): propagate db fix to preview endpoint 2024-08-16 23:43:54 +02:00
Gergő Móricz
803577eeba feat(crawl): webhook 2024-08-16 23:42:48 +02:00
Gergő Móricz
e6738abf96 fix(crawl-status): retrieve from DB in bulk 2024-08-16 23:39:39 +02:00
rafaelsideguide
086ba6280b fixed markdown format 2024-08-16 18:39:13 -03:00
Gergő Móricz
aabfaf0ac5 clean up crawl-status, fix db ddos 2024-08-16 23:29:39 +02:00
rafaelsideguide
e5b807ccc4 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 17:57:31 -03:00
rafaelsideguide
7a61325500 map + search + scrape markdown bug 2024-08-16 17:57:11 -03:00
Gergő Móricz
5896153d19 fix: crawl status and redis fixes 2024-08-16 22:52:48 +02:00
Gergő Móricz
3fcb21930e remove log 2024-08-16 22:48:23 +02:00