Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 11:42:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
1,415 Commits 273 Branches 1 Tag 48 MiB
e7f267b6fe
Commit Graph

873 Commits

Author SHA1 Message Date
Gergő Móricz
e7f267b6fe Merge branch 'main' into v1-webscraper 2024-08-23 17:21:54 +02:00
Gergő Móricz
8d9ff90bcb feat(fire-engine): propagate sentry trace 2024-08-22 23:38:04 +02:00
rafaelsideguide
74ea820bc6 fix: url and check for metadata 2024-08-22 18:32:19 -03:00
Nicolas
1f0abacadf Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-08-22 18:30:54 -03:00
Nicolas
1f779e261a Update rate-limiter.ts 2024-08-22 18:30:45 -03:00
Gergő Móricz
8e3c2b2855 fix(crawler): verify URL 2024-08-22 23:30:19 +02:00
Gergő Móricz
e690a6fda7 fix: remove QueueEvents 2024-08-22 22:38:39 +02:00
Gergő Móricz
76c8e9f996 fix
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-22 22:24:24 +02:00
Gergő Móricz
ad82175fb8 fix(scrape): poll 2024-08-22 22:12:02 +02:00
rafaelsideguide
5f60a55967 workflow and npm now running v1 tests 2024-08-22 15:28:49 -03:00
rafaelsideguide
30e809966f Merge remote-tracking branch 'origin/v1/python-sdk' into v1-webscraper 2024-08-22 15:18:05 -03:00
rafaelsideguide
a37681bdff fix: replace jest, removed map for v0 2024-08-22 15:16:46 -03:00
rafaelsideguide
7473b74021 fix: html and rawlhtmls for pdfs 2024-08-22 15:15:45 -03:00
Gergő Móricz
dd737f1235 feat(sentry): add queue instrumentation to 2024-08-22 19:17:51 +02:00
Nicolas
d2521612b4 Update .gitignore 2024-08-22 14:15:19 -03:00
Gergő Móricz
7265ab7c67 fix(search): filter docs properly 2024-08-22 18:46:56 +02:00
rafaelsideguide
b1d61d8557 Merge remote-tracking branch 'origin/v1-webscraper' into v1/python-sdk 2024-08-22 13:39:09 -03:00
Gergő Móricz
d036738da0 fix(bullmq): duplicate redis connection for QueueEvents 2024-08-22 18:04:09 +02:00
Gergő Móricz
6d48dbcd38 feat(sentry): add trace continuity for queue 2024-08-22 16:47:38 +02:00
Gergő Móricz
6d92b8524d feat(scrape): record job result in span 2024-08-22 16:00:13 +02:00
Gergő Móricz
5ca36fe9fc feat(api): add more captureExceptions 2024-08-22 15:49:16 +02:00
Gergő Móricz
0e8fd6ce70 fix(scrape): ensure extractionSchema is an object if llm-extraction is specified 2024-08-22 14:50:51 +02:00
Gergő Móricz
4bd2ff26d3 fix(llm-extract): pass stacktrace properly 2024-08-22 14:37:09 +02:00
Gergő Móricz
e4adbaa88e fix(llm-extract): handle llm-extract if scrape failed
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-22 14:12:52 +02:00
Gergő Móricz
670d253a8c fix(auth): fix error reporting 2024-08-22 14:08:09 +02:00
Gergő Móricz
7d9f5bf8b1 fix(crawl): don't use sitemap if it's empty 
Fixes FIRECRAWL-SCRAPER-JS-11
 2024-08-22 13:41:33 +02:00
Gergő Móricz
1f580deefc fix(crawl): validate includes.excludes regexes 2024-08-22 13:29:11 +02:00
Gergő Móricz
fbbc3878f1 fix(crawler): make sure includes/excludes is an array 2024-08-22 13:18:26 +02:00
Gergő Móricz
508568f943 fix(search): handle scrape timeouts on search 
Fixes FIRECRAWL-SCRAPER-JS-15
 2024-08-22 13:10:58 +02:00
Gergő Móricz
14fa75cae6 fix(crawl): send error if url is not a string 
Fixes FIRECRAWL-SCRAPER-JS-1E and FIRECRAWL-SCRAPER-JS-Z
 2024-08-22 13:09:08 +02:00
Gergo Moricz
0cdf41587e feat(sentry): add error handles to try-catch blocks
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-22 03:55:40 +02:00
Nicolas
e78d2af1f0 Nick: 2024-08-21 21:51:54 -03:00
rafaelsideguide
fe2e8c0b7a includehtml fix 2024-08-21 15:54:00 -03:00
Gergő Móricz
629da74a5c fix(sentry): decrease tracesSampleRate
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-21 20:51:35 +02:00
Gergő Móricz
55009e51f5 fix: filter out invalid URLs from crawl links 2024-08-21 20:49:25 +02:00
Gergő Móricz
dae1408e66 fix(Dockerfile): retain sentry auth token properly 2024-08-21 20:40:42 +02:00
Gergő Móricz
ac9783ed2f fix(sentry): adjust profiles sample rate to be even lower 2024-08-21 20:21:16 +02:00
Gergő Móricz
9579f03c4b fix: import resolution 2024-08-21 20:16:06 +02:00
Gergő Móricz
6104d74213 fix(sentry): drop profiling sample rate 2024-08-21 20:12:47 +02:00
Gergő Móricz
3d5dc9d90a feat(sentry): add log + server name 2024-08-21 19:39:10 +02:00
Gergő Móricz
85ff0c311e Add worker ID to job attribute 2024-08-21 19:21:29 +02:00
Gergő Móricz
920702cdde Update builder to handle uploading sourcemaps 2024-08-21 19:08:03 +02:00
Gergő Móricz
86942728e3 Add metadata for queue-worker and Express 2024-08-21 17:58:27 +02:00
Nicolas
35decb1af2 Nick: 2024-08-21 12:35:03 -03:00
rafaelsideguide
52abec41c2 fixing delete 2024-08-21 10:35:50 -03:00
Nicolas
db8c84ff0f Update requests.http 2024-08-21 10:19:37 -03:00
rafaelsideguide
b66553867e reverting delete, fixed express bug on checkCredits 2024-08-21 09:28:20 -03:00
rafaelsideguide
138437d616 commenting out delete, crashing on fire-engine
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Test Suite (push) Blocked by required conditions
Details
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Details
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Details
2024-08-21 08:11:24 -03:00
rafaelsideguide
5e48bec1fd commenting out delete, crashing on fire-engine 2024-08-21 08:10:46 -03:00
Nicolas
90b32f16c8 Nick: fixes 2024-08-20 21:38:11 -03:00