Nicolas
|
d410804348
|
Merge pull request #755 from busaud/main
Deploy Images to GHCR / push-app-image (push) Waiting to run
bugfix: self-host crawling doesnt respect limit
|
2024-10-09 22:56:44 -03:00 |
|
Nicolas
|
abb5ec7439
|
Update playwright.ts
|
2024-10-09 22:55:01 -03:00 |
|
Nicolas
|
f6ec45f046
|
Merge pull request #747 from Harsh0707005/timeout-parameter-not-passed
Fixed Issue #734
|
2024-10-09 22:53:26 -03:00 |
|
Nicolas
|
222a34cae8
|
Update playwright.ts
|
2024-10-09 22:53:03 -03:00 |
|
busaud
|
c6ebbc6f6a
|
bugfix: self-host crawling doesnt respect limit
|
2024-10-09 22:52:49 +00:00 |
|
Nicolas
|
52ec43aac3
|
Update index.ts
|
2024-10-09 19:42:25 -03:00 |
|
Nicolas
|
5ff6c64d77
|
Update index.ts
|
2024-10-09 19:30:14 -03:00 |
|
Gergő Móricz
|
17d0ed061e
|
push
|
2024-10-09 23:13:26 +02:00 |
|
Gergő Móricz
|
b2ae1a52d5
|
fix(Dockerfile): remove chromium
|
2024-10-09 23:13:13 +02:00 |
|
Gergő Móricz
|
2d365ebc6d
|
fix(redis): protected mode off
|
2024-10-09 22:09:08 +02:00 |
|
busaud
|
237442fabb
|
Make sure the entrypoint script has the correct line endings
|
2024-10-09 20:58:37 +02:00 |
|
rafaelsideguide
|
ae464ada60
|
tests: teamIds
Deploy Images to GHCR / push-app-image (push) Waiting to run
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Fly Deploy / Deploy app (push) Has been cancelled
Fly Deploy Direct / Deploy app (push) Has been cancelled
|
2024-10-09 15:06:29 -03:00 |
|
Nicolas
|
1cd49a0a95
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-10-09 14:41:25 -03:00 |
|
Nicolas
|
064ce482c2
|
Update blocklist.ts
|
2024-10-09 14:41:23 -03:00 |
|
rafaelsideguide
|
4020a7d781
|
test: added test suite tokens
Deploy Images to GHCR / push-app-image (push) Waiting to run
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-10-08 15:11:08 -03:00 |
|
Gergő Móricz
|
075b63b57b
|
feat(redis): add memory calcualtion when not running on fly
|
2024-10-08 17:03:37 +02:00 |
|
Harsh Master
|
aa3d4b8d6c
|
Fixed Issue #734
|
2024-10-08 11:36:12 +05:30 |
|
Nicolas
|
5c0c952a27
|
Update website_params.ts
Deploy Images to GHCR / push-app-image (push) Waiting to run
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-10-07 14:51:05 -03:00 |
|
Nicolas
|
1f1afeaac4
|
Update system-monitor.ts
|
2024-10-04 15:15:04 -03:00 |
|
Nicolas
|
dba96998e3
|
Update fetch.ts
|
2024-10-03 18:56:51 -03:00 |
|
Nicolas
|
668ff3c71b
|
Update fetch.ts
|
2024-10-03 18:55:39 -03:00 |
|
Nicolas
|
25dd16bf2a
|
Nick: removed 401
|
2024-10-03 18:52:17 -03:00 |
|
Nicolas
|
93657f6a44
|
Update queue-worker.ts
|
2024-10-03 18:44:40 -03:00 |
|
Thomas Kosmas
|
28b64fc704
|
Change the gracefull shutdown signal
|
2024-10-04 00:40:09 +03:00 |
|
Nicolas
|
497ac3328b
|
Merge pull request #732 from mendableai/fix/url-validation-params
[BUG] Fixed URLs with params
|
2024-10-03 17:43:37 -03:00 |
|
rafaelsideguide
|
cfd776a5de
|
fix: now urls with params are passing validation
example: https://www.granitecreek.com?asljhda=akjshd
|
2024-10-03 17:37:04 -03:00 |
|
Nicolas
|
99ca852e5d
|
Merge pull request #731 from mendableai/nsc/crawl-fixes
Fixes crawl failed and webhooks not working properly
|
2024-10-03 17:37:03 -03:00 |
|
Nicolas
|
85e9f7b9b9
|
Merge pull request #727 from mendableai/nsc/error-js-sdk-improv
Improves error handler in Node SDK to return the status code
|
2024-10-03 17:36:31 -03:00 |
|
Nicolas
|
4f7608821f
|
Update package.json
|
2024-10-03 17:36:20 -03:00 |
|
Nicolas
|
f743f2b922
|
Update index.ts
|
2024-10-03 17:34:29 -03:00 |
|
Nicolas
|
c6a29efbed
|
Update crawl-status.ts
|
2024-10-03 17:33:38 -03:00 |
|
Nicolas
|
ddd774ed68
|
Nick:
|
2024-10-03 17:20:57 -03:00 |
|
Nicolas
|
82551bb6bc
|
Update index.test.ts
|
2024-10-03 17:13:30 -03:00 |
|
Nicolas
|
49bd95327e
|
Update types.ts
|
2024-10-03 17:00:33 -03:00 |
|
Nicolas
|
1a1ac9fd60
|
Nick:
|
2024-10-03 16:37:58 -03:00 |
|
Nicolas
|
a150aa820c
|
Nick: shouldnt fallback on a 400 + error code should be correct on page status code
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-10-03 15:21:42 -03:00 |
|
Nicolas
|
489a643391
|
Update index.ts
|
2024-10-02 20:25:52 -03:00 |
|
Gergő Móricz
|
26771e2e71
|
debug(zod): log unsupported protocol errors
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Fly Deploy / Deploy app (push) Has been cancelled
|
2024-10-01 22:13:28 +02:00 |
|
Nicolas
|
d1b838322d
|
Merge pull request #721 from mendableai/feat/concurrency-limit
Concurrency limits
|
2024-10-01 16:15:05 -03:00 |
|
Nicolas
|
ac5e1fc194
|
Update sitemap.ts
|
2024-10-01 16:14:43 -03:00 |
|
Nicolas
|
c6717fecaa
|
Nick: got rid of job interval sleep and math.min
|
2024-10-01 16:11:12 -03:00 |
|
Nicolas
|
18f9cd09e1
|
Nick: fixed more stuff
|
2024-10-01 16:04:39 -03:00 |
|
Gergő Móricz
|
fe721fffbe
|
fix(crawl-redis): normalize URL before locking
|
2024-10-01 20:59:50 +02:00 |
|
Nicolas
|
c0541cc990
|
Update queue-worker.ts
|
2024-10-01 15:38:24 -03:00 |
|
Nicolas
|
37299fc035
|
Update types.ts
|
2024-10-01 15:18:11 -03:00 |
|
Nicolas
|
8aa07afb6d
|
Nick: fixes
|
2024-10-01 15:15:49 -03:00 |
|
Nicolas
|
92dbd33e57
|
Update queue-worker.ts
|
2024-10-01 14:53:26 -03:00 |
|
Nicolas
|
4d5477f357
|
Nick: resolved conflicts
|
2024-10-01 14:39:57 -03:00 |
|
Nicolas
|
96245e387d
|
Update crawl.ts
|
2024-10-01 14:29:53 -03:00 |
|
Nicolas
|
258c67ce67
|
Revert "feat(queue-worker): always crawl links from content even if sitemapped"
This reverts commit 3c045c43a4.
|
2024-10-01 14:20:23 -03:00 |
|
Nicolas
|
445fc432e9
|
Reapply "fix(v1/crawl): always use sitemap"
This reverts commit 339b19ce9d.
|
2024-10-01 14:03:07 -03:00 |
|
Nicolas
|
339b19ce9d
|
Revert "fix(v1/crawl): always use sitemap"
This reverts commit 5dc0fcf644.
|
2024-10-01 13:59:49 -03:00 |
|
Gergő Móricz
|
5dc0fcf644
|
fix(v1/crawl): always use sitemap
|
2024-10-01 18:49:44 +02:00 |
|
Gergő Móricz
|
3c045c43a4
|
feat(queue-worker): always crawl links from content even if sitemapped
|
2024-10-01 18:32:53 +02:00 |
|
Nicolas
|
1af26fe1b4
|
Nick: sitemap fix
|
2024-10-01 12:38:48 -03:00 |
|
Nicolas
|
ff4b7a835b
|
Merge pull request #685 from devflowinc/main
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
bugfix: using onlyIncludeTags and removeTags together
|
2024-09-30 17:18:30 -03:00 |
|
Nicolas
|
986262e1d4
|
Update search.ts
|
2024-09-30 15:23:43 -03:00 |
|
Gergő Móricz
|
0dd06d33ef
|
fix(v0/search): pass job priority
|
2024-09-30 19:20:24 +02:00 |
|
Gergő Móricz
|
20ffdbd15c
|
hotfix
|
2024-09-30 19:17:52 +02:00 |
|
Gergő Móricz
|
a8df85fd9b
|
fix(acuc): remove sentry capture
|
2024-09-30 19:10:24 +02:00 |
|
Gergő Móricz
|
3621e191bd
|
feat(concurrency-limit): set limit based on plan
|
2024-09-28 00:19:54 +02:00 |
|
Gergő Móricz
|
c6a83ab92c
|
fix(api): entrypoint
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Fly Deploy / Deploy app (push) Has been cancelled
|
2024-09-27 22:16:27 +02:00 |
|
Gergő Móricz
|
e44bdf7a54
|
bad dockerfile
|
2024-09-27 21:07:11 +02:00 |
|
Gergő Móricz
|
f0a1a2e45b
|
fix: increase ulimit -n in docker
|
2024-09-27 20:44:52 +02:00 |
|
Gergő Móricz
|
d5e2a80e4a
|
fix(crawl-status): keep 10 megabyte pages if they're the only thing in the output
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-27 20:41:41 +02:00 |
|
Nicolas
|
975f0575b4
|
Nick: max retries with axios-retry
|
2024-09-27 12:58:57 -04:00 |
|
Nicolas
|
92961cf74f
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-09-27 12:23:45 -04:00 |
|
Nicolas
|
1fdff87b3e
|
Update single_url.ts
|
2024-09-27 12:23:44 -04:00 |
|
Gergő Móricz
|
6283e8fc47
|
fix(logger): set default level to trace
|
2024-09-27 17:46:43 +02:00 |
|
Gergő Móricz
|
5e8ef4954e
|
feat(auth): log cache key in acuc update error
|
2024-09-27 17:13:10 +02:00 |
|
Gergő Móricz
|
e98f858eb6
|
fix(api): playground scrape errors
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-26 22:28:14 +02:00 |
|
Nicolas
|
8d44cb33bb
|
Nick: fixed error message
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
2cb493321a
|
fix(ACUC): do not refresh cache every set
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
9bdd344b36
|
fix(redlock): use redlock.using for stability
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
250c3bb5c6
|
fix(auth): move redlock settings
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
81245e68fa
|
fix(auth/redlock): retry cached ACUC lock for 20 seconds
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
0f89f5e7cb
|
fix(billTeam): cache update race condition
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
d13a97f979
|
fix(credit_billing): allow spending of exact credits
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
84bff8add8
|
fix(billTeam): update cached ACUC after billing
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
f22ab5ffaf
|
feat(db): implement bill_team RPC
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
c1f68c3e0a
|
fix(credit_billing): return chunk.remaining_credits
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
2073063fb7
|
fix(db): fix caching and rpc error
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
f8c70fe5dd
|
feat(db): implement auth_credit_usage_chunk RPC
|
2024-09-26 22:15:15 +02:00 |
|
Gergő Móricz
|
29815e084b
|
feat(v1/Document): add warning field
|
2024-09-26 21:19:05 +02:00 |
|
Gergő Móricz
|
095babe70b
|
fix(queue-jobs): jobs with concurrency fails may vanish
|
2024-09-26 21:18:56 +02:00 |
|
Gergő Móricz
|
b696bfc854
|
fix(crawl-status): avoid race conditions where crawl may be deemed failed
|
2024-09-26 21:00:27 +02:00 |
|
Gergő Móricz
|
dec4171937
|
fix(queue-worker, queue-jobs): logic fixes
|
2024-09-26 20:39:19 +02:00 |
|
Gergő Móricz
|
d2881927c1
|
fix(queue-worker): remove concurrency entries when done in sentry-less branch
|
2024-09-26 20:29:17 +02:00 |
|
Gergő Móricz
|
53fce67ca1
|
feat(queue-worker): PoC of concurrency limits
|
2024-09-26 20:24:34 +02:00 |
|
Nicolas
|
30058b1da0
|
Nick: increased timeout for chrome-cdp due to smart wait
|
2024-09-26 20:24:34 +02:00 |
|
Nicolas
|
a9773a24a3
|
Nick: increased timeout for chrome-cdp due to smart wait
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-25 19:27:02 -04:00 |
|
Gergő Móricz
|
953d4fb197
|
fix(redlock): use redlock.using for stability
|
2024-09-25 22:47:42 +02:00 |
|
Gergő Móricz
|
eef116bef8
|
fix(auth): move redlock settings
|
2024-09-25 22:27:51 +02:00 |
|
Gergő Móricz
|
2c96d2eef6
|
fix(auth/redlock): retry cached ACUC lock for 20 seconds
|
2024-09-25 22:25:13 +02:00 |
|
Gergő Móricz
|
1cca9b8ae6
|
fix(billTeam): cache update race condition
|
2024-09-25 22:15:02 +02:00 |
|
Gergő Móricz
|
eb7317c08a
|
fix(credit_billing): allow spending of exact credits
|
2024-09-25 21:44:05 +02:00 |
|
Gergő Móricz
|
e67cbc2ca1
|
fix(billTeam): update cached ACUC after billing
|
2024-09-25 21:37:01 +02:00 |
|
Gergő Móricz
|
5a8eb17a82
|
feat(db): implement bill_team RPC
|
2024-09-25 20:57:45 +02:00 |
|
Gergő Móricz
|
415fd9f333
|
fix(credit_billing): return chunk.remaining_credits
|
2024-09-25 20:37:35 +02:00 |
|
Gergő Móricz
|
417adf8e96
|
fix(db): fix caching and rpc error
|
2024-09-25 19:42:45 +02:00 |
|
Gergő Móricz
|
331e826bca
|
feat(db): implement auth_credit_usage_chunk RPC
|
2024-09-25 19:25:18 +02:00 |
|
Nicolas
|
1da026b26e
|
Update single_url.ts
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-24 23:29:48 -04:00 |
|
Nicolas
|
b8266cc329
|
Update website_params.ts
|
2024-09-24 23:28:58 -04:00 |
|
Gergő Móricz
|
f00c0b82f9
|
fix(v1/scrape): add total wait specified in request to timeout
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-24 21:56:22 +02:00 |
|
Nicolas
|
3f138e559e
|
Update website_params.ts
|
2024-09-24 15:14:26 -04:00 |
|
Gergő Móricz
|
43730b5db6
|
feat(WebScraper): always report error of last scraper in order
|
2024-09-24 20:03:49 +02:00 |
|
Gergő Móricz
|
3e661a2087
|
fix(v1/crawl-cancel): avoid double authing
|
2024-09-24 20:01:34 +02:00 |
|
Gergő Móricz
|
4194525640
|
fix(blocklist): unblock TikTok Business page
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
This is just a regular business site, not social media.
|
2024-09-24 16:55:19 +02:00 |
|
Gergő Móricz
|
4a623c084a
|
fix(fly): don't use Depot builders (doesn't work)
|
2024-09-24 10:50:30 +02:00 |
|
Gergő Móricz
|
a59b5836d5
|
Revert error tallying
|
2024-09-24 10:27:49 +02:00 |
|
Gergő Móricz
|
a4b128e8b7
|
fix(rust): blocklisted error test
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-23 23:03:00 +02:00 |
|
Gergő Móricz
|
483f97d21b
|
fix(v0/search): don't sent scrape fail errors to Sentry
|
2024-09-23 18:49:27 +02:00 |
|
Gergő Móricz
|
d927cafeea
|
fix(queue-worker): don't send scraping errors to sentry
|
2024-09-23 18:48:01 +02:00 |
|
Gergő Móricz
|
677faa27f3
|
fix(WebScraper): explicitly ignore 404s
|
2024-09-23 18:47:07 +02:00 |
|
Gergő Móricz
|
83d8287c14
|
fix(v0, sentry): don't send all scraping methods failed errors to Sentry
|
2024-09-23 18:40:21 +02:00 |
|
Gergő Móricz
|
d2f7031069
|
fix(WebScraper): fatal error handler triggering for 404s
|
2024-09-23 18:33:10 +02:00 |
|
Nicolas
|
848a2b364a
|
Update package.json
|
2024-09-21 21:11:23 -04:00 |
|
Nicolas
|
dfdbae74c6
|
Update fireEngine.ts
|
2024-09-21 21:10:05 -04:00 |
|
Nicolas
|
fbb5f23016
|
Update index.ts
|
2024-09-21 20:53:33 -04:00 |
|
Nicolas
|
607e46267c
|
Update package.json
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-20 19:46:17 -04:00 |
|
Nicolas
|
db161ac55a
|
Nick: press + write
|
2024-09-20 19:45:23 -04:00 |
|
Nicolas
|
3fc5ce17d2
|
Nick: fixed error handling for v0 scrape
|
2024-09-20 18:35:30 -04:00 |
|
Nicolas
|
0690cfeaad
|
Merge branch 'main' into feat/actions
|
2024-09-20 18:24:13 -04:00 |
|
Gergő Móricz
|
95e4c8920b
|
fix(sdk/rust): license
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-20 21:55:05 +02:00 |
|
Gergő Móricz
|
e1a34b0a99
|
Revert "feat(scrape): scroll down/up with actions if fullpagescreenshot"
This reverts commit 815bfc8f07.
|
2024-09-20 21:43:22 +02:00 |
|
Gergő Móricz
|
815bfc8f07
|
feat(scrape): scroll down/up with actions if fullpagescreenshot
revert this if unneeded
|
2024-09-20 21:42:09 +02:00 |
|
Gergő Móricz
|
d663bbf0ca
|
feat(actions): add scroll
|
2024-09-20 21:41:53 +02:00 |
|
Gergő Móricz
|
3dd912ec91
|
feat(actions): add typeText, pressKey, fix playwright screenshot/waitFor
|
2024-09-20 21:02:53 +02:00 |
|
Gergő Móricz
|
719dfbccbb
|
Update docs
|
2024-09-20 20:30:46 +02:00 |
|
Gergő Móricz
|
939040bf44
|
Update docs and example
|
2024-09-20 20:10:11 +02:00 |
|
Gergő Móricz
|
3ec0bbe28d
|
feat(sdk/rust/crawl): paginate through results
|
2024-09-20 20:10:11 +02:00 |
|
Gergő Móricz
|
a078cdbd9d
|
Rust SDK 1.0.0
|
2024-09-20 20:10:11 +02:00 |
|
Gergő Móricz
|
93a20442e3
|
feat(sdk/rust): first batch of changes for 1.0.0
|
2024-09-20 20:10:11 +02:00 |
|
Gergő Móricz
|
506d5c2716
|
Revert "return links"
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
This reverts commit 2d597672be.
|
2024-09-19 20:07:06 +02:00 |
|
Anjor Kanekar
|
2d597672be
|
return links
|
2024-09-19 20:06:07 +02:00 |
|
anjor
|
c45a132cd5
|
Remove print statement in map
|
2024-09-19 20:06:07 +02:00 |
|
Gergő Móricz
|
01f42b980d
|
feat(scrape): add error tallying instead of empty response
|
2024-09-19 19:21:13 +02:00 |
|
Nicolas
|
b8b8522b33
|
Nick: fixed map exception
|
2024-09-19 12:49:33 -04:00 |
|
Nicolas
|
ef71940429
|
Merge pull request #674 from mendableai/nsc/map-pagination
v1/map improvements + higher limits
|
2024-09-19 11:31:23 -04:00 |
|
skeptrune
|
26825fc8fb
|
bugfix: using onlyIncludeTags and removeTags together
|
2024-09-19 00:29:13 -07:00 |
|
Nicolas
|
00bb958b32
|
Update v1-openapi.json
|
2024-09-18 15:59:11 -04:00 |
|
Nicolas
|
03eb47d426
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-09-18 15:58:54 -04:00 |
|
Nicolas
|
3a4dd8fc7e
|
Nick: v1 openapi spec
|
2024-09-18 15:58:53 -04:00 |
|
Gergő Móricz
|
712ca31615
|
minor fixes
|
2024-09-18 21:34:09 +02:00 |
|
Gergő Móricz
|
20d1855ad5
|
feat(js-sdk): actions integration
|
2024-09-18 20:47:56 +02:00 |
|
Gergő Móricz
|
093c064bff
|
feat(v1): add public actions api
|
2024-09-18 20:39:25 +02:00 |
|
Gergő Móricz
|
42d677fe3c
|
feat(fire-engine): port waitFor and screenshot to use actions
|
2024-09-18 20:04:54 +02:00 |
|
Gergő Móricz
|
c28e1e2959
|
fix(v1/zod): formats: add error message if both sc and sc@fP
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-18 19:35:38 +02:00 |
|
Nicolas
|
255db84879
|
Update package.json
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Deploy app (push) Blocked by required conditions
|
2024-09-17 15:06:53 -04:00 |
|
Gergő Móricz
|
b2b7f8d874
|
fix(js-sdk): default type for LLM extract
|
2024-09-17 20:49:01 +02:00 |
|
Nicolas
|
31633317dd
|
Update package.json
|
2024-09-17 12:33:07 -04:00 |
|
rafaelsideguide
|
fb8a2c7549
|
fixed screenshot typo and added test for fullpage screenshot
|
2024-09-17 11:03:33 -04:00 |
|
Nicolas
|
3c2bfe2da2
|
Update single_url.ts
Fly Deploy / Deploy app (push) Waiting to run
|
2024-09-17 01:58:47 -04:00 |
|
Nicolas
|
18b024c238
|
Update single_url.ts
|
2024-09-17 01:41:46 -04:00 |
|
Nicolas
|
a4039bd008
|
Revert "Update single_url.ts"
This reverts commit 0f8c0a570d.
|
2024-09-16 23:36:38 -04:00 |
|
Nicolas
|
0f8c0a570d
|
Update single_url.ts
|
2024-09-16 21:44:56 -04:00 |
|
Nicolas
|
9da3432596
|
Update map.ts
|
2024-09-16 12:03:14 -04:00 |
|
Nicolas
|
af4804e13b
|
Merge branch 'main' into nsc/map-pagination
|
2024-09-16 11:04:44 -04:00 |
|
Nicolas
|
030ecab6ee
|
Update rate-limiter.ts
|
2024-09-13 18:09:59 -04:00 |
|
Gergo Moricz
|
000a316cc3
|
fix(fire-engine): poll more frequently
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Rust SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Fly Deploy / Build and publish Rust SDK (push) Blocked by required conditions
|
2024-09-13 16:41:27 +02:00 |
|
Gergo Moricz
|
f7c4cee404
|
fix(queue-worker): don't send LLM extract hallucination error to Sentry
|
2024-09-13 14:02:49 +02:00 |
|
Gergő Móricz
|
0d1b46d476
|
fix(js-sdk): improve error logging
|
2024-09-12 21:30:17 +02:00 |
|
Nicolas
|
e6ac90c1a0
|
Update package.json
|
2024-09-12 14:01:47 -04:00 |
|
Gergő Móricz
|
d30356a22c
|
fix(js-sdk): infer keyword collision
|
2024-09-12 19:57:33 +02:00 |
|
Gergő Móricz
|
d497284b40
|
feat(api/queue): auto-remove completed jobs after 25 hours
|
2024-09-12 19:47:22 +02:00 |
|
Nicolas
|
620b02f9ca
|
Nick:
|
2024-09-12 12:51:14 -04:00 |
|
Gergő Móricz
|
a2903e75cf
|
feat(js-sdk): type-safe LLM extract
|
2024-09-12 18:48:19 +02:00 |
|
Nicolas
|
e1171ade1b
|
Merge branch 'main' into mog/fix-status-job-get
|
2024-09-12 11:35:36 -04:00 |
|
Nicolas
|
503c8b3efa
|
Update package-lock.json
|
2024-09-12 11:35:26 -04:00 |
|
Nicolas
|
e19dc68718
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-09-11 14:59:40 -04:00 |
|
Nicolas
|
99c1af0a9f
|
Update package.json
|
2024-09-11 14:59:36 -04:00 |
|
Gergő Móricz
|
5adfd74cc5
|
feat(js-sdk/test): add API_URL env var
|
2024-09-11 20:32:34 +02:00 |
|
Gergő Móricz
|
ad70c30be5
|
fix(js-sdk): check at bad if
|
2024-09-11 20:31:58 +02:00 |
|
Gergő Móricz
|
6e1cf2f40d
|
feat(js-sdk): fixes, update tests
|
2024-09-11 20:15:43 +02:00 |
|
Nicolas
|
5791d3a0e5
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-09-11 14:03:36 -04:00 |
|
Nicolas
|
4cd1065ae2
|
Update rate-limiter.ts
|
2024-09-11 14:03:34 -04:00 |
|
Gergő Móricz
|
51da038405
|
Merge branch 'main' into fix-attw
|
2024-09-11 19:45:13 +02:00 |
|
Gergő Móricz
|
75f4bcd2ad
|
Merge pull request #614 from MonsterDeveloper/dependencies
perf(js-sdk): move `dotenv` and `uuid` to `devDependencies`, fix `zod` import
|
2024-09-11 19:34:50 +02:00 |
|
Andrei Bobkov
|
f6fc71b46a
|
fix(js-sdk): bring back cjs exports
|
2024-09-11 17:53:17 +03:00 |
|
Gergő Móricz
|
97ffabff3a
|
fix(v1): converting bad docs always gives null
|
2024-09-10 21:21:20 +02:00 |
|
Gergő Móricz
|
83a165db0f
|
fix(v0/scrape): ensure url is string
|
2024-09-10 21:18:53 +02:00 |
|
Gergő Móricz
|
ad1a6fbc74
|
fix(v1/map): handle invalid URLs gracefully
|
2024-09-10 19:41:01 +02:00 |
|
Gergő Móricz
|
e19f7a102e
|
feat(js-sdk): paginate next on checkCrawlStatus + better types for CSR
|
2024-09-10 19:29:38 +02:00 |
|
Gergő Móricz
|
4ebc35c9dd
|
fix(crawl-status): add success: true
|
2024-09-10 18:59:09 +02:00 |
|
rafaelsideguide
|
f855ad3436
|
bumping py-sdk version
|
2024-09-10 10:29:44 -03:00 |
|
rafaelsideguide
|
ee8a54213c
|
fix(py-sdk): removed asyncio package
tested websocket with example.py without asyncio and it works with no problem.
|
2024-09-10 10:25:27 -03:00 |
|
rafaelsideguide
|
1074e976cd
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-09-10 09:09:43 -03:00 |
|
rafaelsideguide
|
45237a29dd
|
updated js-sdk examples
|
2024-09-10 09:09:39 -03:00 |
|
Nicolas
|
a7406031fa
|
Merge branch 'main' of https://github.com/mendableai/firecrawl
|
2024-09-10 06:53:30 -03:00 |
|
Nicolas
|
a17e1cac92
|
Rate bump
|
2024-09-10 06:53:24 -03:00 |
|
Gergo Moricz
|
b4dbf75537
|
fix(v1): check if url is string in blocklistMiddleware
Fixes FIRECRAWL-SCRAPER-JS-9Z
|
2024-09-10 10:25:14 +02:00 |
|
Gergo Moricz
|
26f2095de6
|
fix(v1): proper Invalid URL handling
|
2024-09-10 09:24:23 +02:00 |
|
Gergo Moricz
|
f8fbc71f91
|
fix(supabase-jobs): do not use RPCs
RPCs are more failure-prone for this use case than regular queries are.
|
2024-09-10 09:20:18 +02:00 |
|
Gergo Moricz
|
a6bcf7b438
|
fix(v0/crawl-status): don't crash on big crawls when requesting jobs from supabase
|
2024-09-10 08:53:40 +02:00 |
|
Nicolas
|
17e419a7fb
|
Nick:
|
2024-09-09 21:06:23 -03:00 |
|
Nicolas
|
ca9a781eb7
|
Update index.ts
|
2024-09-09 12:27:55 -03:00 |
|
Nicolas
|
22a5e85899
|
Update index.ts
|
2024-09-09 12:26:55 -03:00 |
|
Nicolas
|
60a15d00eb
|
Update types.ts
|
2024-09-08 16:39:12 -03:00 |
|
Nicolas
|
fbdfa1256b
|
Update credit_billing.ts
|
2024-09-08 13:07:10 -03:00 |
|
Nicolas
|
48c665519e
|
Update credit_billing.ts
|
2024-09-07 13:42:45 -03:00 |
|