Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 03:32:22 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
2,200 Commits 273 Branches 1 Tag 48 MiB
fbabc779f5
Commit Graph

2200 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas
d1b838322d
Merge pull request #721 from mendableai/feat/concurrency-limit 
Concurrency limits
 2024-10-01 16:15:05 -03:00
Nicolas
ac5e1fc194 Update sitemap.ts 2024-10-01 16:14:43 -03:00
Nicolas
c6717fecaa Nick: got rid of job interval sleep and math.min 2024-10-01 16:11:12 -03:00
Nicolas
18f9cd09e1 Nick: fixed more stuff 2024-10-01 16:04:39 -03:00
Gergő Móricz
fe721fffbe fix(crawl-redis): normalize URL before locking 2024-10-01 20:59:50 +02:00
Nicolas
c0541cc990 Update queue-worker.ts 2024-10-01 15:38:24 -03:00
Nicolas
37299fc035 Update types.ts 2024-10-01 15:18:11 -03:00
Nicolas
8aa07afb6d Nick: fixes 2024-10-01 15:15:49 -03:00
Nicolas
92dbd33e57 Update queue-worker.ts 2024-10-01 14:53:26 -03:00
Nicolas
4d5477f357 Nick: resolved conflicts 2024-10-01 14:39:57 -03:00
Nicolas
96245e387d Update crawl.ts 2024-10-01 14:29:53 -03:00
Nicolas
258c67ce67 Revert "feat(queue-worker): always crawl links from content even if sitemapped" 
This reverts commit 3c045c43a4.
 2024-10-01 14:20:23 -03:00
Nicolas
445fc432e9 Reapply "fix(v1/crawl): always use sitemap" 
This reverts commit 339b19ce9d.
 2024-10-01 14:03:07 -03:00
Nicolas
339b19ce9d Revert "fix(v1/crawl): always use sitemap" 
This reverts commit 5dc0fcf644.
 2024-10-01 13:59:49 -03:00
Gergő Móricz
5dc0fcf644 fix(v1/crawl): always use sitemap 2024-10-01 18:49:44 +02:00
Gergő Móricz
3c045c43a4 feat(queue-worker): always crawl links from content even if sitemapped 2024-10-01 18:32:53 +02:00
Nicolas
1af26fe1b4 Nick: sitemap fix 2024-10-01 12:38:48 -03:00
Nicolas
ff4b7a835b
Merge pull request #685 from devflowinc/main
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details 
bugfix: using onlyIncludeTags and removeTags together
 2024-09-30 17:18:30 -03:00
Nicolas
986262e1d4 Update search.ts 2024-09-30 15:23:43 -03:00
Gergő Móricz
0dd06d33ef fix(v0/search): pass job priority 2024-09-30 19:20:24 +02:00
Gergő Móricz
20ffdbd15c hotfix 2024-09-30 19:17:52 +02:00
Gergő Móricz
a8df85fd9b fix(acuc): remove sentry capture 2024-09-30 19:10:24 +02:00
Gergő Móricz
3621e191bd feat(concurrency-limit): set limit based on plan 2024-09-28 00:19:54 +02:00
Gergő Móricz
c6a83ab92c fix(api): entrypoint
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Details
Fly Deploy / Deploy app (push) Has been cancelled
Details
2024-09-27 22:16:27 +02:00
Gergő Móricz
e44bdf7a54 bad dockerfile 2024-09-27 21:07:11 +02:00
Gergő Móricz
f0a1a2e45b fix: increase ulimit -n in docker 2024-09-27 20:44:52 +02:00
Gergő Móricz
d5e2a80e4a fix(crawl-status): keep 10 megabyte pages if they're the only thing in the output
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-27 20:41:41 +02:00
Nicolas
975f0575b4 Nick: max retries with axios-retry 2024-09-27 12:58:57 -04:00
Nicolas
92961cf74f Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-27 12:23:45 -04:00
Nicolas
1fdff87b3e Update single_url.ts 2024-09-27 12:23:44 -04:00
Gergő Móricz
6283e8fc47 fix(logger): set default level to trace 2024-09-27 17:46:43 +02:00
Gergő Móricz
5e8ef4954e feat(auth): log cache key in acuc update error 2024-09-27 17:13:10 +02:00
Gergő Móricz
e98f858eb6 fix(api): playground scrape errors
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-09-26 22:28:14 +02:00
Nicolas
8d44cb33bb Nick: fixed error message 2024-09-26 22:15:15 +02:00
Gergő Móricz
2cb493321a fix(ACUC): do not refresh cache every set 2024-09-26 22:15:15 +02:00
Gergő Móricz
9bdd344b36 fix(redlock): use redlock.using for stability 2024-09-26 22:15:15 +02:00
Gergő Móricz
250c3bb5c6 fix(auth): move redlock settings 2024-09-26 22:15:15 +02:00
Gergő Móricz
81245e68fa fix(auth/redlock): retry cached ACUC lock for 20 seconds 2024-09-26 22:15:15 +02:00
Gergő Móricz
0f89f5e7cb fix(billTeam): cache update race condition 2024-09-26 22:15:15 +02:00
Gergő Móricz
d13a97f979 fix(credit_billing): allow spending of exact credits 2024-09-26 22:15:15 +02:00
Gergő Móricz
84bff8add8 fix(billTeam): update cached ACUC after billing 2024-09-26 22:15:15 +02:00
Gergő Móricz
f22ab5ffaf feat(db): implement bill_team RPC 2024-09-26 22:15:15 +02:00
Gergő Móricz
c1f68c3e0a fix(credit_billing): return chunk.remaining_credits 2024-09-26 22:15:15 +02:00
Gergő Móricz
2073063fb7 fix(db): fix caching and rpc error 2024-09-26 22:15:15 +02:00
Gergő Móricz
f8c70fe5dd feat(db): implement auth_credit_usage_chunk RPC 2024-09-26 22:15:15 +02:00
Gergő Móricz
29815e084b feat(v1/Document): add warning field 2024-09-26 21:19:05 +02:00
Gergő Móricz
095babe70b fix(queue-jobs): jobs with concurrency fails may vanish 2024-09-26 21:18:56 +02:00
Gergő Móricz
b696bfc854 fix(crawl-status): avoid race conditions where crawl may be deemed failed 2024-09-26 21:00:27 +02:00
Eric Ciarla
75e32b0e5c
Merge pull request #707 from mendableai/new_examples 
o1 job recommender
 2024-09-26 14:51:45 -04:00
Eric Ciarla
20b998e66a Delete o1_job_recommender.ipynb 2024-09-26 14:51:07 -04:00