Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 03:32:22 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
2,128 Commits 273 Branches 1 Tag 48 MiB
feat/iframe-support
Commit Graph

2128 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas
d0bd450f86 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-10-03 19:07:44 -03:00
Nicolas
bfbd2c83f8 Update README.md 2024-10-03 19:07:36 -03:00
Nicolas
4a21a55925
Merge pull request #733 from mendableai/nsc/fixed-self-host-envs 
Fixed the self host issues where methods don't work
 2024-10-03 19:03:51 -03:00
rafaelsideguide
d316d52c96 fixes docker-compose and 401 error 2024-10-03 19:02:32 -03:00
Nicolas
dba96998e3 Update fetch.ts 2024-10-03 18:56:51 -03:00
Nicolas
668ff3c71b Update fetch.ts 2024-10-03 18:55:39 -03:00
Nicolas
25dd16bf2a Nick: removed 401 2024-10-03 18:52:17 -03:00
Nicolas
93657f6a44 Update queue-worker.ts 2024-10-03 18:44:40 -03:00
Nicolas
75658c58a2 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-10-03 18:41:48 -03:00
Nicolas
9a056919b9 Create deploy-image.yml 2024-10-03 18:41:41 -03:00
Thomas Kosmas
28b64fc704 Change the gracefull shutdown signal 2024-10-04 00:40:09 +03:00
Nicolas
50c59b6de9 Update docker-compose.yaml 2024-10-03 18:18:02 -03:00
Nicolas
497ac3328b
Merge pull request #732 from mendableai/fix/url-validation-params 
[BUG] Fixed URLs with params
 2024-10-03 17:43:37 -03:00
rafaelsideguide
cfd776a5de fix: now urls with params are passing validation 
example: https://www.granitecreek.com?asljhda=akjshd
 2024-10-03 17:37:04 -03:00
Nicolas
99ca852e5d
Merge pull request #731 from mendableai/nsc/crawl-fixes 
Fixes crawl failed and webhooks not working properly
 2024-10-03 17:37:03 -03:00
Nicolas
85e9f7b9b9
Merge pull request #727 from mendableai/nsc/error-js-sdk-improv 
Improves error handler in Node SDK to return the status code
 2024-10-03 17:36:31 -03:00
Nicolas
4f7608821f Update package.json 2024-10-03 17:36:20 -03:00
Nicolas
f743f2b922 Update index.ts 2024-10-03 17:34:29 -03:00
Nicolas
c6a29efbed Update crawl-status.ts 2024-10-03 17:33:38 -03:00
Nicolas
ddd774ed68 Nick: 2024-10-03 17:20:57 -03:00
Nicolas
82551bb6bc Update index.test.ts 2024-10-03 17:13:30 -03:00
Nicolas
49bd95327e Update types.ts 2024-10-03 17:00:33 -03:00
Nicolas
1a1ac9fd60 Nick: 2024-10-03 16:37:58 -03:00
Nicolas
a150aa820c Nick: shouldnt fallback on a 400 + error code should be correct on page status code
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details
2024-10-03 15:21:42 -03:00
Nicolas
a66ef18635
Merge pull request #728 from bytrangle/patch-1 
Docs: Remove wait_until_done from python-sdk example
 2024-10-03 12:50:32 -03:00
Trang Le
961a87452e
Remove wait_until_done from python-sdk example 2024-10-03 16:24:58 +07:00
Nicolas
489a643391 Update index.ts 2024-10-02 20:25:52 -03:00
Gergő Móricz
26771e2e71 debug(zod): log unsupported protocol errors
Some checks failed
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Details
Fly Deploy / Deploy app (push) Has been cancelled
Details
2024-10-01 22:13:28 +02:00
Nicolas
d1b838322d
Merge pull request #721 from mendableai/feat/concurrency-limit 
Concurrency limits
 2024-10-01 16:15:05 -03:00
Nicolas
ac5e1fc194 Update sitemap.ts 2024-10-01 16:14:43 -03:00
Nicolas
c6717fecaa Nick: got rid of job interval sleep and math.min 2024-10-01 16:11:12 -03:00
Nicolas
18f9cd09e1 Nick: fixed more stuff 2024-10-01 16:04:39 -03:00
Gergő Móricz
fe721fffbe fix(crawl-redis): normalize URL before locking 2024-10-01 20:59:50 +02:00
Nicolas
c0541cc990 Update queue-worker.ts 2024-10-01 15:38:24 -03:00
Nicolas
37299fc035 Update types.ts 2024-10-01 15:18:11 -03:00
Nicolas
8aa07afb6d Nick: fixes 2024-10-01 15:15:49 -03:00
Nicolas
92dbd33e57 Update queue-worker.ts 2024-10-01 14:53:26 -03:00
Nicolas
4d5477f357 Nick: resolved conflicts 2024-10-01 14:39:57 -03:00
Nicolas
96245e387d Update crawl.ts 2024-10-01 14:29:53 -03:00
Nicolas
258c67ce67 Revert "feat(queue-worker): always crawl links from content even if sitemapped" 
This reverts commit 3c045c43a4.
 2024-10-01 14:20:23 -03:00
Nicolas
445fc432e9 Reapply "fix(v1/crawl): always use sitemap" 
This reverts commit 339b19ce9d.
 2024-10-01 14:03:07 -03:00
Nicolas
339b19ce9d Revert "fix(v1/crawl): always use sitemap" 
This reverts commit 5dc0fcf644.
 2024-10-01 13:59:49 -03:00
Gergő Móricz
5dc0fcf644 fix(v1/crawl): always use sitemap 2024-10-01 18:49:44 +02:00
Gergő Móricz
3c045c43a4 feat(queue-worker): always crawl links from content even if sitemapped 2024-10-01 18:32:53 +02:00
Nicolas
1af26fe1b4 Nick: sitemap fix 2024-10-01 12:38:48 -03:00
Nicolas
ff4b7a835b
Merge pull request #685 from devflowinc/main
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Details
Fly Deploy / Deploy app (push) Blocked by required conditions
Details 
bugfix: using onlyIncludeTags and removeTags together
 2024-09-30 17:18:30 -03:00
Nicolas
986262e1d4 Update search.ts 2024-09-30 15:23:43 -03:00
Gergő Móricz
0dd06d33ef fix(v0/search): pass job priority 2024-09-30 19:20:24 +02:00
Gergő Móricz
20ffdbd15c hotfix 2024-09-30 19:17:52 +02:00
Gergő Móricz
a8df85fd9b fix(acuc): remove sentry capture 2024-09-30 19:10:24 +02:00