Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 11:42:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
601 Commits 273 Branches 1 Tag 48 MiB
5ae4d1caf5
Commit Graph

601 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas
f15b8f855e Update crawl.json 2024-05-15 12:57:24 -07:00
Nicolas
98dd672d0a Update crawl.json 2024-05-15 12:55:04 -07:00
Nicolas
499671c87f Update crawl.test.ts 2024-05-15 12:50:13 -07:00
Nicolas
58053eb423 Update rate-limiter.ts 2024-05-15 12:47:35 -07:00
Nicolas
4745d114be Update crawl.test.ts 2024-05-15 12:42:14 -07:00
Nicolas
1601e93d69 Merge branch 'main' into test/crawl-options 2024-05-15 12:34:47 -07:00
Nicolas
3678d3c986 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-05-15 12:11:18 -07:00
Nicolas
fd82982a31 Nick: 2024-05-15 12:11:16 -07:00
rafaelsideguide
4925ee59f6 added crawl test suite 2024-05-15 15:50:50 -03:00
Nicolas
ed211dc7f8
Merge pull request #149 from mendableai/nsc/speed-up-crawl-4x 
feat: 4x-5x faster crawler (fast mode)
 2024-05-15 11:48:32 -07:00
Nicolas
1b0d6341d3 Update index.ts 2024-05-15 11:48:12 -07:00
Nicolas
d10f81e7fe Nick: fixes 2024-05-15 11:28:20 -07:00
Nicolas
87570bdfa1 Update index.ts 2024-05-15 11:06:03 -07:00
rafaelsideguide
d4574851be Added rpc definition 2024-05-15 08:40:21 -03:00
rafaelsideguide
47c20c80ab Update auth.ts 2024-05-15 08:34:49 -03:00
Ikko Eltociear Ashimine
e91c122c69
Merge branch 'main' into patch-1 2024-05-15 12:14:52 +09:00
Nicolas
7d8ceab6de Merge branch 'feat/rate-limits' of https://github.com/mendableai/firecrawl into feat/rate-limits 2024-05-14 14:48:01 -07:00
Nicolas
0e0faa28b3 Update auth.ts 2024-05-14 14:47:36 -07:00
rafaelsideguide
672eddb999 updated rpc 2024-05-14 18:47:21 -03:00
Nicolas
4761ea510b Update rate-limiter.ts 2024-05-14 14:26:42 -07:00
rafaelsideguide
40ad97dee8 added rate limits 2024-05-14 18:08:31 -03:00
Nicolas
27e1e22a0a Update index.test.ts 2024-05-14 12:28:25 -07:00
Nicolas
a0fdc6f7c6 Nick: 2024-05-14 12:12:40 -07:00
Nicolas
7f31959be7 Nick: 2024-05-14 12:04:36 -07:00
Nicolas
8a72cf556b Nick: 2024-05-13 21:10:58 -07:00
Nicolas
26a092f780 Update index.ts 2024-05-13 21:04:49 -07:00
Nicolas
8101cbee37 Update index.ts 2024-05-13 21:02:47 -07:00
Nicolas
86b8439844 Nick: 2024-05-13 20:51:42 -07:00
Nicolas
a96fc5b96d Nick: 4x speed 2024-05-13 20:45:11 -07:00
Nicolas
e26008a833 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-05-13 19:54:13 -07:00
Nicolas
512449e1aa Nick: v21 2024-05-13 19:54:12 -07:00
Nicolas
bd27b0e17e
Merge pull request #142 from mendableai/doc/crawl-limit-default 
[Doc] Added default value for crawlOptions.limit
 2024-05-13 18:38:09 -07:00
Nicolas
aa0c8188c9 Nick: 408 handling 2024-05-13 18:34:00 -07:00
Nicolas
999176d576 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-05-13 13:57:34 -07:00
Nicolas
f3ec21d9c4 Update runWebScraper.ts 2024-05-13 13:57:22 -07:00
Nicolas
c9133f3d15
Merge pull request #145 from mendableai/nsc/timeout-scrape 
Timeout on /scrape
 2024-05-13 13:07:25 -07:00
Nicolas
65d89afba9 Nick: 2024-05-13 13:01:43 -07:00
Nicolas
3f090ffd7c
Merge pull request #144 from mendableai/feat/gpt-4o 
Update models.ts
 2024-05-13 12:24:30 -07:00
Eric Ciarla
4cc46d4af8 Update models.ts 2024-05-13 15:23:31 -04:00
rafaelsideguide
8eb2e95f19 Cleaned up 2024-05-13 16:13:10 -03:00
Nicolas
2ce045912f Nick: disable vision right now 2024-05-13 10:56:08 -07:00
rafaelsideguide
4737fe8711 Added missing instruction 2024-05-13 13:47:49 -03:00
rafaelsideguide
f4348024c6 Added check during scraping to deal with pdfs 
Checks if the URL is a PDF during the scraping process (single_url.ts).

TODO: Run integration tests - Does this strat affect the running time?

ps. Some comments need to be removed if we decide to proceed with this strategy.
 2024-05-13 09:13:42 -03:00
chand1012
5cbce060ed
chore: Update docker-compose.yaml with default values for PORT and HOST 2024-05-10 17:26:00 -04:00
chand1012
b498e9881c
chore: Update docker-compose.yaml network configuration 2024-05-10 17:23:22 -04:00
chand1012
2021a822ff
chore: Add firecrawl network to docker-compose.yaml 2024-05-10 17:20:33 -04:00
chand1012
0245066009
chore: Update docker-compose.yaml with default values for REDIS_URL and PLAYWRIGHT_MICROSERVICE_URL 2024-05-10 17:15:32 -04:00
Rafael Miller
5a2712fa5a
Merge branch 'main' into detect-pdfs 2024-05-10 15:53:13 -03:00
rafaelsideguide
bc6b929b43 [Bug] Fixing /crawl limit 2024-05-10 12:15:54 -03:00
rafaelsideguide
df16890f84 Added default value for crawlOptions.limit 2024-05-10 11:59:33 -03:00