Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 03:32:22 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Commit Graph

  • 95c4652fd4 Nick: 10min cooldown on auto charge Nicolas 2024-10-25 16:05:23 -0300
  • 4468a49aee concurrency limit fix PoC II. Gergő Móricz 2024-10-25 20:21:12 +0200
  • 256a98b86e always replace paths with absolutes fix/issue-736 rafaelsideguide 2024-10-25 15:14:43 -0300
  • 4590577cba Merge branch 'main' into nsc/pay-as-you-go-lw2 Nicolas 2024-10-25 14:02:24 -0300
  • a4f5e47a93 added tests for google drive content .txt and .pdf txrp0x9 2024-10-25 21:47:44 +0530
  • 91e7eaaf33 added verification_token to sdks txrp0x9 2024-10-25 21:27:52 +0530
  • 97b36c4755 webhook verification with verificationToken txrp0x9 2024-10-24 23:55:02 +0530
  • 177496d739 fix anthropic output parsing for extraction without schema - likely to be inconsistent because anthropic does not have a response_format param aar2dee2 2024-10-25 11:25:09 +0530
  • c7c846ae45 Merge branch 'main' into fly-dep aar2dee2 2024-10-25 10:53:25 +0530
  • dbcf2d7ff6 Nick: fix loggin for batch scrape Nicolas 2024-10-24 23:44:08 -0300
  • 73e6db45de Update email_notification.ts Nicolas 2024-10-24 23:14:41 -0300
  • d965f2ce7d Nick: fixes Nicolas 2024-10-24 23:13:30 -0300
  • 29b34270c8 Merge branch 'main' into nsc/pay-as-you-go-lw2 Nicolas 2024-10-24 22:31:04 -0300
  • 9a4ccd0801 Claude Web Crawler with Batch Scrape Eric Ciarla 2024-10-24 14:40:53 -0400
  • 629bf34bad Improved google drive support, added downloadable text type txrp0x9 2024-10-24 19:33:49 +0530
  • a1e8d2ee28 made secretkey optional txrp0x9 2024-10-24 21:32:36 +0530
  • d987f727fc added support for webhook authentication txrp0x9 2024-10-24 21:26:17 +0530
  • 2840991a50 add ANTHROPIC_API_KEY to .env.exampel aar2dee2 2024-10-24 20:45:59 +0530
  • 0e2a68d9ae get structured json using anthropic claude aar2dee2 2024-10-24 14:23:49 +0530
  • 8870cce75b cleanup: remove mongo commands from package.json skeptrune 2024-10-23 19:47:24 -0700
  • 6188f0f2f4 bugfix: do not exclude <prefix><excludeurl> patterns + require a subdomain skeptrune 2024-10-23 19:20:02 -0700
  • 23242d7150 cleanup: fully remove supabase dependency skeptrune 2024-10-23 17:58:59 -0700
  • 2a15d55e14 cleanup: add additional ignore URL patterns skeptrune 2024-10-23 16:58:17 -0700
  • b024f08639 cleanup: remove SLACK_WEBHOOK_URL env and code skeptrune 2024-10-23 16:36:25 -0700
  • a1756b0a38 cleanup: remove fire-engine skeptrune 2024-10-23 16:24:59 -0700
  • 8153225c43 cleanup: completely remove all code which uses supabase skeptrune 2024-10-23 14:01:30 -0700
  • c1b2a1c664 bugfix: use regex to exclude all subdomains for social media URLs + add wikipedia skeptrune 2024-10-23 13:15:37 -0700
  • 1da6360b77 feat(batch/scrape): restructure logs, add webhooks Gergő Móricz 2024-10-23 21:55:21 +0200
  • e3cb00990a Merge branch 'mog/bulk-scrape' Nicolas 2024-10-23 16:31:14 -0300
  • 19cac2220f Nick: mog/bulk-scrape Nicolas 2024-10-23 16:31:01 -0300
  • 76ca7fdcb5
         Merge pull request #789 from mendableai/mog/bulk-scrape Nicolas 2024-10-23 16:12:42 -0300
  • b11035814a Nick: Nicolas 2024-10-23 16:10:21 -0300
  • f0054da934 Nick: lgtm Nicolas 2024-10-23 16:06:08 -0300
  • c7f2170980 Update example.py Nicolas 2024-10-23 16:04:46 -0300
  • 60b6e6b1d4 Nick: fixes Nicolas 2024-10-23 15:59:40 -0300
  • d8abd15716 Nick: from bulk to batch Nicolas 2024-10-23 15:37:24 -0300
  • 70c4e7c334 feat(bulk/scrape): check credits via url list length Gergő Móricz 2024-10-23 19:42:02 +0200
  • 66e505317e Merge branch 'main' into mog/bulk-scrape Nicolas 2024-10-23 14:36:26 -0300
  • e0d3b761fc
         Merge pull request #808 from mendableai/feat/skipTlsVerification Nicolas 2024-10-22 20:47:13 -0300
  • 7432f25523
         Merge pull request #807 from mendableai/mog/acuc-cache-clear Nicolas 2024-10-22 20:43:18 -0300
  • d375bca167 Update acuc-cache-clear.ts mog/acuc-cache-clear Nicolas 2024-10-22 20:42:59 -0300
  • bbfdda8867 Nick: init Nicolas 2024-10-22 19:47:23 -0300
  • acde353e56 skipTlsVerification on robots.txt scraping feat/skipTlsVerification Thomas Kosmas 2024-10-23 01:07:03 +0300
  • 69bcbbe3c0 bugfix: only exclude social links w/ the full http:// skeptrune 2024-10-22 14:45:31 -0700
  • bd55464b52 skipTlsVerification Thomas Kosmas 2024-10-22 22:28:02 +0300
  • 6ed3104eb6 feat: clear ACUC cache endpoint based on team ID Gergő Móricz 2024-10-22 20:28:10 +0200
  • 3cd328cf93 feat(bulk/scrape): add node and python SDK integration + docs Gergő Móricz 2024-10-22 18:58:48 +0200
  • 5ab3a8824e
         chore(test): adapt src and add phpunit tests sanix-darker 2024-10-21 22:02:20 +0200
  • 76c0073829 Nick: grok 2 example Nicolas 2024-10-21 16:27:15 -0300
  • a7cb973aa1
         chore(php-sdk): initial commit sanix-darker 2024-10-21 06:20:26 +0200
  • d2344aa14b Revert "Nick: improved map ranking algorithm" Nicolas 2024-10-21 16:11:32 -0300
  • edac6850c6
         Merge pull request #797 from rishi-raj-jain/patch-2 Eric Ciarla 2024-10-21 12:05:00 -0400
  • 22d375ad29 Updates Eric Ciarla 2024-10-21 12:01:09 -0400
  • 9ab922837c
         Merge pull request #796 from mendableai/fix/issue-663 Nicolas 2024-10-21 12:24:57 -0300
  • d31b85fa91
         Merge pull request #793 from mendableai/fix/issue-665 Nicolas 2024-10-21 12:24:46 -0300
  • 9a2658768e
         Merge pull request #799 from Mefisto04/contribute Nicolas 2024-10-21 12:23:56 -0300
  • e1d8e1584e
         Update SELF_HOST.md Nicolas 2024-10-21 12:23:27 -0300
  • 209bbd1346
         Merge pull request #798 from mendableai/nsc/improved-map-search Nicolas 2024-10-21 12:22:19 -0300
  • b37d2a0000
         apps/test-suite(deps-dev): bump the dev-deps group dependabot[bot] 2024-10-21 07:11:05 +0000
  • 85a296fe4d
         apps/test-suite(deps): bump the prod-deps group dependabot[bot] 2024-10-21 07:10:12 +0000
  • 2b338c08c4
         apps/api(deps-dev): bump the dev-deps group in /apps/api with 12 updates dependabot[bot] 2024-10-21 06:35:00 +0000
  • 5014d6972a
         apps/api(deps): bump the prod-deps group in /apps/api with 37 updates dependabot[bot] 2024-10-21 06:32:34 +0000
  • 0a448608bf
         apps/playwright-service(deps): bump the prod-deps group dependabot[bot] 2024-10-21 06:15:40 +0000
  • cf98d69bbb
         Update requirements.txt Rishi Raj Jain 2024-10-20 18:09:38 +0530
  • d113199a29
         Update app.py Rishi Raj Jain 2024-10-20 18:08:38 +0530
  • 2b0c52ff67
         Update SELF_HOST.md Mayur Kawale 2024-10-20 12:33:45 +0530
  • 7acd8d2edb Nick: improved map ranking algorithm nsc/improved-map-search Nicolas 2024-10-19 13:27:47 -0300
  • 8a4ee4482d
         Create output_01f6efd5-1297-4745-94b5-5972c10f17d6.json Rishi Raj Jain 2024-10-19 03:54:14 +0530
  • 42ec08c76e
         Update websites.csv Rishi Raj Jain 2024-10-19 03:53:41 +0530
  • 7d8519218a
         Update app.py Rishi Raj Jain 2024-10-19 02:27:39 +0530
  • 2022db7f0a
         Update websites.csv Rishi Raj Jain 2024-10-19 02:27:25 +0530
  • f5af938ea2
         Update requirements.txt Rishi Raj Jain 2024-10-19 02:27:17 +0530
  • ba3ee8ead6
         Create .env.example Rishi Raj Jain 2024-10-19 00:52:47 +0530
  • adfc493c9b
         Create websites.csv Rishi Raj Jain 2024-10-19 00:52:26 +0530
  • 11fd630e55
         Create requirements.txt Rishi Raj Jain 2024-10-19 00:52:14 +0530
  • 10381b5d3c
         Create app.py Rishi Raj Jain 2024-10-19 00:51:18 +0530
  • 18f69c90b1 fix/missing error in response fix/issue-663 rafaelsideguide 2024-10-18 15:18:57 -0300
  • aed11e72a6 fix encoding if error fix/issue-665 rafaelsideguide 2024-10-18 11:50:58 -0300
  • 6ebfcc85cf
         Merge pull request #790 from rishi-raj-jain/patch-1 Eric Ciarla 2024-10-18 10:19:33 -0400
  • 79e65f31ef Update v1.ts Nicolas 2024-10-17 17:57:44 -0300
  • 7d8df7d53b
         Update requirements.txt Rishi Raj Jain 2024-10-18 02:11:15 +0530
  • a110fdeb85
         Update requirements.txt Rishi Raj Jain 2024-10-18 02:10:39 +0530
  • dff71a8179 Delete README.md Eric Ciarla 2024-10-17 15:48:36 -0400
  • 5f69358ce8 Swarm Extractor Example Eric Ciarla 2024-10-17 15:47:28 -0400
  • 03b37998fd feat: bulk scrape Gergő Móricz 2024-10-17 19:40:18 +0200
  • 081d7407b3
         Merge pull request #788 from mendableai/nsc/log-extractpr-options Nicolas 2024-10-16 23:51:22 -0300
  • 06b8d24a4c Update scrape.ts nsc/log-extractpr-options Nicolas 2024-10-16 23:50:21 -0300
  • a73b06589c
         Merge pull request #785 from mendableai/nsc/support-for-all-metadata Nicolas 2024-10-16 23:37:26 -0300
  • 2ac50a16f5 Update metadata.ts nsc/support-for-all-metadata Nicolas 2024-10-16 23:37:07 -0300
  • 8974230db4 Nick: formatting + error handling Nicolas 2024-10-16 23:35:03 -0300
  • c0384ea381 Nick: added tests Nicolas 2024-10-16 23:32:44 -0300
  • 417c7697c3 Update metadata.ts Nicolas 2024-10-16 23:26:46 -0300
  • ff906f7750 Update excludeTags.ts Nicolas 2024-10-16 13:40:34 -0300
  • 2c1a98f019 Update excludeTags.ts Nicolas 2024-10-16 13:37:40 -0300
  • cf8fe93281 Update credit_billing.ts Nicolas 2024-10-16 01:09:57 -0300
  • e5a5ca2446 Update credit_billing.ts Nicolas 2024-10-16 01:06:10 -0300
  • ffbf16048f
         Merge pull request #784 from mendableai/nsc/geolocation Nicolas 2024-10-15 21:49:16 -0300
  • 027158fa44 Nick: nsc/geolocation Nicolas 2024-10-15 21:47:27 -0300
  • 795e5a9228 Update metadata.ts Nicolas 2024-10-15 21:36:13 -0300
  • b4f6a0f919 Nick: geolocation Nicolas 2024-10-15 21:12:33 -0300