Austin/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2024-11-16 11:42:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
1,000 Commits 273 Branches 1 Tag 48 MiB
8160c311c0
Commit Graph

1000 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
dependabot[bot]
ad3e73b445
apps/test-suite(deps): bump the prod-deps group 
Bumps the prod-deps group in /apps/test-suite with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [@anthropic-ai/sdk](https://github.com/anthropics/anthropic-sdk-typescript) | `0.20.8` | `0.24.3` |
| [@dqbd/tiktoken](https://github.com/dqbd/tiktoken) | `1.0.14` | `1.0.15` |
| [@supabase/supabase-js](https://github.com/supabase/supabase-js) | `2.43.1` | `2.44.2` |
| [openai](https://github.com/openai/openai-node) | `4.40.2` | `4.52.2` |
| [playwright](https://github.com/microsoft/playwright) | `1.43.1` | `1.45.0` |
| [ts-jest](https://github.com/kulshekhar/ts-jest) | `29.1.2` | `29.1.5` |


Updates `@anthropic-ai/sdk` from 0.20.8 to 0.24.3
- [Release notes](https://github.com/anthropics/anthropic-sdk-typescript/releases)
- [Changelog](https://github.com/anthropics/anthropic-sdk-typescript/blob/main/CHANGELOG.md)
- [Commits](https://github.com/anthropics/anthropic-sdk-typescript/compare/sdk-v0.20.8...sdk-v0.24.3)

Updates `@dqbd/tiktoken` from 1.0.14 to 1.0.15
- [Release notes](https://github.com/dqbd/tiktoken/releases)
- [Changelog](https://github.com/dqbd/tiktoken/blob/main/CHANGELOG.md)
- [Commits](https://github.com/dqbd/tiktoken/compare/@dqbd/tiktoken@1.0.14...@dqbd/tiktoken@1.0.15)

Updates `@supabase/supabase-js` from 2.43.1 to 2.44.2
- [Release notes](https://github.com/supabase/supabase-js/releases)
- [Changelog](https://github.com/supabase/supabase-js/blob/master/RELEASE.md)
- [Commits](https://github.com/supabase/supabase-js/compare/v2.43.1...v2.44.2)

Updates `openai` from 4.40.2 to 4.52.2
- [Release notes](https://github.com/openai/openai-node/releases)
- [Changelog](https://github.com/openai/openai-node/blob/master/CHANGELOG.md)
- [Commits](https://github.com/openai/openai-node/compare/v4.40.2...v4.52.2)

Updates `playwright` from 1.43.1 to 1.45.0
- [Release notes](https://github.com/microsoft/playwright/releases)
- [Commits](https://github.com/microsoft/playwright/compare/v1.43.1...v1.45.0)

Updates `ts-jest` from 29.1.2 to 29.1.5
- [Release notes](https://github.com/kulshekhar/ts-jest/releases)
- [Changelog](https://github.com/kulshekhar/ts-jest/blob/main/CHANGELOG.md)
- [Commits](https://github.com/kulshekhar/ts-jest/compare/v29.1.2...v29.1.5)

---
updated-dependencies:
- dependency-name: "@anthropic-ai/sdk"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: "@dqbd/tiktoken"
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-deps
- dependency-name: "@supabase/supabase-js"
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: openai
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: playwright
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: ts-jest
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: prod-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-07-02 12:47:58 +00:00
dependabot[bot]
60de6bb6e3
apps/playwright-service(deps): bump the prod-deps group 
Bumps the prod-deps group in /apps/playwright-service with 3 updates: [hypercorn](https://github.com/pgjones/hypercorn), [fastapi](https://github.com/tiangolo/fastapi) and [playwright](https://github.com/Microsoft/playwright-python).


Updates `hypercorn` from 0.16.0 to 0.17.3
- [Changelog](https://github.com/pgjones/hypercorn/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pgjones/hypercorn/compare/0.16.0...0.17.3)

Updates `fastapi` from 0.110.0 to 0.111.0
- [Release notes](https://github.com/tiangolo/fastapi/releases)
- [Commits](https://github.com/tiangolo/fastapi/compare/0.110.0...0.111.0)

Updates `playwright` from 1.42.0 to 1.44.0
- [Release notes](https://github.com/Microsoft/playwright-python/releases)
- [Commits](https://github.com/Microsoft/playwright-python/compare/v1.42.0...v1.44.0)

---
updated-dependencies:
- dependency-name: hypercorn
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: fastapi
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
- dependency-name: playwright
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: prod-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-07-02 12:47:09 +00:00
Rafael Miller
3d530b461b
Merge pull request #337 from Sanix-Darker/f/cleaner-docker-compose 
[PROPOSAL] (docker-compose) regroup envs vars between services
 2024-07-02 09:46:22 -03:00
Rafael Miller
46ddc813e0
Merge pull request #338 from Sanix-Darker/dependabot 
[PROPOSAL] (deps): making sure all deps are always up to date
 2024-07-02 09:46:08 -03:00
Rafael Miller
f0f449fe51
Merge pull request #336 from snippet/allow-external-content-links 
[Proposal] new feature allowExternalContentLinks
 2024-07-02 09:45:21 -03:00
rafaelsideguide
db4a743365 Added e2e test 2024-07-02 09:44:08 -03:00
Eric Ciarla
0821017f5b
Update README.md 2024-07-02 07:08:46 -04:00
Nicolas
42cd58a679
Merge pull request #332 from mendableai/feat/rawHtmlExtraction 
Adds pageOptions.includeRawHtml and new extraction mode "llm-extraction-from-raw-html"
 2024-07-01 18:23:26 -03:00
Nicolas
c4f423981f Update pnpm-lock.yaml 2024-07-01 18:22:22 -03:00
rafaelsideguide
16aac7f8c5 Update single_url.ts 2024-07-01 18:21:15 -03:00
Nicolas
6d0c7a9ccd
Merge pull request #323 from mendableai/tests/crawl-limit-unit-tests 
[Tests] Added crawl limit unit test
 2024-07-01 17:56:04 -03:00
rafaelsideguide
4d6e25619b minor spacing and comment stuff 2024-07-01 16:05:34 -03:00
Eric Ciarla
e1af815f8c Update scrape.ts 2024-07-01 08:48:21 -04:00
sanix-darker
cf8208e3f0
feat(deps): making sure all deps are always up to date 2024-06-30 15:31:38 +02:00
sanix-darker
ff62b26001
feat: regroup what could be regroup in terms of environments variables between services (api and worker) 2024-06-30 15:22:21 +02:00
Eric Ciarla
7ae195bacc Update index.test.ts 2024-06-29 10:13:12 -04:00
Eric Ciarla
837b446390 Update index.test.ts 2024-06-29 08:48:42 -04:00
Eric Ciarla
fe6e3aeadc Update index.test.ts 2024-06-29 08:44:21 -04:00
Eric Ciarla
6c9f0dfc91 Add tests 2024-06-29 08:32:20 -04:00
Jeff Pereira
a5fb45988c new feature allowExternalContentLinks 2024-06-28 17:23:40 -07:00
Eric Ciarla
87b54488d3 update to includeRawHtml 2024-06-28 17:07:47 -04:00
Eric Ciarla
70fcf2ce03 init 2024-06-28 16:39:09 -04:00
Nicolas
9bf74bc774 Update single_url.ts 2024-06-28 15:51:18 -03:00
Nicolas
7e17498bcf Update single_url.ts 2024-06-28 15:45:16 -03:00
rafaelsideguide
7dffaaa3e2 Changed port and added "using with firecrawl" section on readme 2024-06-28 11:51:24 -03:00
rafaelsideguide
d66e1f7846 looking good 2024-06-27 16:00:45 -03:00
Nicolas
dbfae2d9bf
Merge pull request #329 from george-zakharov/patch-1 
Update CONTRIBUTING.md
 2024-06-27 12:12:31 -03:00
George Zakharov
5a0ec070bf
Update CONTRIBUTING.md 
Fix typo
 2024-06-27 13:50:31 +04:00
Nicolas
017b0b2556
Merge pull request #328 from mendableai/nsc/includeOnlyTags 
pageOptions.onlyIncludeTags param
 2024-06-26 21:33:10 -03:00
Nicolas
9e7298945c Update openapi.json 2024-06-26 21:25:38 -03:00
Nicolas
1ec0bf8adf Update openapi.json 2024-06-26 21:22:46 -03:00
Nicolas
042f81ddf2 Update removeUnwantedElements.test.ts 2024-06-26 21:20:11 -03:00
Nicolas
388ce3cbce Nick: small changes 2024-06-26 21:15:42 -03:00
Nicolas
1d4907acc9 Nick: 2024-06-26 21:02:58 -03:00
rafaelsideguide
c40da77be0 Added implementation for saving docs on supabase 
- TODO: remove the comments on `log_job.ts` before deploying to prod
 2024-06-26 18:23:28 -03:00
Jeff Pereira
d833a132a5 new playwright service 2024-06-26 12:32:30 -07:00
Nicolas
3b92fb8433
Merge pull request #322 from mendableai/tests/metadata 
[Test] Added E2E tests for checking metadata values
 2024-06-26 12:09:18 -03:00
rafaelsideguide
67d7650cf3 Added to e2e_noAuth 2024-06-26 12:07:55 -03:00
Nicolas
ac08e20c33
Merge pull request #321 from mendableai/bug/fix-issue-310 
[Bug] Added default values and fixed pdf bug
 2024-06-26 11:50:42 -03:00
Eric Ciarla
d80046d17c Gemini caching example 2024-06-26 09:48:15 -04:00
rafaelsideguide
009df6c930 Added crawl limit unit test 
I think this test is over relying on mocks but I have no idea on how to fix this without changing the code arch structure
 2024-06-26 09:54:25 -03:00
rafaelsideguide
05eaa3c68d Update index.test.ts 2024-06-26 09:32:02 -03:00
rafaelsideguide
4381109dd8 added default values and fixed pdf bug 2024-06-26 09:00:54 -03:00
Nicolas
45f2765601
Merge pull request #316 from snippet/types-webscraper 
add some types
 2024-06-25 22:03:21 -03:00
Nicolas
768a131b5c
Merge pull request #318 from mendableai/bug/fix-custom-scrape-pdf-google-drive 
[Bug] Fixed the regex test for google drive pdf files
 2024-06-25 18:27:11 -03:00
rafaelsideguide
5f69fc7677 Fixed the regex test 2024-06-25 18:24:01 -03:00
Nicolas
dbb22c8f0d
Merge pull request #317 from mendableai/bug/fix-clean-jobs 
[Bug] Fixed clean jobs
 2024-06-25 17:50:55 -03:00
rafaelsideguide
d02829d335 fixed clean jobs 2024-06-25 17:49:29 -03:00
Jeff Pereira
199cbe8bcb add some types 2024-06-25 12:20:25 -07:00
Nicolas
749b0c05dc Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-06-25 15:21:15 -03:00