Commit Graph

Select branches

Hide Pull Requests

194-fixing-2

194-sdk-ci-pipeline-for-publishing-pythonnode-sdk

ETL-unicorn-example-app

added-issue-templates

bug/crawl-limit

bug/data-check-in-python-sdk

bug/dealing-with-dns-error

bug/fix-clean-jobs

bug/fix-custom-scrape-pdf-google-drive

bug/fix-issue-310

bug/fixing-cicd

bug/issue-258

bugfix/crawl-docker

bugfix/empty-excludes

bugfix/empty-tags

bugfix/fire-engine-content

bugfix/issue-291

bugfix/issue-435

bugfix/issue-466

bugfix/issue-477

bugfix/lowercase-all-urls

bugfix/malformed-url-sitemap

bugfix/normalized-api-on-crawl-status

bugfix/partial-data-js-sdk

bugfix/scrape-test-failing

bugfix/switch-case-ratelimiter

bugfix/update-env-example

calebpeffer-github-seo

cjp/contributors-guide-and

cjp/email-to-posthog-logging

cjp/image-added-to-extract

cjp/linksOnPage

cjp/switch-license-to-AGPL

dependabot/npm_and_yarn/apps/api/dev-deps-0f7ee87ebe

dependabot/npm_and_yarn/apps/api/prod-deps-65027e8b4e

detect-pdfs

doc/crawl-limit-default

doc/issue-764

docs/update-docs

feat-sdks/cancel-crawl

feat/actions

feat/added-anthropic-vision-api

feat/allowbackwardcrawling-option

feat/allowed-urls

feat/beta-customers

feat/better-gdrive-pdf-fetch

feat/better-types-for-jssdk

feat/blocklist-social-media

feat/cancel-fire-engine

feat/check-redirect

feat/ci-cd

feat/ci-cd-env-secrets

feat/ci-cd-workflow

feat/concurrency-limit

feat/coupons

feat/custom-scraping-readme

feat/developer-notion-special-case

feat/drizzle

feat/fire-engine-chrome-cdp

feat/fire-engine-fallback-for-sitemap

feat/fullpage-screenshot

feat/gdrive-pdfs

feat/go-html-to-md-parser

feat/go-sdk-submodule

feat/gpt-4o

feat/idempotency-key

feat/iframe-support

feat/improv-filter-perfomance

feat/improv-self-host

feat/improving-reative-paths

feat/issue-205

feat/issue-264

feat/issue-266

feat/issue-273

feat/issue-342

feat/issue-413

feat/js-sdk-v0011

feat/key-auth

feat/logger

feat/max-depth

feat/maxDepthRelative

feat/parse-to-markdown-tables

feat/pdf-parser

feat/posthog-logging

feat/python-sdk-502

feat/queue-scrapes

feat/rate-limit-singleton

feat/rate-limits

feat/rawHtmlExtraction

feat/replace-all-paths-to-absolute-paths

feat/rpc-rework

feat/save-docs-on-supabase

feat/scrape-monitoring

feat/screenshot-support

feat/scroll-xpaths

feat/server-health-check-slack-message

feat/skipTlsVerification

feat/test-suite

feat/to-markdown

feat/type-improvements

feat/yoastseo-generated-sitemaps

fix--rawHtml-return-on-crawl

fix-for-partial-docs-timeout

fix/added-special-case-notion

fix/check-files

fix/e2e-tests

fix/filter-status-unknown-jobs

fix/fly-deploy-go-sdk

fix/issue-644

fix/issue-663

fix/issue-665

fix/issue-736

fix/issue-758

fix/js-sdk-full-page-screenshot

fix/new-url-on-utils-extract-links

fix/remove-base64-images

fix/rename-variables

fix/screenshot-fullpage

fix/scroll-action

fix/turndown-cpu-load-big-tables

fix/url-validation-params

fix/v1-maxDepth

go-parser-singleton

greenpay-fixes

improv/sdk-attempts

js-sdk-types-stuff

lead_enrichment

llm-extraction

llm-extraction-search

main

mog/acuc-cache-clear

mog/bulk-scrape

mog/complex-webhook

mog/concurrency-limit-2

mog/doctor

mog/fix-infinite-regex

mog/fix-job-stuck-2

mog/fix-relative-url-crawl

mog/js-sdk-cjs

mog/js-sdk-tests-fix

mog/no-cluster

mog/webscraper-refactor

mog/websocket-sdk

more-logging

new-read-me

new_examples

nit/change-firecrawl-email

nsc/Nick-set-the-crawl-limit-to-the-remaining-credits

nsc/abs-path-fix

nsc/added-support-for-ficrawl-npm

nsc/added-support-for-firecrawl-pip

nsc/admin-notifications

nsc/admin-router

nsc/allowed-keywords-in-blocklist

nsc/amazon

nsc/bill-team-async

nsc/cacheable-lookup

nsc/cancel-job

nsc/check-credits-optimization

nsc/check-team-credits-limit

nsc/chrome-cdp-for-params

nsc/cjs-support

nsc/clean-content

nsc/clusters

nsc/crawl-fixes

nsc/crawler-fixes

nsc/custom-vanta-refactor

nsc/data-del

nsc/digikey-tls

nsc/docx-support

nsc/error-js-sdk-improv

nsc/exec-js

nsc/exploration-play

nsc/fire-engine-beta

nsc/fire-engine-first-layer

nsc/fix-deadlocks

nsc/fix-headers

nsc/fix-json-error-handler

nsc/fix-scaling-fix-email

nsc/fix-tables

nsc/fixed-self-host-envs

nsc/fixes-credit-usage

nsc/free-credits-increase

nsc/fwd-header-auth

nsc/geo-to-location

nsc/geolocation

nsc/google-drive-pdf

nsc/hyper-dx-integration

nsc/hyper-v81

nsc/improved-billing-notifications

nsc/improved-blocklist

nsc/improved-map-search

nsc/improvemnts-fixes-misc

nsc/includeOnlyTags

nsc/initial-web-refac

nsc/job-logs

nsc/job-priority

nsc/llm-extraction-zod-integration

nsc/load-testing-nick

nsc/log-extractpr-options

nsc/logging-scrapers

nsc/manual-rate-limit

nsc/map-pagination

nsc/max-retries-fire-engine

nsc/mobile-screenshot

nsc/mobile-support

nsc/mvp-search

nsc/new-actions

nsc/new-extract

nsc/new-pricing

nsc/no-cluster-all

nsc/notion-fix

nsc/pageoptions-crawler

nsc/parallelize-fetching-sitemaps

nsc/partial-data-changes

nsc/pay-as-you-go-lw2

nsc/prevent-single-url-logs

nsc/profiling-full-scrape

nsc/python-sdk-waiting-fixes

nsc/rate-limit-fixes

nsc/rate-limiter-tests

nsc/redlock-cache-auth

nsc/redlock-email

nsc/refactor-scraping-order

nsc/removal-of-tags-in-include-main-content

nsc/retries-acuc-price-credits-fallback

nsc/returnOnlyUrls

nsc/rm-wait-before-click

nsc/rpc-cloudflare-issues

nsc/search-fix-version-v0

nsc/search-js-sdk

nsc/search-playground

nsc/sentry

nsc/seperate-rate-limit

nsc/serper-params

nsc/sitemap-fix-fire-engine

nsc/sitemap-only

nsc/slack-alerts-1

nsc/small-fe-print

nsc/small-handle-for-client-side-errors

nsc/smart-crawl-init-testing

nsc/speed-improvements

nsc/speed-up-crawl-4x

nsc/support-for-all-metadata

nsc/timeout-scrape

nsc/transactional-emails

nsc/usage-based-overuse

nsc/wait-for-param

nsc/webhook-self-host-fix

nsc/website-param-fixes

o1-crawler

o1_crawler_actions

python-sdk/next-handler

rafa/investigating-crawl-bugs

refactor-index

revert-246-194-sdk-ci-pipeline-for-publishing-pythonnode-sdk

test-sdks

test/crawl-options

test/e2e-tests-for-all-parameters

test/k8s-load-testing

test/load-testing

tests/crawl-limit-unit-tests

tests/e2e-to-unit

tests/metadata

tests/suite-v1

uiTemplate

v1-webscraper

v1/llm-extract

v1/mockup-controllers

v1/node-sdk

v1/python-sdk

v1/webhooks

website-specific-params

#10

#109

#11

#110

#118

#119

#12

#120

#126

#129

#13

#130

#131

#132

#134

#135

#137

#139

#14

#140

#142

#143

#144

#145

#148

#149

#151

#153

#158

#16

#160

#166

#167

#168

#17

#170

#174

#175

#176

#178

#180

#181

#183

#185

#186

#190

#191

#196

#20

#200

#204

#207

#21

#211

#212

#213

#216

#218

#22

#221

#223

#224

#228

#229

#232

#234

#235

#236

#237

#238

#239

#24

#24

#242

#244

#246

#248

#25

#250

#251

#252

#253

#255

#256

#260

#262

#263

#265

#268

#269

#27

#271

#272

#274

#275

#276

#277

#278

#278

#28

#280

#280

#282

#283

#285

#29

#290

#294

#296

#297

#298

#30

#301

#302

#303

#307

#308

#31

#312

#313

#316

#317

#318

#32

#321

#322

#323

#325

#326

#328

#329

#331

#332

#336

#337

#338

#34

#343

#343

#344

#344

#346

#347

#348

#349

#35

#350

#355

#355

#357

#36

#361

#362

#363

#37

#370

#371

#373

#374

#375

#378

#379

#38

#380

#381

#382

#386

#389

#391

#393

#394

#395

#396

#397

#398

#399

#4

#40

#403

#404

#405

#406

#410

#414

#417

#42

#420

#423

#424

#427

#429

#432

#433

#436

#438

#443

#444

#445

#447

#449

#450

#453

#454

#455

#457

#458

#459

#46

#460

#461

#469

#470

#471

#472

#473

#475

#478

#479

#480

#482

#483

#484

#485

#491

#492

#493

#494

#496

#497

#498

#499

#5

#500

#501

#502

#503

#504

#505

#506

#508

#512

#515

#516

#52

#520

#521

#522

#523

#524

#527

#529

#53

#531

#532

#534

#535

#536

#537

#541

#541

#542

#542

#549

#55

#550

#551

#552

#553

#554

#555

#559

#56

#560

#561

#562

#564

#565

#566

#571

#572

#573

#574

#575

#576

#579

#580

#582

#584

#584

#586

#590

#591

#593

#594

#596

#601

#601

#603

#604

#605

#607

#608

#61

#611

#612

#613

#614

#615

#615

#619

#62

#620

#623

#624

#626

#627

#628

#629

#63

#631

#631

#634

#634

#635

#637

#638

#639

#64

#640

#643

#645

#646

#648

#649

#65

#653

#654

#655

#66

#664

#669

#67

#670

#671

#672

#673

#674

#676

#677

#678

#679

#68

#682

#685

#689

#690

#691

#692

#693

#694

#695

#704

#706

#707

#714

#715

#716

#717

#718

#719

#72

#721

#726

#727

#728

#731

#732

#733

#739

#740

#741

#742

#743

#746

#746

#747

#75

#753

#754

#755

#757

#759

#760

#761

#762

#763

#765

#765

#766

#767

#768

#77

#772

#772

#773

#774

#775

#776

#777

#778

#779

#780

#783

#784

#785

#788

#789

#790

#793

#796

#797

#798

#799

#8

#800

#800

#801

#802

#803

#804

#805

#807

#808

#809

#816

#817

#817

#818

#818

#820

#820

#823

#823

#824

#827

#827

#828

#828

#829

#83

#830

#831

#832

#833

#834

#835

#836

#837

#837

#838

#84

#841

#842

#842

#843

#843

#844

#844

#847

#849

#855

#856

#858

#859

#861

#862

#863

#864

#865

#867

#869

#872

#875

#875

#876

#878

#880

#881

#886

#887

#888

#889

#890

#890

#891

#892

#893

#894

#894

#895

#899

#9

#90

#900

#900

#901

v1.0.0

Mono Color

• 6f45ab6691 feat: added timeouts to requests to prevent blocking requests Rui Rua 2024-11-11 14:32:31 +0000
• e241871b43 fixed scroll action on js sdk rafaelmmiller 2024-11-11 10:36:02 -0300
• f097cddf23 feat(scrapeURL/fire-engine): adjust timeout for waitFor/wait actions Móricz Gergő 2024-11-11 11:43:59 +0100
• e97864b806 fix(scrapeURL/llmExtract): better schema normalization Móricz Gergő 2024-11-11 10:55:45 +0100
• 1c55ce41be feat(ci): add sentry auth token to builds Móricz Gergő 2024-11-11 10:32:17 +0100
• 49df553768 fix(scrapeURL, logger): remove buggy ArrayTransport that causes memory leak Móricz Gergő 2024-11-11 10:27:55 +0100
• 720b6dddf6
  apps/api(deps): bump the prod-deps group across 1 directory with 41 updates dependabot[bot] 2024-11-11 07:11:37 +0000
• dfaa2aeb54
  apps/playwright-service(deps): bump the prod-deps group dependabot[bot] 2024-11-11 06:33:51 +0000
• 9defa133ff
  apps/test-suite(deps-dev): bump the dev-deps group dependabot[bot] 2024-11-11 06:14:01 +0000
• ca1226be07
  apps/test-suite(deps): bump the prod-deps group dependabot[bot] 2024-11-11 06:13:08 +0000
• 6172b6c76a
  Merge a858cb8970 into 84ad45c01f Prathamesh Pawar 2024-11-10 01:21:44 +0530
• 84ad45c01f
  Merge pull request #872 from mendableai/nsc/exec-js Nicolas 2024-11-08 22:02:01 -0500
• caa1c48e1c add parameter to crawleroptions Gergő Móricz 2024-11-08 23:39:11 +0100
• 628a98d594 fix(scrapeURL): only retain ArrayTransport in testing Gergő Móricz 2024-11-08 23:12:17 +0100
• eac3714c12 fixes scroll action rafaelmmiller 2024-11-08 17:40:45 -0300
• 085ac3e71c debug: worker stall check Móricz Gergő 2024-11-08 20:19:44 +0100
• 27c5a93f4e added next handler for python sdk (js is ok) rafaelmmiller 2024-11-08 15:39:38 -0300
• ef505f8d99 feat(scrapeURL/fire-engine): adjust timeout tuning Gergő Móricz 2024-11-08 17:24:19 +0100
• 1acef8e49b fix: converter missing Gergő Móricz 2024-11-08 17:11:22 +0100
• b8a6fb3524 fix(scrapeURL/checkStatus): bad handling of f-e under load Gergő Móricz 2024-11-08 16:29:56 +0100
• dc3a4e27fd move param to the right place Gergő Móricz 2024-11-08 16:25:11 +0100
• 6ecf24b85e feat(crawl): URL deduplication Gergő Móricz 2024-11-08 16:22:06 +0100
• 25e94ffd28 fix(scrapeURL): do not submit LLM schema errors to Sentry Gergő Móricz 2024-11-07 23:21:07 +0100
• a297c99ba8 fix(scrapeURL): error displaying Gergő Móricz 2024-11-07 23:18:24 +0100
• 79cadcb769 fix(scrapeURL/llmExtract): fill in required field as well Gergő Móricz 2024-11-07 22:48:57 +0100
• 0588f340c3 fix(scrapeURL/llmExtract): array schema fix Gergő Móricz 2024-11-07 22:46:59 +0100
• 552d55c8fc fix(scrapeURL): includeTags/excludeTags Gergő Móricz 2024-11-07 21:10:27 +0100
• dc34714c9f
  apps/api(deps): bump the prod-deps group across 1 directory with 40 updates dependabot[bot] 2024-11-07 20:06:10 +0000
• 8d467c8ca7
  WebScraper refactor into scrapeURL (#714) Gergő Móricz 2024-11-07 20:57:33 +0100
• b9e732bdd5 Update index.ts mog/webscraper-refactor Nicolas 2024-11-07 14:55:06 -0500
• ec0542e942 trim url-specific params Gergő Móricz 2024-11-07 20:43:25 +0100
• f9e775acdf Merge branch 'mog/webscraper-refactor' of https://github.com/mendableai/firecrawl into mog/webscraper-refactor Nicolas 2024-11-07 14:36:49 -0500
• a02c42ab01 Nick: Nicolas 2024-11-07 14:36:45 -0500
• 7198a28eed move geolocation to global f-e option, fix removeBase64Images Gergő Móricz 2024-11-07 20:31:16 +0100
• 8641829985 update comments Gergő Móricz 2024-11-07 20:23:23 +0100
• e1806ac027 Update index.test.ts Nicolas 2024-11-07 14:21:30 -0500
• 3db2212c9d Merge branch 'mog/webscraper-refactor' of https://github.com/mendableai/firecrawl into mog/webscraper-refactor rafaelmmiller 2024-11-07 16:10:47 -0300
• 054b73b758 fixed rawHtml rafaelmmiller 2024-11-07 16:10:45 -0300
• 7949ac2a3b Update index.test.ts Nicolas 2024-11-07 14:09:46 -0500
• 40d88820a1 comment rafaelmmiller 2024-11-07 15:50:33 -0300
• 692f42ce0c fixed some tests rafaelmmiller 2024-11-07 15:49:54 -0300
• cc89094e89 Merge branch 'test/e2e-tests-for-all-parameters' into mog/webscraper-refactor Nicolas 2024-11-07 13:07:29 -0500
• 49801ac779 ingest scrape events Móricz Gergő 2024-11-07 12:33:03 +0100
• 461eda8d33 expose engine results tracker for ScrapeEvents implementation Móricz Gergő 2024-11-07 00:35:38 +0100
• be40dcb217 add warning when final engine has feature deficit Móricz Gergő 2024-11-07 00:22:37 +0100
• 7a54291d12 yeet headers from url specific params Móricz Gergő 2024-11-07 00:13:57 +0100
• 66a6f919c6 fixes Móricz Gergő 2024-11-06 23:55:05 +0100
• ec782405bf todo test/e2e-tests-for-all-parameters rafaelmmiller 2024-11-06 19:46:45 -0300
• 1e098fe118 todo rafaelmmiller 2024-11-06 19:46:27 -0300
• 3bd27905be 403 rafaelmmiller 2024-11-06 19:31:41 -0300
• 8616fe6a31 Nick: skipTls feature flag? Nicolas 2024-11-06 17:24:51 -0500
• 13b4eea5fe Merge branch 'test/e2e-tests-for-all-parameters' of https://github.com/mendableai/firecrawl into test/e2e-tests-for-all-parameters rafaelmmiller 2024-11-06 19:22:20 -0300
• 3a374b21e8 added actions and base64 check rafaelmmiller 2024-11-06 19:21:28 -0300
• 48025e4cdf Update index.ts Nicolas 2024-11-06 17:19:49 -0500
• 81511077e9 Update scrape.ts Nicolas 2024-11-06 17:17:41 -0500
• 621df866d6 Nick: Nicolas 2024-11-06 16:49:14 -0500
• 9b271d759a fixed type rafaelmmiller 2024-11-06 18:11:52 -0300
• a539ad75e6 Merge remote-tracking branch 'origin/mog/webscraper-refactor' into test/e2e-tests-for-all-parameters Nicolas 2024-11-06 15:59:38 -0500
• 0f208facfe added e2e tests for most parameters. Still a few actions, location and iframes to be done. rafaelmmiller 2024-11-06 16:22:30 -0300
• 7500ebe4c6 Nick: exec js actions nsc/exec-js Nicolas 2024-11-06 11:56:23 -0500
• 3689b135a7
  Merge 6fbeb2846b into ed5a0d3cf2 dolonfly 2024-11-06 15:17:51 +0000
• 153f76b6ff
  Merge ad404995fa into ed5a0d3cf2 dolonfly 2024-11-06 15:17:51 +0000
• c3e2fb0fff
  Merge 9c475d63b7 into ed5a0d3cf2 Rafael Miller 2024-11-06 15:17:51 +0000
• cfd9e0dd78
  Merge 0834fd3107 into ed5a0d3cf2 Travis James 2024-11-06 15:17:51 +0000
• f030d134dd
  Merge 633235a69e into ed5a0d3cf2 barbarian360 2024-11-06 15:17:51 +0000
• d678778d79
  Merge 5ab3a8824e into ed5a0d3cf2 darker 2024-11-06 15:17:51 +0000
• d94b1c8fb0
  Merge a4f5e47a93 into ed5a0d3cf2 txrp0x9 2024-11-06 15:17:51 +0000
• 5d6a7ac313
  Merge 1a3d8a8752 into ed5a0d3cf2 txrp0x9 2024-11-06 15:17:51 +0000
• 80aa586df7
  Merge 256a98b86e into ed5a0d3cf2 Rafael Miller 2024-11-06 15:17:51 +0000
• c13c5741d2
  Merge 3bec0090bc into ed5a0d3cf2 txrp0x9 2024-11-06 15:17:51 +0000
• 3817a71d32
  Merge db90b67667 into ed5a0d3cf2 txrp0x9 2024-11-06 15:17:51 +0000
• f10c29889b
  Merge 7568743fa2 into ed5a0d3cf2 samir paudyal 2024-11-06 15:17:51 +0000
• d493521e71
  Merge 4549772058 into ed5a0d3cf2 sulav7 2024-11-06 15:17:51 +0000
• ed5a0d3cf2 Update readme and examples Eric Ciarla 2024-11-05 18:06:40 -0500
• 5e2124c6f9 feat(scrapeURL): add url-specific parameters Gergő Móricz 2024-11-06 00:03:35 +0100
• e5385e62ee Merge branch 'main' into mog/webscraper-refactor Gergő Móricz 2024-11-05 23:49:31 +0100
• 96beff83e2 feat(scrapeURL): playwright engine Gergő Móricz 2024-11-05 23:43:42 +0100
• fdec4e8cd2 feat(scrapeURL): basic fetch engine Gergő Móricz 2024-11-05 23:31:31 +0100
• 3f623fc9cb fix(tests/v0): grant larger response size to v0 crawl status Gergő Móricz 2024-11-05 22:55:36 +0100
• 7a1cf439b9 fix(scrapeURL/v0): search fix Gergő Móricz 2024-11-05 22:41:42 +0100
• 6ba51aaba9 fix(scrapeURL): LLM extract Gergő Móricz 2024-11-05 22:30:19 +0100
• 9144dba585 fix(scrapeURL): crawl stuff Gergő Móricz 2024-11-05 22:16:29 +0100
• 71e512b80f predicted-outputs Eric Ciarla 2024-11-05 16:16:24 -0500
• 75b48dced3
  Merge pull request #849 from swyxio/patch-1 Nicolas 2024-11-05 13:30:41 -0500
• 56179924e6 Merge branch 'main' of https://github.com/mendableai/firecrawl Nicolas 2024-11-05 13:08:02 -0500
• 3cd1c4260d Update rate-limiter.ts Nicolas 2024-11-05 13:08:01 -0500
• ae5ba74e2d
  Merge pull request #869 from mendableai/fix/new-url-on-utils-extract-links Nicolas 2024-11-05 11:26:26 -0500
• 8b69ccb7ff feat(scrapeURL): Add retry logic to robustFetch Móricz Gergő 2024-11-05 13:51:14 +0100
• cd534326ba fix crawl option conversion Móricz Gergő 2024-11-05 12:28:44 +0100
• f07bbef78e added trycatch and removed redundancy fix/new-url-on-utils-extract-links rafaelmmiller 2024-11-05 08:11:49 -0300
• 2a96717f67 fix(scrapeURL/extractMetadata): extract custom metadata Móricz Gergő 2024-11-05 11:40:55 +0100
• bc64ae3155 feat(requests.http): use dotenv expression Móricz Gergő 2024-11-05 11:30:10 +0100
• 262e733eec fix(logger): remove error debug key Móricz Gergő 2024-11-05 09:20:38 +0100
• d41b2d84dc fix(scrapeURL/fire-engine/chromecdp): fix wait action Móricz Gergő 2024-11-05 09:20:04 +0100
• 2fa25cb992
  [Fix] Prevent Python Firecrawl logger from interfering with loggers in client applications (#613) Dmitriy Vasilyuk 2024-11-04 23:33:39 -0800
• d62f5b82a4
  Merge branch 'main' into fix/python-logging Gergő Móricz 2024-11-05 08:33:12 +0100
• 9e22c9a428 Nick: etier1a Nicolas 2024-11-04 18:14:38 -0500
• f6e6f2ef9f Update pnpm-lock.yaml nsc/usage-based-overuse Nicolas 2024-11-04 15:00:24 -0500
• 5d55ef3f4d Merge branch 'main' into nsc/usage-based-overuse Nicolas 2024-11-04 15:00:04 -0500
• a5c9823495 haiku example Eric Ciarla 2024-11-04 14:58:05 -0500