Commit Graph

675 Commits

Author SHA1 Message Date
Kevin Swiber
33aa5cf0de
Moving comments of .env.example values from end-of-line to above-line. Self-host docs suggest using .env.example as a base. However, Docker doesn't respect end-of-line comments. It sets the comment as the actual value of the variable. This fix prevents that. 2024-08-12 12:24:46 -07:00
Nicolas
74a5125185 Nick: removed redlock 2024-08-12 15:07:30 -04:00
Nicolas
0bd1a820ee Update auth.ts 2024-08-12 13:42:09 -04:00
Nicolas
25a899eae3 Nick: redlock cache in auth 2024-08-12 13:37:47 -04:00
rafaelsideguide
0591000b64 bugfix includes excludes 2024-08-09 14:30:41 -03:00
Nicolas
f1f5605010 Update website_params.ts 2024-08-08 12:31:58 -04:00
Nicolas
b0abad07da
Merge pull request #496 from tak-s/improve-logging-level
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Improve logs
2024-08-07 22:01:12 -04:00
Gergo Moricz
fe9fdb578b revert bad hotfixes 2024-08-07 16:34:25 +02:00
Gergo Moricz
b7c01dcb9b fix(webScraperQueue): reduce retries to 2 2024-08-07 16:31:50 +02:00
Gergo Moricz
cdf7bad5b4 fix(runWebScraper): don't move to completed 2024-08-07 15:20:56 +02:00
Gergo Moricz
9df8719efa fix(queue-worker): raise queue log level to info 2024-08-07 14:56:04 +02:00
Gergo Moricz
7bb922071c fix(queue-worker): manually renew lock (testing) 2024-08-07 14:35:20 +02:00
Gergo Moricz
8216266d16 fix(scrape_log): display error properly 2024-08-07 14:19:20 +02:00
Gergo Moricz
2e2e80d679 fix(scrape-events): updateScrapeResult fix 2024-08-07 14:17:50 +02:00
Gergo Moricz
b5ec47fd96 fix(runWebScraper): don't fetch next job 2024-08-07 13:53:04 +02:00
Nicolas
3321ca9398
Merge pull request #504 from mendableai/feat/fullpage-screenshot
[Feat] Added fullpagescreenshot capabilities
2024-08-06 13:52:29 -04:00
rafaelsideguide
4d24a99d50 fix params 2024-08-06 09:34:43 -03:00
rafaelsideguide
3edc3a3d15 added fullpagescreenshot capabilities, wip on fire-engine side 2024-08-05 18:17:37 -03:00
rafaelsideguide
f32e8de156 fixes the empty excludes.filter undefined bug 2024-08-05 18:13:31 -03:00
tak-s
af9bc5c8bb Suppressed repetitive logs 2024-08-04 15:09:36 +09:00
Nicolas
1742e4ceae Nick: 2024-08-02 19:25:15 -04:00
Nicolas
39aecd974b Update redis-health.ts 2024-08-02 17:43:45 -04:00
Nicolas
b448e3c3ad Update website_params.ts 2024-08-02 14:26:35 -04:00
rafaelsideguide
4051630632 Update sitemap.ts 2024-08-02 11:32:48 -03:00
rafaelsideguide
8568b61015 bugfix for sitemaps 2024-08-02 11:03:01 -03:00
Nicolas
af68b7a785
Merge pull request #475 from mendableai/bugfix/issue-466
Some checks failed
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Check Redis / clean-jobs (push) Has been cancelled
[Bug] pdfs and logging pdf events, also added trycatchs for docx
2024-08-01 22:05:26 -04:00
rafaelsideguide
f48ff36b32 added .inc files and forced lower case comparison 2024-07-31 09:28:43 -03:00
Nicolas
ad6f6eff4b Update fireEngine.ts 2024-07-30 19:15:54 -04:00
Nicolas
f9827b2151 Update credit_billing.ts 2024-07-30 19:13:17 -04:00
Nicolas
6d99dedd3c Nick: fixed tests 2024-07-30 19:11:01 -04:00
Nicolas
a28ecc1f61 Nick: caching 2024-07-30 18:59:35 -04:00
rafaelsideguide
8f5174ffc7 Update auth.ts 2024-07-30 10:37:33 -03:00
rafaelsideguide
d25d7e7244 special case: developer.apple.com
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
2024-07-30 10:13:09 -03:00
Nicolas
5e8ffcf505 Update website_params.ts 2024-07-29 20:43:47 -04:00
Nicolas
7b813883ef Nick: first layer 2024-07-29 20:31:51 -04:00
Nicolas
e99c2568f4 Update auth.ts 2024-07-29 18:44:18 -04:00
Nicolas
968a2dc753 Nick: 2024-07-29 18:37:09 -04:00
Nicolas
04942bb9de Nick: 2024-07-29 18:31:43 -04:00
Nicolas
267d4681bf Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-07-29 17:21:15 -04:00
Nicolas
b4833c1694 Nick: increasing default timeout to 45s 2024-07-29 17:21:11 -04:00
Nicolas
7fa08100bf
Merge pull request #414 from NiuBlibing/support_model_name
Some checks are pending
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
support custom models
2024-07-29 13:21:29 -04:00
rafaelsideguide
49e3e64787 bugfix for pdfs and logging pdf events, also added trycatchs for docx 2024-07-29 14:13:46 -03:00
Nicolas
4c9d62f6d3 Nick: fixing sitemap fallback 2024-07-26 18:25:44 -04:00
Nicolas
091924a636 Nick: moving machines from mia to virginia 2024-07-26 17:37:46 -04:00
Nicolas
cb97871ff9 Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-07-26 17:21:11 -04:00
Nicolas
ff4266f09e Update pdfProcessor.ts 2024-07-26 17:21:09 -04:00
Nicolas
0c2e3a72cc
Merge pull request #460 from mendableai/nsc/admin-router
Admin router + Improve redis notifications
2024-07-26 12:16:14 -04:00
rafaelsideguide
96cec2a673 fix checking scrape log success content length 2024-07-26 12:00:52 -03:00
Nicolas
542270f4c2
Merge pull request #461 from mendableai/nsc/small-handle-for-client-side-errors
Client side error handling
2024-07-25 20:53:10 -04:00
Nicolas
dc6f825270 Update email_notification.ts 2024-07-25 20:43:50 -04:00