Nicolas
|
5abd26a267
|
Nick: set the crawl limit to the remaining credits
|
2024-08-20 14:16:54 -03:00 |
|
rafaelsideguide
|
e1c9cbf709
|
bug fixed. crawl should not stop if sitemap url is invalid
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
|
2024-08-20 09:11:58 -03:00 |
|
rafaelsideguide
|
ecd472356b
|
added variables to beta customers
|
2024-08-19 16:41:54 -03:00 |
|
Nicolas
|
8e4ca86463
|
Update crawl.ts
|
2024-08-19 11:02:24 -03:00 |
|
Nicolas
|
36b35dbc67
|
Update crawl.ts
|
2024-08-19 11:01:26 -03:00 |
|
rafaelsideguide
|
4ffc60596a
|
Update queue-worker.ts
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
|
2024-08-19 09:29:23 -03:00 |
|
rafaelsideguide
|
b8170aaa47
|
Update blocklist.ts
|
2024-08-19 08:51:48 -03:00 |
|
Nicolas
|
3fe82b4f12
|
Update queue-worker.ts
Fly Deploy / Pre-deploy checks (push) Has been cancelled
Fly Deploy / Test Suite (push) Has been cancelled
Fly Deploy / Python SDK Tests (push) Has been cancelled
Fly Deploy / JavaScript SDK Tests (push) Has been cancelled
Fly Deploy / Go SDK Tests (push) Has been cancelled
Fly Deploy / Deploy app (push) Has been cancelled
Fly Deploy / Build and publish Python SDK (push) Has been cancelled
Fly Deploy / Build and publish JavaScript SDK (push) Has been cancelled
|
2024-08-17 03:09:31 -04:00 |
|
Nicolas
|
f797380112
|
Nick:
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
|
2024-08-16 22:17:38 -04:00 |
|
Nicolas
|
47123be783
|
Nick: weird activity block
|
2024-08-16 22:01:56 -04:00 |
|
Gergő Móricz
|
c281fe62c0
|
fix(crawl): propagate db fix to preview endpoint
|
2024-08-16 23:43:54 +02:00 |
|
Gergő Móricz
|
e6738abf96
|
fix(crawl-status): retrieve from DB in bulk
|
2024-08-16 23:39:39 +02:00 |
|
Nicolas
|
78ca94251c
|
Merge pull request #480 from mendableai/nsc/hyper-v81
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
Reduce metrics ingestion w/ HyperDX v0.8.1
|
2024-08-16 14:34:14 -04:00 |
|
Gergő Móricz
|
fd6432e7fd
|
fix(queue-worker): correct job success
|
2024-08-16 19:16:08 +02:00 |
|
Gergő Móricz
|
6e54942265
|
fix(queue-worker): add cancelled to crawl log
|
2024-08-16 19:11:53 +02:00 |
|
Gergő Móricz
|
d0a8382a5b
|
fix(queue-worker): crawl finishing race condition
|
2024-08-16 18:48:52 +02:00 |
|
Gergő Móricz
|
6bd52e63bf
|
fix(queue-worker): fix linksOnPage undefined error
|
2024-08-16 18:42:24 +02:00 |
|
Gergő Móricz
|
5a6570cba2
|
fix(webhooks): call back with parent crawl ID
|
2024-08-16 17:42:42 +02:00 |
|
Nicolas
|
ec361609d2
|
Nick: added growth-2x plan
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
|
2024-08-15 18:37:19 -04:00 |
|
Nicolas
|
32c6b1f136
|
Nick: remove active job alerts
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
|
2024-08-15 14:50:30 -04:00 |
|
Gergő Móricz
|
0c14366720
|
fix: add checkandupdateurl to crawlPreview
|
2024-08-15 20:30:25 +02:00 |
|
Nicolas
|
81b2479db3
|
Merge pull request #459 from mendableai/feat/queue-scrapes
feat: Move scraper to queue
|
2024-08-15 14:19:55 -04:00 |
|
Gergő Móricz
|
fc08ff450d
|
search port
|
2024-08-15 20:10:59 +02:00 |
|
Nicolas
|
86326f34e9
|
Update single_url.test.ts
|
2024-08-15 13:48:42 -04:00 |
|
Gergő Móricz
|
129a882bcc
|
fix(scrape): give scrapes their real job id
|
2024-08-15 19:29:47 +02:00 |
|
Gergő Móricz
|
965a5817d1
|
fix(queue-worker): log jobs correctly
|
2024-08-15 19:27:15 +02:00 |
|
Gergő Móricz
|
dad9d353d9
|
use thomas's url validation
|
2024-08-15 19:19:02 +02:00 |
|
Gergő Móricz
|
e3279274f1
|
fix: make playground crawl work
|
2024-08-15 19:14:32 +02:00 |
|
Gergő Móricz
|
c5597bc722
|
fix: robots.txt laoding
|
2024-08-15 19:11:07 +02:00 |
|
Gergő Móricz
|
29f0d9ec94
|
propagate priority to fire-engine
|
2024-08-15 19:04:46 +02:00 |
|
Gergő Móricz
|
b79d3d1754
|
fix
|
2024-08-15 19:02:05 +02:00 |
|
Gergő Móricz
|
57730f6a35
|
priority changes
|
2024-08-15 18:58:07 +02:00 |
|
Gergő Móricz
|
846610681b
|
fix: fix posthog, add dummy crawl DB items
|
2024-08-15 18:55:18 +02:00 |
|
Nicolas
|
6e1074cdd1
|
Update website_params.ts
Fly Deploy / Pre-deploy checks (push) Waiting to run
Fly Deploy / Test Suite (push) Blocked by required conditions
Fly Deploy / Python SDK Tests (push) Blocked by required conditions
Fly Deploy / JavaScript SDK Tests (push) Blocked by required conditions
Fly Deploy / Go SDK Tests (push) Blocked by required conditions
Fly Deploy / Deploy app (push) Blocked by required conditions
Fly Deploy / Build and publish Python SDK (push) Blocked by required conditions
Fly Deploy / Build and publish JavaScript SDK (push) Blocked by required conditions
|
2024-08-14 17:39:54 -04:00 |
|
Thomas Kosmas
|
6410e1a81d
|
Update params
|
2024-08-15 00:10:14 +03:00 |
|
Gergő Móricz
|
8a5cad72f6
|
fix(queue-worker): variable name collision
|
2024-08-14 22:02:05 +02:00 |
|
Gergő Móricz
|
b8ec40dd72
|
fix(crawl): submit sitemapped jobs in bulk
|
2024-08-14 20:34:19 +02:00 |
|
Gergő Móricz
|
2ca1017fc3
|
fix(crawl): make request 0 of crawl jobs higher priority
|
2024-08-14 19:34:18 +02:00 |
|
Gergő Móricz
|
cfad067a63
|
fix(fly): change proxy limits
|
2024-08-14 18:52:40 +02:00 |
|
Gergő Móricz
|
a6c81f9d62
|
fix: return all data when calling webhook
|
2024-08-14 17:53:47 +02:00 |
|
Gergo Moricz
|
2e5e480cc2
|
fix(crawl): call webhooks
|
2024-08-13 22:10:17 +02:00 |
|
Gergo Moricz
|
a33596de3c
|
fix(log_job): add crawl_id
|
2024-08-13 22:03:46 +02:00 |
|
Gergo Moricz
|
9252940b52
|
fix(crawl-status): sort data
|
2024-08-13 21:55:13 +02:00 |
|
Gergo Moricz
|
8dbac0268c
|
feat: offload crawl results to the DB
|
2024-08-13 21:40:59 +02:00 |
|
Gergo Moricz
|
4bbc9db1df
|
fix: prioritize scrape jobs over crawl jobs
|
2024-08-13 21:31:34 +02:00 |
|
Gergo Moricz
|
5f2af37880
|
fix(scrape): remove scrape job from queue after the job is done
|
2024-08-13 21:26:41 +02:00 |
|
Gergo Moricz
|
2413e33359
|
fix(queue-worker): remove console.log
|
2024-08-13 21:07:36 +02:00 |
|
Gergo Moricz
|
d7549d4dc5
|
feat: remove webScraperQueue
|
2024-08-13 21:03:24 +02:00 |
|
Gergő Móricz
|
4a2c37dcf5
|
Merge branch 'main' into feat/queue-scrapes
|
2024-08-13 20:53:49 +02:00 |
|
Gergo Moricz
|
86e136beca
|
feat: crawl to scrape conversion
|
2024-08-13 20:51:43 +02:00 |
|