308 Commits

Author	SHA1	Message	Date
takatost	73c86ee6a0	fix: prompt of title generation (#1084)	2023-09-01 14:55:58 +08:00
takatost	33562a9d8d	feat: optimize prompt (#1080)	2023-09-01 11:46:06 +08:00
takatost	a199fa6388	feat: optimize high load sql query of document segment (#1078)	2023-09-01 10:52:39 +08:00
takatost	4c8608dc61	feat: optimize conversation title generation output must be a valid JSON (#1077)	2023-09-01 10:31:42 +08:00
takatost	df6604a734	feat: optimize generation of conversation title (#1075)	2023-09-01 02:28:37 +08:00
takatost	78e26f8b75	fix: summary no docs (#1073)	2023-08-31 20:19:26 +08:00
takatost	2191312bb9	fix: segments query missing idx hit (#1072)	2023-08-31 19:39:44 +08:00
takatost	fcc6b41ab7	feat: decrease claude model request time by set max top_k to 10 (#1071)	2023-08-31 18:23:44 +08:00
takatost	d75e8aeafa	feat: disable anthropic retry (#1067)	2023-08-31 16:44:46 +08:00
takatost	2eba98a465	feat: optimize anthropic connection pool (#1066)	2023-08-31 16:18:59 +08:00
takatost	a7a7aab7a0	fix: csv import error (#1063)	2023-08-31 15:42:28 +08:00
yezhwi	d33a269548	refactor(file extractor): file extractor (#1059)	2023-08-31 14:45:31 +08:00
yezhwi	8b8e510bfe	fix: handle AttributeError for datasets and index (#1052)	2023-08-30 11:14:16 +08:00
takatost	417c19577a	feat: add LocalAI local embedding model support (#1021) ... Co-authored-by: StyleZhang <jasonapring2015@outlook.com>	2023-08-29 22:22:02 +08:00
Jyong	a43e80dd9c	add qdrant migration (#1046) ... Co-authored-by: jyong <jyong@dify.ai>	2023-08-29 10:37:04 +08:00
WangBooth	ad5f27bc5f	fix openpyxl dimensions error (#1041)	2023-08-29 10:36:48 +08:00
takatost	7b3314c5db	fix: dataset desc (#1045)	2023-08-29 09:07:27 +08:00
Jyong	a55ba6e614	Fix/ignore economy dataset (#1043) ... Co-authored-by: jyong <jyong@dify.ai>	2023-08-29 03:37:45 +08:00
takatost	02452421d5	fix: pub generate message text return null (#1037)	2023-08-28 16:43:54 +08:00
takatost	0796791de5	feat: hf inference endpoint stream support (#1028)	2023-08-26 19:48:34 +08:00
takatost	6c148b223d	fix: dataset query truncated (#1026)	2023-08-26 17:35:17 +08:00
Uranus	2d9616c29c	fix: xinference last token being ignored (#1013)	2023-08-25 18:15:05 +08:00
Jyong	2d604d9330	Fix/filter empty segment (#1004) ... Co-authored-by: jyong <jyong@dify.ai>	2023-08-25 15:50:29 +08:00
Jyong	b1fd1b3ab3	Feat/vector db manage (#997) ... Co-authored-by: jyong <jyong@dify.ai>	2023-08-24 21:27:31 +08:00
takatost	9ae91a2ec3	feat: optimize xinference request max token key and stop reason (#998)	2023-08-24 18:11:15 +08:00
takatost	701e2b334f	feat: remove unnecessary prompt of baichuan (#993)	2023-08-24 15:30:59 +08:00
takatost	bd3a9b2f8d	fix: xinference-chat-stream-response (#991)	2023-08-24 14:39:34 +08:00
takatost	18d3877151	feat: optimize xinference stream (#989)	2023-08-24 13:58:34 +08:00
takatost	53e83d8697	feat: optimize baichuan prompt (#988)	2023-08-24 12:07:10 +08:00
takatost	2c30d19cbe	feat: add baichuan prompt (#985)	2023-08-24 10:22:36 +08:00
takatost	9b247fccd4	feat: adjust hf max tokens (#979)	2023-08-23 22:24:50 +08:00
takatost	a76fde3d23	feat: optimize hf inference endpoint (#975)	2023-08-23 19:47:50 +08:00
Jyong	1fc57d7358	normalize embedding (#974) ... Co-authored-by: jyong <jyong@dify.ai>	2023-08-23 19:10:11 +08:00
takatost	5cba2e7087	fix: web reader tool retrieve content empty (#957)	2023-08-22 18:01:16 +08:00
Jyong	5623839c71	update document segment (#950) ... Co-authored-by: jyong <jyong@dify.ai>	2023-08-22 17:59:24 +08:00
takatost	78d3aa5fcd	fix: embedding init err (#956)	2023-08-22 17:43:59 +08:00
takatost	4f3053a8cc	fix: xinference chat completion error (#952)	2023-08-22 15:58:04 +08:00
takatost	866ee5da91	fix: openllm generate cutoff (#945)	2023-08-22 13:43:36 +08:00
takatost	e0a48c4972	fix: xinference chat support (#939)	2023-08-21 20:44:29 +08:00
Jyong	4b53bb1a32	Feat/token support (#909) ... Co-authored-by: StyleZhang <jasonapring2015@outlook.com> Co-authored-by: jyong <jyong@dify.ai>	2023-08-21 13:57:18 +08:00
takatost	4c49ecedb5	feat: optimize web reader summary in 3.5 (#933)	2023-08-21 11:58:01 +08:00
takatost	6c832ee328	fix: remove openllm pypi package because of this package too large (#931)	2023-08-21 02:12:28 +08:00
takatost	25264e7852	feat: add xinference embedding model support (#930)	2023-08-20 19:35:07 +08:00
takatost	18dd0d569d	fix: xinference max_tokens alisa error (#929)	2023-08-20 19:12:52 +08:00
takatost	3ea8d7a019	feat: add openllm support (#928)	2023-08-20 19:04:33 +08:00
takatost	da3f10a55e	feat: server xinference support (#927)	2023-08-20 17:46:41 +08:00
takatost	95b179fb39	fix: replicate text generation model validate (#923)	2023-08-19 21:40:42 +08:00
takatost	3a0a9e2d8f	fix: embedding get price definition missing (#922)	2023-08-19 21:31:40 +08:00
takatost	0a0d63457d	feat: record price unit in messages (#919)	2023-08-19 18:51:40 +08:00
takatost	920fb6d0e1	fix: embedding price config (#918)	2023-08-19 16:54:08 +08:00
Krasus.Chen	fd0fc8f4fe	Fix/price calc (#862)	2023-08-19 16:41:35 +08:00
takatost	1c552ff23a	fix: azure embedding model credentials include base_model_name is invalid for openai sdk (#917)	2023-08-19 16:24:18 +08:00
takatost	5163dd38e5	fix: run extra model serval ex not return (#916)	2023-08-19 14:35:16 +08:00
takatost	2a27dad2fb	fix: run model serval ex not return (#915)	2023-08-19 14:16:41 +08:00
Jyong	db7156dafd	Feature/mutil embedding model (#908) ... Co-authored-by: JzoNg <jzongcode@gmail.com> Co-authored-by: jyong <jyong@dify.ai> Co-authored-by: StyleZhang <jasonapring2015@outlook.com>	2023-08-18 17:37:31 +08:00
takatost	d9afebe216	feat: optimize output parse (#906)	2023-08-18 17:00:40 +08:00
takatost	1d9cc5ca05	fix: universal chat when default model invalid (#905)	2023-08-18 16:20:42 +08:00
takatost	edb06f6aed	fix: react router agent direct output (#904)	2023-08-18 14:31:20 +08:00
takatost	9adbeadeec	feat: claude paid optimize (#890)	2023-08-17 16:56:20 +08:00
takatost	2f7b234cc5	fix: max token not exist in generate summary when calc rest tokens (#891)	2023-08-17 16:33:32 +08:00
takatost	0cc0b6e052	fix: error raise status code not exist (#888)	2023-08-17 15:33:35 +08:00
takatost	f42e7d1a61	feat: add spark v2 support (#885)	2023-08-17 15:08:57 +08:00
takatost	c4d759dfba	fix: wenxin error not raise when stream mode (#884)	2023-08-17 13:40:00 +08:00
takatost	39574dcf6b	feat: optimize prompt of suggested_questions_after_answer (#881)	2023-08-17 10:46:33 +08:00
takatost	cc2d71c253	feat: optimize override app model config convert (#874)	2023-08-16 20:48:42 +08:00
takatost	2dfb3e95f6	feat: optimize error record in agent (#869)	2023-08-16 15:55:42 +08:00
Jyong	f207e180df	fix multi thread app context (#868) ... Co-authored-by: jyong <jyong@dify.ai>	2023-08-16 15:39:31 +08:00
takatost	948d64bbef	fix: get_num_tokens_from_messages params error (#866)	2023-08-16 14:58:44 +08:00
takatost	7a8590980e	fix: dataset direct output (#860)	2023-08-15 22:27:31 +08:00
takatost	e8c14bb732	feat: rename title in site both rename name in app (#857)	2023-08-15 20:42:32 +08:00
takatost	cc52cdc2a9	Feat/add free provider apply (#829)	2023-08-14 12:44:35 +08:00
takatost	8e15ba6cd6	Fix/no trial provider (#823)	2023-08-13 14:56:32 +08:00
takatost	7898937eae	feat: optimize message return (#822)	2023-08-13 13:51:12 +08:00
takatost	1bd0a76a20	feat: optimize error raise (#820)	2023-08-13 00:59:36 +08:00
takatost	2f179d61dc	fix: completion error when dataset was deleted (#819)	2023-08-13 00:25:05 +08:00
takatost	f18ce203b5	feat: optimize error logging (#808)	2023-08-12 02:22:43 +08:00
takatost	b81b8637ec	feat: temp remove paid option of anthropic (#807)	2023-08-12 01:54:38 +08:00
takatost	5fa2161b05	feat: server multi models support (#799)	2023-08-12 00:57:00 +08:00
lixiaoyin	cc277227ad	fix i is not incremented due to violating the uniqueness constraint w… (#771) ... Co-authored-by: 李啸吟 <746963140@qq.com>	2023-08-08 21:19:06 +08:00
Matri	dfe10e9dfe	fix: generate_more_like_this function issue (#722)	2023-08-03 11:37:09 +08:00
takatost	654985177f	fix: segment resort in dataset retrieve by index_node_id_to_position (#721)	2023-08-02 21:31:54 +08:00
takatost	e18211ffea	feat: fix azure completion choices return empty (#708)	2023-08-01 15:36:53 +08:00
Jyong	fa73aa8dbf	add embedding max retries (#699)	2023-07-31 23:28:37 +08:00
Jyong	174ebb51db	add qa thread control (#677)	2023-07-29 17:49:18 +08:00
John Wang	626c78a690	fix: agent parse result error (#676)	2023-07-29 17:00:38 +08:00
Jyong	9eaae770a6	Feat/add thread control (#675)	2023-07-29 17:00:21 +08:00
Jyong	ca60610306	logging qa error (#672)	2023-07-29 01:51:18 +08:00
Jyong	082f8b17ab	Feat/milvus support (#671) ... Co-authored-by: StyleZhang <jasonapring2015@outlook.com> Co-authored-by: JzoNg <jzongcode@gmail.com>	2023-07-28 22:19:39 +08:00
KVOJJJin	cf93d8d6e2	Feat: Q&A format segmentation support (#668) ... Co-authored-by: jyong <718720800@qq.com> Co-authored-by: StyleZhang <jasonapring2015@outlook.com>	2023-07-28 20:47:15 +08:00
John Wang	aae2fb8a30	fix: dataset retrieve npe when dataset desc is null (#669)	2023-07-28 17:40:36 +08:00
John Wang	d979955c8a	feat: optimize current time (#661)	2023-07-27 22:15:07 +08:00
John Wang	b5825142d1	feat: add current time tool in universal chat agent (#659)	2023-07-27 17:39:36 +08:00
John Wang	ba3dc8cae0	feat: fix dataset retrieve agent llm not support error (#656)	2023-07-27 15:45:52 +08:00
John Wang	4fdb37771a	feat: universal chat in explore (#649) ... Co-authored-by: StyleZhang <jasonapring2015@outlook.com>	2023-07-27 13:08:57 +08:00
John Wang	e9810a6df2	fix: azure openai embedding model name error (#612)	2023-07-20 13:52:54 +08:00
John Wang	cae15013e0	fix: azure openai deployment list was deprecated suddenly (#611)	2023-07-20 13:46:39 +08:00
John Wang	eff115267f	fix: anthropic completion error in blocking mode (#591)	2023-07-18 15:12:52 +08:00
John Wang	0d3cd3b16a	fix: azure provider select error when use custom azure provider (#587)	2023-07-18 14:34:09 +08:00
John Wang	3dc82fb044	feat: remove davinci required model from azure provider (#586)	2023-07-18 14:14:56 +08:00
John Wang	7599f79a17	feat: claude api support (#572)	2023-07-17 00:14:19 +08:00
zxhlyh	a03a92e9db	Feat/chat support voice input (#532)	2023-07-07 17:50:42 +08:00
John Wang	6eee7cb42c	feat: fix azure embedding Too many inputs problem (#530)	2023-07-07 11:17:36 +08:00
John Wang	b9b0866a46	fix: generate summary error when tokens=4097 (#488)	2023-06-29 12:54:50 +08:00
John Wang	df68a7c82b	feat: Optimize the quality of the title generate (#484)	2023-06-28 19:59:20 +08:00
John Wang	9d98669e7d	fix: dataset destination error (#477)	2023-06-28 15:51:07 +08:00
John Wang	408fbb0c70	fix: title, summary, suggested questions generate (#476)	2023-06-28 15:43:33 +08:00
crazywoola	998f819b04	use sub to operate all (#475)	2023-06-28 14:58:40 +08:00
Jyong	334f46d0b6	Fix/json format (#466)	2023-06-28 13:58:50 +08:00
Jyong	2eea114ac0	fix special code (#473)	2023-06-28 13:58:36 +08:00
Jyong	c67ecff3fe	Fix/json format (#465)	2023-06-27 17:15:03 +08:00
John Wang	d5b42c09ee	fix: template parse error when history include {{any}} (#463)	2023-06-27 16:35:50 +08:00
John Wang	835bf9fd8d	fix: template parse error when pre prompt include {{}} (#462)	2023-06-27 15:51:55 +08:00
John Wang	c720f831af	feat: optimize template parse (#460)	2023-06-27 15:30:38 +08:00
John Wang	df5763be37	feat: optimize openai error raise (#459)	2023-06-27 12:34:47 +08:00
John Wang	cad1532f7c	feat: optimize index_struct copy (#442)	2023-06-25 17:52:22 +08:00
John Wang	951afcaaed	feat: optimize weaviate error msg (#441)	2023-06-25 17:05:56 +08:00
John Wang	3241e4015b	feat: upgrade langchain (#430) ... Co-authored-by: jyong <718720800@qq.com>	2023-06-25 16:49:14 +08:00
lisaifei@cvte.com	23ef2262bd	fix: filter empty value in xlsx to improve vector similarity hit (#422)	2023-06-21 11:25:52 +08:00
Jyong	36dc05c4da	fix chinese encoding (#411)	2023-06-19 18:41:17 +08:00
Jyong	226f28edcb	Feature/self host notion import (#397)	2023-06-17 19:50:21 +08:00
Jyong	9253f72dea	Feat/dataset notion import (#392) ... Co-authored-by: StyleZhang <jasonapring2015@outlook.com> Co-authored-by: JzoNg <jzongcode@gmail.com>	2023-06-16 21:47:51 +08:00
John Wang	184afa69ff	feat: add gpt-3.5-turbo-16k support and update openai gpt-3.5-turbo & Embedding Ada v2 unit price (#358)	2023-06-14 12:17:43 +08:00
John Wang	ab115b5f87	fix: completion stop invalid (#355)	2023-06-13 17:47:42 +08:00
John Wang	a11f36ca60	fix: stop completion response not save to db (#351)	2023-06-13 15:47:58 +08:00
John Wang	cd136fb293	feat: add WEAVIATE_BATCH_SIZE (#349)	2023-06-13 14:49:40 +08:00
John Wang	6a3ab36101	feat: optimize weaviate batch size (#348)	2023-06-13 11:28:15 +08:00
Jyong	3d14431b96	Fix/excel data format (#334)	2023-06-09 20:21:11 +08:00
lisaifei@cvte.com	0abd67288b	feat: support xlsx file parsing (#304) ... Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com>	2023-06-09 15:57:19 +08:00
Jyong	2bf48514bc	fix markdown parser (#230)	2023-06-06 19:51:40 +08:00
John Wang	a3ee037d6d	feat: optimize output parse failed error (#298)	2023-06-05 11:23:51 +08:00
John Wang	490858a4d5	feat: auto rule generator (#273)	2023-05-31 22:03:15 +08:00
John Wang	44a1aa5e44	fix: dataset_tool npe (#274)	2023-05-31 17:16:27 +08:00
John Wang	75f7a96025	feat: ignore validate failed error log (#256)	2023-05-30 12:25:42 +08:00
John Wang	5ca88a4fd9	fix: raw json parse in llm router chain (#254)	2023-05-30 12:16:45 +08:00
John Wang	bb7f454ecd	fix: dataset desc npe (#246)	2023-05-29 19:56:36 +08:00
John Wang	7f48fadd41	fix: prompt template parantheses select error (#244)	2023-05-29 19:10:31 +08:00
John Wang	af2138e8b8	fix: json parse in router chain output (#243)	2023-05-29 18:25:01 +08:00
John Wang	88545184be	feat: support multi datasets router chain mode (#231)	2023-05-28 22:44:54 +08:00
John Wang	2c23caacd4	fix: introduction key error (#221)	2023-05-26 20:49:38 +08:00
John Wang	1a5acf43aa	Fix/shared lock (#210)	2023-05-25 21:31:11 +08:00
John Wang	d93365d429	fix: azure embedding not support batch (#188)	2023-05-24 18:55:07 +08:00
John Wang	4350bb9a00	Fix/human in answer (#174)	2023-05-23 19:54:04 +08:00
John Wang	056898bf21	fix: quota update error on azure openai (#172)	2023-05-23 16:16:22 +08:00
John Wang	6da5e54180	Feat/open azure validate (#163)	2023-05-23 14:16:26 +08:00
John Wang	f3219ff107	fix: template string in template error (#162)	2023-05-23 13:16:33 +08:00
John Wang	219011b62a	fix: disable template string in query (#160)	2023-05-23 12:57:26 +08:00
John Wang	90150a6ca9	Feat/optimize chat prompt (#158)	2023-05-23 12:26:28 +08:00
John Wang	9a5ae9f51f	Feat: optimize error desc (#152)	2023-05-22 17:39:28 +08:00
John Wang	54a6571462	fix: extra input for opening statement was not suitable for prompt (#143)	2023-05-22 14:32:22 +08:00
John Wang	b42cd38cc9	fix: internal error when user is none in service api call (#129)	2023-05-21 17:29:47 +08:00
John Wang	e8e8f9e97d	Fix: move pre prompt to user massasge in chat mode (#126)	2023-05-21 17:06:04 +08:00
John Wang	1b6e3ef964	Feat: optimize inner prompt (#121)	2023-05-21 11:29:10 +08:00
John Wang	8fc2663693	fix: weaviate batch insert timeout (#108)	2023-05-19 21:57:32 +08:00
John Wang	f68b05d5ec	Feat: support azure openai for temporary (#101)	2023-05-19 13:24:45 +08:00
John Wang	0587ff0fba	fix: remove empty segment in splitter (#68)	2023-05-17 15:02:58 +08:00
John Wang	815f794eef	feat: optimize split rule when use custom split segment identifier (#35)	2023-05-16 12:57:25 +08:00
John Wang	f5b2271c8c	fix: import wrong user (#32)	2023-05-15 23:30:58 +08:00
John Wang	db896255d6	Initial commit	2023-05-15 08:51:32 +08:00