Paper: arXiv 2312.14203
Authors: Zhongyang Guo, Guanran Jiang, Zhongdan Zhang, Peng Li, Zhefeng Wang, Yinchun Wang
Abstract
This paper introduces “Shai” a 10B level large language model specifically designed for the asset management industry, built upon an open-source foundational model. With continuous pre-training and fine-tuning using a targeted corpus, Shai demonstrates enhanced performance in tasks relevant to its domain, outperforming baseline models. Our research includes the development of an innovative evaluation framework, which integrates professional qualification exams, tailored tasks, open-ended question answering, and safety assessments, to comprehensively assess Shai’s capabilities. Furthermore, we discuss the challenges and implications of utilizing large language models like GPT-4 for performance assessment in asset management, suggesting a combination of automated evaluation and human judgment. Shai’s development, showcasing the potential and versatility of 10B-level large language models in the financial sector with significant performance and modest computational requirements, hopes to provide practical insights and methodologies to assist industry peers in their similar endeavors.
Complexity vs Empirical Score
- Math Complexity: 1.0/10
- Empirical Rigor: 7.0/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper is a technical implementation report for building a domain-specific large language model (LLM), focusing on data collection, fine-tuning, and a custom evaluation framework, with minimal theoretical math presented. It demonstrates high empirical rigor through the description of specific datasets, training processes, and comparative evaluations against baseline models on proprietary tasks.
Research Flowchart
flowchart TD
A["Research Goal: Develop & Evaluate a 10B-level LLM for Asset Management"] --> B["Methodology: Continuous Pre-training & Fine-tuning"]
B --> C{"Data Inputs: Targeted Corpus & Professional Exams"}
C --> D["Computational Process: 10B Parameter Model Training"]
D --> E["Innovative Evaluation Framework"]
E --> F["Key Findings: Outperforms Baselines"]
E --> G["Key Findings: Significant Performance with Modest Compute"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper proposes Shai, a 10B‑parameter large language model tailored for asset management.
Source sentence
This paper introduces "Shai" a 10B level large language model specifically designed for the asset management industry
Assumes
No explicit assumptions found in the text.
Data
- proprietary evaluation dataset benchmark proprietary
Source sentence
bring Shai into direct comparison with mainstream 10B-level open-source models, such as baichuan2[18], Qwen[19], InterLM[20], and Xverse[21], on our proprietary dataset.
Costs
Not applicable
Finds 2
- Shai achieves the best performance among mainstream 10B‑level open‑source LLMs in asset‑management tasks. performance
Source sentence
we are the first to build a 10B level LLM for asset management, which achieve the best performance comparing to the mainstream 10B-level LLMs.
- Shai excels in financial examinations, practical tasks, and safety assessments, demonstrating high reliability for the asset‑management industry. performance
Source sentence
The results demonstrate that Shai excels not only in financial examinations and practical tasks but also in critical safety aspects, such as economic security and adherence to ethical standards.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 7 related papers). Coverage is partial, so this is neither replication nor refutation.
Code
No repository by the authors found in the paper.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (56)
- Baichuan 2: Open large-scale language models (2023) — link
- Qwen technical report (2023) — link
- Llama 2: Open foundation and fine-tuned chat models (2023) — link
- Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters (2023)
- GPT-4 Technical Report (2023) — link
- XVERSE-13B: A multilingual large language model developed by XVERSE Technology Inc. (2023)
- ChatML (2022)
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2023) — link
and 48 more
- Attention is all you need (2017)
- Llama: Open and efficient foundation language models (2023) — link
- Palm 2 technical report (2023) — link
- Palm: Scaling language modeling with pathways (2022) — link
- Bloomberggpt: A large language model for finance (2023) — link
- Moss: Training conversational language models from synthetic data (2023) — link
- Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (2023)
- Stanford alpaca: an instruction-following llama model (2023) (2023)
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023) — link
- RefGPT: Reference-to-Dialogue by GPT and for GPT (2023)
- Agieval: A human-centric benchmark for evaluating foundation models (2023) — link
- C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models (2023) — link
- Falcon-40B: an open large language model with state-of-the-art performance (2023)
- Internlm: A multilingual language model with progressively enhanced capabilities (2023)
- Glm-130b: An open bilingual pre-trained model (2022) — link
- Alpaca-CoT: An instruction-tuning platform with unified interface of instruction collection, parameter-efficient methods, and large language models, 2023
- Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering (2022)
- Agieval: A human-centric benchmark for evaluating foundation models (2023) — link
- C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models (2023) — link
- CMMLU: Measuring massive multitask language understanding in Chinese (2023) — link
- Evaluating the Performance of Large Language Models on GAOKAO Benchmark (2023) — link
- Measuring mathematical problem solving with the math dataset (2021) — link
- JEC-QA: a legal-domain question answering dataset (2020)
- Measuring massive multitask language understanding (2020) — link
- Training verifiers to solve math word problems (2021) — link
- BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark (2023) — link
- Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering (2022) — link
- Duee-fin: A document-level event extraction dataset in the financial domain released by baidu (2021)
- Safetybench: Evaluating the safety of large language models with multiple choice questions (2023) — link
- Wider and deeper llm networks are fairer llm evaluators (2023) — link
- Prefix-tuning: Optimizing continuous prompts for generation (2021) — link
- P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks (2022)
- Chain-of-thought prompting elicits reasoning in large language models (2022)
- Large language models are not fair evaluators (2023) — link
- A Wilcoxon-type test for trend (1985)
- Verbosity Bias in Preference Labeling by Large Language Models (2023) — link
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023) — link
- Chatlaw: Open-source legal large language model with integrated external knowledge bases (2023) — link
- A Brief Report on LawGPT 1.0: A Virtual Legal Assistant Based on GPT-3 (2023) — link
- Lawyer LLaMA Technical Report (2023) — link
- Doctorglm: Fine-tuning your chinese doctor is not a herculean task (2023) — link
- Huatuo: Tuning llama model with chinese medical knowledge (2023) — link
- SoulChat: Improving LLMs’ Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversations (2023) — link
- BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT (2023) — link
- FinGPT: Open-Source Financial Large Language Models — → 2306.06031
- DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning (2023) — link
- Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters (2023)
- Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link
1 of 56 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.