Paper: arXiv 2312.14203

Authors: Zhongyang Guo, Guanran Jiang, Zhongdan Zhang, Peng Li, Zhefeng Wang, Yinchun Wang

Abstract

This paper introduces “Shai” a 10B level large language model specifically designed for the asset management industry, built upon an open-source foundational model. With continuous pre-training and fine-tuning using a targeted corpus, Shai demonstrates enhanced performance in tasks relevant to its domain, outperforming baseline models. Our research includes the development of an innovative evaluation framework, which integrates professional qualification exams, tailored tasks, open-ended question answering, and safety assessments, to comprehensively assess Shai’s capabilities. Furthermore, we discuss the challenges and implications of utilizing large language models like GPT-4 for performance assessment in asset management, suggesting a combination of automated evaluation and human judgment. Shai’s development, showcasing the potential and versatility of 10B-level large language models in the financial sector with significant performance and modest computational requirements, hopes to provide practical insights and methodologies to assist industry peers in their similar endeavors.

Complexity vs Empirical Score

  • Math Complexity: 1.0/10
  • Empirical Rigor: 7.0/10
  • Quadrant: Street Traders — empirical and applied, lower math complexity

Why this score: The paper is a technical implementation report for building a domain-specific large language model (LLM), focusing on data collection, fine-tuning, and a custom evaluation framework, with minimal theoretical math presented. It demonstrates high empirical rigor through the description of specific datasets, training processes, and comparative evaluations against baseline models on proprietary tasks.

Research Flowchart

  flowchart TD
  A["Research Goal: Develop & Evaluate a 10B-level LLM for Asset Management"] --> B["Methodology: Continuous Pre-training & Fine-tuning"]
  B --> C{"Data Inputs: Targeted Corpus & Professional Exams"}
  C --> D["Computational Process: 10B Parameter Model Training"]
  D --> E["Innovative Evaluation Framework"]
  E --> F["Key Findings: Outperforms Baselines"]
  E --> G["Key Findings: Significant Performance with Modest Compute"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes Shai, a 10B‑parameter large language model tailored for asset management.

Source sentence
This paper introduces "Shai" a 10B level large language model specifically designed for the asset management industry

Assumes

No explicit assumptions found in the text.

Data

  • proprietary evaluation dataset benchmark proprietary
    Source sentence
    bring Shai into direct comparison with mainstream 10B-level open-source models, such as baichuan2[18], Qwen[19], InterLM[20], and Xverse[21], on our proprietary dataset.

Costs

Not applicable The paper reports no trading or execution result.

Finds 2

  • Shai achieves the best performance among mainstream 10B‑level open‑source LLMs in asset‑management tasks. performance
    Source sentence
    we are the first to build a 10B level LLM for asset management, which achieve the best performance comparing to the mainstream 10B-level LLMs.
  • Shai excels in financial examinations, practical tasks, and safety assessments, demonstrating high reliability for the asset‑management industry. performance
    Source sentence
    The results demonstrate that Shai excels not only in financial examinations and practical tasks but also in critical safety aspects, such as economic security and adherence to ethical standards.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 7 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (56)

  1. Baichuan 2: Open large-scale language models (2023) — link
  2. Qwen technical report (2023) — link
  3. Llama 2: Open foundation and fine-tuned chat models (2023) — link
  4. Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters (2023)
  5. GPT-4 Technical Report (2023) — link
  6. XVERSE-13B: A multilingual large language model developed by XVERSE Technology Inc. (2023)
  7. ChatML (2022)
  8. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2023) — link
and 48 more
  1. Attention is all you need (2017)
  2. Llama: Open and efficient foundation language models (2023) — link
  3. Palm 2 technical report (2023) — link
  4. Palm: Scaling language modeling with pathways (2022) — link
  5. Bloomberggpt: A large language model for finance (2023) — link
  6. Moss: Training conversational language models from synthetic data (2023) — link
  7. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (2023)
  8. Stanford alpaca: an instruction-following llama model (2023) (2023)
  9. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023) — link
  10. RefGPT: Reference-to-Dialogue by GPT and for GPT (2023)
  11. Agieval: A human-centric benchmark for evaluating foundation models (2023) — link
  12. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models (2023) — link
  13. Falcon-40B: an open large language model with state-of-the-art performance (2023)
  14. Internlm: A multilingual language model with progressively enhanced capabilities (2023)
  15. Glm-130b: An open bilingual pre-trained model (2022) — link
  16. Alpaca-CoT: An instruction-tuning platform with unified interface of instruction collection, parameter-efficient methods, and large language models, 2023
  17. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering (2022)
  18. Agieval: A human-centric benchmark for evaluating foundation models (2023) — link
  19. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models (2023) — link
  20. CMMLU: Measuring massive multitask language understanding in Chinese (2023) — link
  21. Evaluating the Performance of Large Language Models on GAOKAO Benchmark (2023) — link
  22. Measuring mathematical problem solving with the math dataset (2021) — link
  23. JEC-QA: a legal-domain question answering dataset (2020)
  24. Measuring massive multitask language understanding (2020) — link
  25. Training verifiers to solve math word problems (2021) — link
  26. BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark (2023) — link
  27. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering (2022) — link
  28. Duee-fin: A document-level event extraction dataset in the financial domain released by baidu (2021)
  29. Safetybench: Evaluating the safety of large language models with multiple choice questions (2023) — link
  30. Wider and deeper llm networks are fairer llm evaluators (2023) — link
  31. Prefix-tuning: Optimizing continuous prompts for generation (2021) — link
  32. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks (2022)
  33. Chain-of-thought prompting elicits reasoning in large language models (2022)
  34. Large language models are not fair evaluators (2023) — link
  35. A Wilcoxon-type test for trend (1985)
  36. Verbosity Bias in Preference Labeling by Large Language Models (2023) — link
  37. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023) — link
  38. Chatlaw: Open-source legal large language model with integrated external knowledge bases (2023) — link
  39. A Brief Report on LawGPT 1.0: A Virtual Legal Assistant Based on GPT-3 (2023) — link
  40. Lawyer LLaMA Technical Report (2023) — link
  41. Doctorglm: Fine-tuning your chinese doctor is not a herculean task (2023) — link
  42. Huatuo: Tuning llama model with chinese medical knowledge (2023) — link
  43. SoulChat: Improving LLMs’ Empathy, Listening, and Comfort Abilities through Fine-tuning with Multi-turn Empathy Conversations (2023) — link
  44. BianQue: Balancing the Questioning and Suggestion Ability of Health LLMs with Multi-turn Health Conversations Polished by ChatGPT (2023) — link
  45. FinGPT: Open-Source Financial Large Language Models — → 2306.06031
  46. DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning (2023) — link
  47. Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters (2023)
  48. Don’t stop pretraining: Adapt language models to domains and tasks (2020) — link

1 of 56 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.