Paper: arXiv 2311.13326

Authors: Woosung Koh, Insu Choi, Yuntae Jang, Gimin Kang, Woo Chang Kim

Abstract

Curriculum learning and imitation learning have been leveraged extensively in the robotics domain. However, minimal research has been done on leveraging these ideas on control tasks over highly stochastic time-series data. Here, we theoretically and empirically explore these approaches in a representative control task over complex time-series data. We implement the fundamental ideas of curriculum learning via data augmentation, while imitation learning is implemented via policy distillation from an oracle. Our findings reveal that curriculum learning should be considered a novel direction in improving control-task performance over complex time-series. Our ample random-seed out-sample empirics and ablation studies are highly encouraging for curriculum learning for time-series control. These findings are especially encouraging as we tune all overlapping hyperparameters on the baseline – giving an advantage to the baseline. On the other hand, we find that imitation learning should be used with caution.

Complexity vs Empirical Score

  • Math Complexity: 6.5/10
  • Empirical Rigor: 6.0/10 (capped from 8.0: transaction costs not modelled; see penalty flags)
  • Quadrant: Holy Grail — high math complexity, high empirical rigor (effective empirical rigor capped from 8.0)

Why this score: The paper utilizes advanced mathematical frameworks including Markov Decision Processes, signal/noise decomposition, and reinforcement learning theory, while also featuring substantial empirical validation through out-of-sample testing, ablation studies, and hyperparameter tuning on multiple datasets.

Research Flowchart

  flowchart TD
  A["Research Goal<br>Model-Free Control on Financial Time-Series<br>Using Curriculum & Imitation Learning"] --> B["Methodology<br>Data Augmentation (Curriculum)<br>Policy Distillation (Imitation)"]
  B --> C["Input Data<br>Complex Financial Time-Series"]
  C --> D["Computational Process<br>Model-Free Control via RL"]
  D --> E["Comparative Analysis<br>Curriculum vs. Imitation vs. Baseline"]
  E --> F["Key Findings<br>Curriculum Learning improves performance<br>Imitation Learning requires caution"]
  F --> G["Outcome<br>Novel Direction for Time-Series Control"]

Evidence Card

What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card

Proposes

The paper proposes applying curriculum learning via data augmentation and imitation learning via policy distillation from an oracle to improve control tasks on financial time-series.

Source sentence
We implement the fundamental ideas of curriculum learning via data augmentation, while imitation learning is implemented via policy distillation from an oracle.

Assumes 3

  • The training data is a fixed sampled approximation to the data-generating process and cannot be further sampled without the passage of time. data
    Source sentence
    the training data ˆpdata, which is a sampled approximation to pdata, is fixed and can not be further sampled without the passage of time.
  • Portfolio exposure is constrained: max aggregate gross exposure of 1 (net [-1,1]) for Data Set 1 and 2 (net [-2,2]) for Data Set 2. execution
    Source sentence
    Constraint 1 (Data Set 1): The portfolio’s maximum aggregate gross exposure is 1. Equivalently, the net exposure is bound by [-1, 1]. Constraint 2 (Data Set 2): The portfolio’s maximum aggregate gross exposure is 2. Equivalently, the net exposure is bound by [-2, 2].
  • All overlapping hyperparameters are tuned for the baseline algorithm, giving the baseline an advantage. model
    Source sentence
    These findings are especially encouraging as we tune all overlapping hyperparameters on the baseline -- giving an advantage to the baseline.

Data

  • Data Set 1 macro unknown
    Source sentence
    We hereforth refer to Data Sets 1 and 2 as representing the Macro ETFs and Commodity Futures environments in Table 1, respectively.
  • Data Set 2 other unknown
    Source sentence
    We hereforth refer to Data Sets 1 and 2 as representing the Macro ETFs and Commodity Futures environments in Table 1, respectively.

Costs

Not modeled A trading or execution result is reported without costs.

Finds 2

  • Curriculum learning methods (EMA5, IS8, TIS) achieve the best performance across all empirical environments, with EMA5 attaining statistical significance (<0.05 p‑value) in all six cases. performance
    Source sentence
    each best result is the EMA-based CL we propose—{EMA5, IS8, TIS}. Consistent with expectations, the naïve rounding approach’s performance significantly lags the EMA-based approaches.
  • Imitation learning approaches (A2C‑DPD, A2C‑DPD‑LGN, etc.) dramatically worsen performance, often failing to take any position and yielding near‑flat test‑set results. negative
    Source sentence
    The most extreme case can be viewed in Figure 11 where A2C-DPD and A2C-DPD-LGN refuses to take any position at all, resulting in a nearly flat test set performance.

Checked by others

No other paper in the archive has been found to support or contradict these findings (compared with 16 related papers). Coverage is partial, so this is neither replication nor refutation.

Code

No repository by the authors found in the paper.

Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.

Cited References (37)

  1. SAR: Generalization of Physiological Dexterity via Synergistic Action Representation (2023) — link
  2. Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023) — link
  3. Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrations (2023) — link
  4. Learning Reward Functions from a Combination of Demonstration and Evaluative Feedback (2022) — link
  5. Recent advances in reinforcement learning in finance (2021) — link
  6. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning (2021) — link
  7. A Survey on Curriculum Learning (2021) — link
  8. Portfolio optimization with return prediction using deep learning and machine learning (2021) — link
and 29 more
  1. Universal Trading for Order Execution with Oracle Policy Distillation (2021) — link
  2. Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals (2020) — link
  3. Learning perturbation sets for robust machine learning (2020) — link
  4. Collective Decision Making under Incomplete Knowledge: Possible and Necessary Solutions (2020) — link
  5. Recent Advances in Robot Learning from Demonstration (2020) — link
  6. Adaptive Quantitative Trading: An Imitative Deep Reinforcement Learning Approach (2020) — link
  7. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020) — link
  8. Universal Adversarial Perturbation via Prior Driven Uncertainty Approximation (2019) — link
  9. Curriculum Learning in Deep Neural Networks for Financial Forecasting (2019) — link
  10. On The Power of Curriculum Learning in Training Deep Networks (2019) — link
  11. Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly (2019) — link
  12. Algorithmic financial trading with deep convolutional neural networks: Time series to image conversion approach (2018) — link
  13. QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018) — link
  14. A deep learning based stock trading model with 2-D CNN trend detection (2017) — link
  15. Proximal Policy Optimization Algorithms (2017) — link
  16. Deep Direct Reinforcement Learning for Financial Signal Representation and Trading (2017) — link
  17. Asynchronous Methods for Deep Reinforcement Learning (2016) — link
  18. Implementing deep neural networks for financial market prediction on the Intel Xeon Phi (2015) — link
  19. Trust Region Policy Optimization (2015) — link
  20. Long Short-Term Memory (1997) — link
  21. Random Walks in Stock Market Prices (1965) — link
  22. Scaling Up Multi-Task Robotic Reinforcement Learning (2021)
  23. The $100 trillion machine (2021)
  24. A Hybrid Machine Learning and Dynamic Nonlinear Framework for Determination of Optimum Portfolio Structure (2019) — link
  25. Author manuscript, published in “American Control Conference (2012)” Model-Free Reinforcement Learning with Continuous Action in Practice (2012)
  26. Universal Portfolios (1996) — link
    1. Machine learning port-folio allocation
  27. Expert Systems With Applications
  28. International conference on machine learning , 1928–1937

0 of 37 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.