{"paper_id":"2412.15298","title":"A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluation","authors":["Bhaskarjit Sarmah","Kriti Dutta","Anna Grigoryan","Sachin Tiwari","Stefano Pasquali","Dhagash Mehta"],"paper_date":"2024-12-19","paper_type":"empirical","paper_url":"https://arxiv.org/pdf/2412.15298v1","page_url":"https://thequant.space/flowcharts/a-comparative-study-of-dspy-teleprompter-algorithms-for-alig/","scores":{"empirical_rigor":7.5,"math_complexity":4,"quadrant":"Street Traders"},"proposes":{"status":"available","statement":"The paper proposes a comparative analysis of five DSPy teleprompter algorithms to optimize prompts for hallucination detection aligned with human annotations.","quote":"We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO), Multi-Stage Instruction Prompt Optimization (MIPRO), BootstrapFewShot, BootstrapFewShot with Optuna, and K-Nearest Neighbor Few Shot, within the DSPy framework with respect to their ability to align with ","summary":"We argue that the Declarative Self-improving Python (DSPy) optimizers are a way to align the large language model (LLM) prompts and their evaluations to the human annotations. We present a comparative analysis of five teleprompter algorithms, namely, Cooperative Prompt Optimization (COPRO)…","strategy":null},"assumptions":{"status":"none_found","items":[]},"datasets":{"status":"available","items":[{"name":"HaluBench","provider":"HuggingFace (PatronusAI)","kind":"benchmark","access":"public","period":null,"universe":null,"quote":"dataset to evaluate the performance of all DSPy teleprompter algorithms. HaluBench is an open-source hallucination evaluation benchmark dataset comprising around 15K Context-Question-Answer triplets as well as human-annotated ground-truth labels ’Pass’ (i.e., not hallucinated) and ’Fail’ (hallucinated)."}],"known":[],"sample":{"start":null,"end":null,"market":null,"asset_class":null,"frequency":null},"data_resolution":"none"},"transaction_costs":{"status":"not_applicable","modeled":"n/a","headline_net_of_costs":null,"detail":null,"quote":null,"source":"front_matter","rule":"R3","candidates":{"front_matter":"n/a","evidence_card":"n/a","evidence_card_quoted":false},"meaning":"the paper reports no trading or execution result"},"results":{"status":"available","evaluation_type":"forecast-accuracy","findings":[{"finding":"Baseline GPT-4o achieves reasonable overall performance but suffers from low macro F1 scores, indicating poor sensitivity to minority classes.","kind":"performance","quote":"The results demonstrate that while baseline GPT-4o achieves reasonable performance, it struggles with class imbalance, as evidenced by its low macro F1 scores."},{"finding":"Optimizers such as Bootstrap Few Shot Random Search and MIPROv2 significantly improve hallucination detection, especially on structured datasets.","kind":"performance","quote":"Optimizers such as Bootstrap Few Shot Random Search and MIPROv2 effectively address these limitations, demonstrating significant advancements in hallucination detection. Their performance is particularly pronounced in structured datasets, where clear patterns enable optimization techniques to achieve robust results."},{"finding":"COPRO and KNN Few Shot perform poorly on FinanceBench due to variable dataset structure.","kind":"performance","quote":"Prompt-Specific Adaptations: COPRO and KNN Few Shot struggled with datasets like FinanceBench, where prompt-specific adaptations were less effective due to the dataset’s variable structure."},{"finding":"Bootstrap Few Shot with Random Search improves minority class detection by balancing class-specific F1 scores.","kind":"performance","quote":"Class-Level Balance: Optimizers such as Bootstrap Few Shot with Random Search and with Optuna improved minority class detection by balancing class-specific F1 scores."},{"finding":"MIPROv2 achieved the best Weighted F1 score by leveraging structured learning and batching.","kind":"performance","quote":"Batch and Structured Learning: MIPROv2 achieved the best Weighted F1 score by leveraging structured learning techniques and batching."}],"reported_performance":null,"out_of_sample":{"method":"holdout","headline_out_of_sample":null}},"independent_evidence":{"status":"none_found","supports":[],"contradicts":[],"qualifies":[],"compared_with":33,"citations":{"cited_by_in_corpus":0,"by_independent_authors":0,"how_cited":{"used_as_baseline":0,"uses_method_or_data":0,"extends":0,"discusses_findings":0,"mentions":0,"unclassified":0},"independent_papers":[],"scope":"Citations between papers in this corpus only; citations from outside it are not counted."}},"implementation":{"status":"available","public":true,"code_url":"https://github.com/stanfordnlp/dspy","note":null},"replication":{"status":"none_found","card":null,"note":"No replication of this paper has been run by thequant.space yet."},"extraction":{"text_source":"fulltext","model":"openai/gpt-oss-120b","version":1,"extracted_at":"2026-11-14T17:48:49Z"},"provenance":{"claims":"Everything here is what the authors state in the paper. Nothing was re-run or reproduced by the platform.","quotes":"Assumptions, datasets, cost detail and findings were extracted by a language model. Every quote is verbatim paper text. Where an item has `support`, \"full\" means every figure in it appears in the quoted sentences and \"partial\" means a figure is stated elsewhere in the paper; items with a figure found nowhere in the paper were discarded.","datasets_known":"datasets.known lists well-known vendors and public datasets by a stable id, found by a dictionary over the full text where the sentence says the data is used; it is independent of the model and covers only names in the dictionary.","flags":"transaction_costs.modeled, results.evaluation_type and results.out_of_sample.method are platform classifications (structured-decision models); see https://thequant.space/score-guide/#penalty-flags","comparisons":"supports / contradicts / qualifies are a language model's reading of two papers' quoted findings, restricted to closely related papers that share no author. They are pointers to read, not verified replications, and coverage is partial.","implementation":"code_url is a repository the paper itself presents as its own (rule-scored, ambiguous cases judged by a model), checked for liveness when it was recorded."},"docs":"https://thequant.space/agents/"}