Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenization
Current tokenization methods process sequential data without accounting for signal quality, limiting their effectiveness on noisy real-world corpora. We present QA-Token (Quality-Aware Tokenization), which incorporates data reliability directly into vocabulary construction. We make three key contrib