Stylistics for CoTs

Can we evaluate the quality of a CoT?

We want concise, interpretable CoTs. We want to keep track of this during LM development

We need to show: (1) better CoTs = better models, (2) we have an eval to measure ""goodness"" of a CoT

Notes:

I built a sentence-classifier for CoTs here: https://github.com/davidheineman/traces. It creates some interesting looking CoT distribution figures.

I was also looking at tagging this SFT dataset of CoTs (https://huggingface.co/datasets/Mingyin0312/Genome-Bench/viewer/default/train) and I feel like there’s some dual-constraint with RLVR — You want both correct tasks but also for them to have the same stylistics that we care about in chat tasks.