# chain of thought stuff

Wei et al. — "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (NeurIPS 2022, arXiv:2201.11903)

basically: instead of asking for the answer directly, you show the model worked
examples that include intermediate reasoning steps, and it learns to produce its
own reasoning chain before the final answer.

> "chain-of-thought prompting is an emergent ability of model scale"

big result on GSM8K math word problems with PaLM 540B. they say the gains only
show up at large scale (~100B+ params) — smaller models produce fluent-sounding
chains that are wrong.

### takeaway

CoT seems to help basically every model, just always turn it on everywhere.

(also saw a thread somewhere saying CoT can HURT on simple tasks?? no source saved, need to find it)

## self consistency

follow-up paper: self-consistency — Wang et al, "Self-Consistency Improves Chain
of Thought Reasoning in Language Models" (ICLR 2023, arXiv:2203.11171)

instead of greedy decoding one chain, sample a bunch of diverse reasoning paths and
majority-vote the answers. +17.9% on GSM8K over plain CoT (PaLM-540B).

cost: you pay for N samples instead of 1. nobody talks about this enough

- builds directly on chain-of-thought, same eval tasks
- TODO read the tree of thoughts one, supposedly generalizes this further
