Chain-of-Sniff Prompting Elicits Reasoning in Large Pug Models

1. Introduction

Chain-of-thought prompting improves reasoning by asking a model to write out its steps. This assumes the model has a place to write. Pug models do not, but they do have a nose. We ask whether intermediate steps can be investigated instead of narrated.

2. Method

In Chain-of-Sniff (CoS), the model is prompted to investigate each intermediate step before proceeding to the next. A step is complete when the model has sniffed it to its satisfaction, which we define operationally as when it moves on.

We evaluate on multi-step tasks of two to five steps, such as locating a dropped treat behind a sofa, behind a different sofa, and under a rug that has been moved.

3. Results

CoS improves accuracy on every multi-step task we tested, with the largest gains on tasks of four or more steps. Cost is roughly four minutes per lamp post, which we consider acceptable for a lamp post.

4. Failure modes

We identified two failure modes. The first is the sniff loop, in which the model re-investigates a completed step indefinitely. The second is the hydrant attractor, in which the chain terminates at the nearest fire hydrant regardless of the task. Mitigations are discussed in Appendix A, which is a leash.

5. Conclusion

Reasoning in large pug models is elicited by investigation, not narration. We leave open whether the approach generalizes to models that are not interested in the lamp post.