Zero-shot & multi-task
Shared context. Multiple answers.
Give a language model one document and several questions or requested outputs. Return the answers together in one inference pass.
Project description
From the original project page. Results and time references describe that version of the work.
Recent works leveraging the in-context learning capabilities of modern large language models (LLMs) have independently proposed variations of a novel technique: Batch Prompting (BP), wherein b question prompts are indexed and concatenated into a single BatchPrompt. The queried LLM then learns from batch-formatted in-context examples how to answer all b questions in one generation. In this work, we consider sufficient conditions of a BatchPrompt in order to explore the technique’s efficacy in the zero-shot and multitask scenarios. We present new BatchPrompt templates and demonstrate on both open- and closed-source models of varying size that BP is possible without few-shot exemplars and also robust to in-batch task diversity provided sufficient instructions. We conclude with a modification to the token efficiency metric η proposed by the original work and a discussion of which regimes of NLG are best suited to the technique.
The shared context
A document can support several questions or tasks. Rather than send that context repeatedly, one prompt contains the document and all requested outputs. The model produces indexed answers in a single generation. The visualization shows this shared-document pattern; the original research also explores batching across tasks.
The question
Does batch prompting require worked examples, and can the questions in a batch come from different tasks?
The approach
Index and combine questions into one prompt with instructions for corresponding answers. The project explores zero-shot templates and task diversity across open- and closed-source models.
The finding
The reported experiments support batching without few-shot examples and across task types when instructions are sufficient. The work also revisits how token efficiency is measured.
The tradeoff
Token efficiency and task quality need to be evaluated together. The illustration shows request organization, not measured savings or live inference.