Draft2026Expectations and forecasting

Whom to Trust, and When? Aggregating Probability Forecasts in Small Crowds with Short Track Records

Niklas Wallmeier

30 seconds from the problem to the result: why organizations need few people, the simple average and patience with track records.

In short

Should organizations weight forecasters by their track record or take the simple average? A model and 5,898 forecasts by 177 forecasters show that with small crowds and short records, the simple average is hard to beat.

Whom to trust, and when? Try it for your crowd

Averaging pays off fast: two forecasters already capture half of what averaging can achieve, five about 80 percent.
On the platform studied, a record of K resolved questions reflects skill only to the share K / (K + 63). After 12 questions, about 16 percent of the differences between track records are skill, the rest is luck.
Weighting by track record starts to pay only after a critical record length, about 50 questions in a crowd of five. Until then, the simple average is hard to beat.
Weights by raw past accuracy, without shrinking them towards equal weights, need about twice that record length before they help.

People in the crowd 5

Resolved questions per person 12

80%of what averaging can achieve, from this many people

16%of the differences in track records reflect skill rather than luck

19%weight the record deserves; equal weights get the rest

Model of the paper with K₀ = 63, estimated from the platform's data: a record of K resolved questions reflects skill to the share K / (K + 63). Crowds with more varied skill learn faster. Unshrunk weights by past accuracy need about twice the critical record length before they pay (Appendix A).

Cite

Niklas Wallmeier (2026). “Whom to Trust, and When? Aggregating Probability Forecasts in Small Crowds with Short Track Records.” Work in progress.

@unpublished{wallmeier2026whom,
  author = {Wallmeier, Niklas},
  title = {{Whom to Trust, and When? Aggregating Probability Forecasts in Small Crowds with Short Track Records}},
  year = {2026},
  note = {Work in progress}
}