On rank-based effectiveness measures and optimisation

Many current retrieval models and scoring functions contain free parameters which need to be set – ideally, optimised. The process of optimisation normally involves some training corpus of the usual document-query-relevance judgement type, and some choice of measure that is to be optimised. The paper proposes a way to think about the process of exploring the space of parameter values, and how moving around in this space might be expected to affect different measures. One result, concerning local optima, is demonstrated for a range of rank-based evaluation measures.