International Seminar on Selective Inference

A weekly online seminar on selective inference, multiple testing, and post-selection inference.

Gratefully inspired by the Online Causal Inference Seminar

Mailing List

For announcements and Zoom invitations please subscribe to our mailing list.

Upcoming Seminar Presentations

All seminars take place Thursdays at 8:30 am PT / 11:30 am ET / 4:30 pm London / 6:30 pm Tel Aviv. Past seminar presentations are posted here.

• The ISSI is on hiatus until September 30, 2021

• Thursday, September 30, 2021 [Recording]

  • Speaker: Pallavi Basu (Indian School of Business)

  • Title: Empirical Bayes Control of the False Discovery Exceedance

  • Abstract: We propose an empirical Bayes procedure that guarantees control of the False Discovery eXceedance (FDX) by ranking and thresholding hypotheses based on their local false discovery rate (lfdr) test statistic. In a two-group independent model or Gaussian with exchangeable hypotheses, we show that ranking by the lfdr delivers the ``optimal'' ranking for FDX control. We propose a computationally efficient procedure that does not empirically lose validity and power and illustrate its properties by analyzing two million stock trading strategies.

Joint work with Luella Fu, Alessio Saretto, and Wenguang Sun.

• • Discussant: Sebastian Döhler (Darmstadt University of Applied Sciences)

  • Links: [Relevant papers:]

• Thursday, October 7, 2021 [Recording]

  • Speaker: Kenneth Hung (Facebook)

  • Title: Statistical Methods for Replicability Assessment

  • Abstract: Large-scale replication studies like the Reproducibility Project: Psychology (RP:P) provide invaluable systematic data on scientific replicability, but most analyses and interpretations of the data fail to agree on the definition of “replicability” and disentangle the inexorable consequences of known selection bias from competing explanations. We discuss three concrete definitions of replicability based on: (1) whether published findings about the signs of effects are mostly correct, (2) how effective replication studies are in reproducing whatever true effect size was present in the original experiment and (3) whether true effect sizes tend to diminish in replication. We apply techniques from multiple testing and postselection inference to develop new methods that answer these questions while explicitly accounting for selection bias. Our analyses suggest that the RP:P dataset is largely consistent with publication bias due to selection of significant effects. The methods in this paper make no distributional assumptions about the true effect sizes.

  • Discussant: Marcel van Assen (Tilburg University)

  • Links: [Relevant papers: paper #1]

• Thursday, October 14, 2021 [Recording]

  • Speaker: Byol Kim (University of Chicago)

  • Title: Predictive inference is free with the jackknife+-after-bootstrap

  • Abstract: Ensemble learning is widely used in applications to make predictions in complex decision problems --- for example, averaging models fitted to a sequence of samples bootstrapped from the available training data. While such methods offer more accurate, stable, and robust predictions and model estimates, much less is known about how to perform valid, assumption-lean inference on the output of these types of procedures. In this paper, we propose the jackknife+-after-bootstrap (J+aB), a procedure for constructing a predictive interval, which uses only the available bootstrapped samples and their corresponding fitted models, and is therefore "free" in terms of the cost of model fitting. The J+aB offers a predictive coverage guarantee that holds with no assumptions on the distribution of the data, the nature of the fitted model, or the way in which the ensemble of models are aggregated --- at worst, the failure rate of the predictive interval is inflated by a factor of 2. Our numerical experiments verify the coverage and accuracy of the resulting predictive intervals on real data. This work is joint with Chen Xu and Rina Foygel Barber.

  • Discussant: Yachong Yang (University of Pennsylvania)

  • Links: [Relevant papers: paper #1]

• Thursday, October 21, 2021 [Recording]

  • Speaker: Yao Zhang (University of Cambridge)

  • Title: Multiple conditional randomization tests

  • Abstract: We propose a general framework for (multiple) conditional randomization tests that incorporate several important ideas in the recent literature. We establish a general sufficient condition on the construction of multiple conditional randomization tests under which their p-values are "independent", in the sense that their joint distribution stochastically dominates the product of uniform distributions under the null. Conceptually, we argue that randomization should be understood as the mode of inference precisely based on randomization. We show that under a change of perspective, many existing statistical methods, including permutation tests for (conditional) independence and conformal prediction, are special cases of the general conditional randomization test. The versatility of our framework is further illustrated with an example concerning lagged treatment effects in stepped-wedge randomized trials.

  • Discussant: Panos Toulis (University of Chicago)

  • Links: [Relevant papers: paper #1]

• Thursday, October 28, 2021 [Recording]

  • Speaker: Chiara Sabatti (Stanford University)

  • Title: Searching for consistent associations with a multi-environment knockoff filter

  • Abstract: This paper develops a method based on model-X knockoffs to find conditional associations that are consistent across diverse environments, controlling the false discovery rate. The motivation for this problem is that large data sets may contain numerous associations that are statistically significant and yet misleading, as they are induced by confounders or sampling imperfections. However, associations consistently replicated under different conditions may be more interesting. In fact, consistency sometimes provably leads to valid causal inferences even if conditional associations do not. While the proposed method is flexible and can be deployed in a wide range of applications, this paper highlights its relevance to genome-wide association studies, in which consistency across populations with diverse ancestries mitigates confounding due to unmeasured variants. The effectiveness of this approach is demonstrated by simulations and applications to the UK Biobank data.

  • Discussant:

  • Links: [Relevant papers: paper #1]

• Thursday, December 2, 2021 [Recording]

  • Speaker: Daniel Garcia Rasines (Imperial College London)

  • Title: TBD

• Thursday, December 16, 2021 [Recording]

  • Speaker: Marina Bogomolov (Technion - Israel Institute of Technology)

  • Title: TBD

What is selective inference?

Broadly construed, selective inference means searching for interesting patterns in data, usually with inferential guarantees that account for the search process. It encompasses:

• Multiple testing: testing many hypotheses at once (and paying disproportionate attention to rejections)

• Post-selection inference: examining the data to decide what question to ask, or what model to use, then carrying out one or more appropriate inferences

• Adaptive / interactive inference: sequentially asking one question after another of the same data set, where each question is informed by the answers to preceding questions

• Cheating: cherry-picking, double dipping, data snooping, data dredging, p-hacking, HARKing, and other low-down dirty rotten tricks; basically any of the above, but done wrong!