# International Seminar on Selective Inference

A weekly online seminar on selective inference, multiple testing, and post-selection inference.

Gratefully inspired by the Online Causal Inference Seminar

## Upcoming Seminar Presentations

All seminars take place **Thursdays at 8:30 am PT / 11:30 am ET / 4:30 pm London / 6:30 pm Tel Aviv. **Past seminar presentations are posted here.

**Thursday, May 1****9****, 2022**[Link to join]**Speaker:**Daniel Wilhelm (University College London)**Title:**Inference for Ranks**Abstract:**It is often desired to rank different populations according to the value of some feature of each population. For example, it may be desired to rank neighborhoods according to some measure of intergenerational mobility or countries according to some measure of academic achievement. These rankings are invariably computed using estimates rather than the true values of these features. As a result, there may be considerable uncertainty concerning the rank of each population. In this paper, we consider the problem of accounting for such uncertainty by constructing confidence sets for the rank of each population. We consider both the problem of constructing marginal confidence sets for the rank of a particular population as well as simultaneous confidence sets for the ranks of all populations. We show how to construct such confidence sets under weak assumptions. An important feature of all of our constructions is that they remain computationally feasible even when the number of populations is very large. We apply our theoretical results to re-examine the rankings of both neighborhoods in the United States in terms of intergenerational mobility and developed countries in terms of academic achievement. The conclusions about which countries do best and worst at reading, math, and science are fairly robust to accounting for uncertainty. The confidence sets for the ranking of the 50 most populous commuting zones by measures of mobility are also found to be small. These rankings, however, become much less informative if one includes all commuting zones, if one considers neighborhoods at a more granular level (counties, Census tracts), or if one uses movers across areas to address concerns about selection.**Discussant:**Aldo Solari (University of Milano-Bicocca)

**Thursday,****May 26****, 2022**[Link to join]**Speaker:**James Leiner (Carnegie Mellon University)**Title:**Data blurring: sample splitting a single sample**Abstract:**Suppose we observe a random vector $X$ from some distribution $P$ in a known family with unknown parameters. We ask the following question: when is it possible to split $X$ into two parts $f(X)$ and $g(X)$ such that neither part is sufficient to reconstruct $X$ by itself, but both together can recover $X$ fully, and the joint distribution of $(f(X),g(X))$ is tractable? As one example, if $X=(X_1,\dots,X_n)$ and $P$ is a product distribution, then for any $m<n$, we can split the sample to define $f(X)=(X_1,\dots,X_m)$ and $g(X)=(X_{m+1},\dots,X_n)$. Rasines and Young (2021) offers an alternative route of accomplishing this task through randomization of $X$ with additive Gaussian noise which enables post-selection inference in finite samples for Gaussian distributed data and asymptotically for non-Gaussian additive models. In this paper, we offer a more general methodology for achieving such a split in finite samples by borrowing ideas from Bayesian inference to yield a (frequentist) solution that can be viewed as a continuous analog of data splitting. We call our method data blurring, as an alternative to data splitting, data carving and p-value masking. We exemplify the method on a few prototypical applications, such as post-selection inference for trend filtering and other regression problems.**Discussant:**Daniel Garcia Rasines (ICMAT - CSIC)**Links:**[Relevant papers: paper #1]

**Thursday,****June 2****, 2022**[Link to join]**Speaker:**Matteo Sesia (University of Southern California)**Title:**Individualized conditional independence testing under model-X with heterogeneous samples and interactions**Abstract:**Model-X knockoffs and the conditional randomization test are methods that search for conditional associations in large data sets, controlling the type-I errors if the joint distribution of the predictors is known. However, they cannot test for interactions nor find whether an association is only significant within a latent subset of a heterogeneous population. We address this limitation by developing an extension of the knockoff filter that tests conditional associations within automatically detected subsets of individuals, provably controlling the false discovery rate for the selected hypotheses. Then, under the additional assumption of a partially linear model with a binary predictor, we extend the conditional randomization test as to make inferences about quantiles of individual effects that are robust to sample heterogeneity and interactions. The performances of these methods are investigated through simulations and with the analysis of data from a randomized blood donation experiment with several treatments.**Discussant:**Brad Ross (Stanford University)**Links:**[Relevant papers: paper #1]

**Thursday, June****9****, 2022**[Link to join]**Speaker:**Anna Neufeld (University of Washington)**Title:**Inference after latent variable estimation for single-cell RNA sequencing data**Abstract:**In the analysis of single-cell RNA sequencing data, researchers often ﬁrst characterize the variation between cells by estimating a latent variable, representing some aspect of the individual cell’s state. They then test each gene for association with the estimated latent variable. If the same data are used for both of these steps, then standard methods for computing p-values and conﬁdence intervals in the second step will fail to achieve statistical guarantees such as Type 1 error control or nominal coverage. Furthermore, approaches such as sample splitting that can be fruitfully applied to solve similar problems in other settings are not applicable in this context. In this paper, we introduce count splitting, an extremely ﬂexible framework that allows us to carry out valid inference in this setting, for virtually any latent variable estimation technique and inference approach, under a Poisson assumption. We demonstrate the Type 1 error control and power of count splitting in a simulation study, and apply count splitting to a dataset of pluripotent stem cells diﬀerentiating to cardiomyocytes.**Discussant:**James Leiner (Carnegie Mellon University)**Links:**[Relevant papers: ]

**Thursday, June****16****, 2022 (****ISSI-STAMPS joint seminar****)**[Link to join]**Speaker:**Ann Lee (Carnegie Mellon University)**Title:**Likelihood-Free Frequentist Inference: Confidence Sets with Correct Conditional Coverage**Abstract:**Many areas of science make extensive use of computer simulators that implicitly encode likelihood functions of complex systems. Classical statistical methods are poorly suited for these so-called likelihood-free inference (LFI) settings, outside the asymptotic and low-dimensional regimes. Although new machine learning methods, such as normalizing flows, have revolutionized the sample efficiency and capacity of LFI methods, it remains an open question whether they produce confidence sets with correct conditional coverage. In this talk, I will describe our group's recent and ongoing research on developing scalable and modular procedures for (i) constructing Neyman confidence sets with finite-sample guarantees of nominal coverage, and for (ii) computing diagnostics that estimate conditional coverage over the entire parameter space. We refer to our framework as likelihood-free frequentist inference (LF2I). Any method that defines a test statistic, like the likelihood ratio, can be adapted to LF2I to create valid confidence sets and diagnostics, without costly Monte Carlo samples at fixed parameter settings. In my talk, I will discuss where we stand with LF2I and challenges that still remain. (Part of these efforts are joint with Niccolo Dalmasso, Rafael Izbicki, Luca Masserano, Tommaso Dorigo, Mikael Kuusela, and David Zhao. Our general framework is described in arXiv:2107.03920)**Discussant:**

**Thursday, June****23****, 2022**[Link to join]**Speaker:**Yixiang Luo (University of California, Berkeley)

**Thursday, June****30****, 2022**[Link to join]**Speaker:**Zhanrui Cai (Carnegie Mellon University)**Title:**Robust Cross Validation with Confidence**Abstract:**Cross validation is one of the most popular tools for model selection and tunning parameter selection in the modern statistics and machine learning community. By dividing the sample into K-folds, cross validation first train the models on $K-1$ folds of data, and test the prediction error on the remaining dataset. Then it chooses the model / tunning parameter that has the smallest test error. Recent studies aim to improve the confidence level for the models selected by cross validation (Lei, 2020), but may not be suitable for skewed/ heavy tailed data, or data with outliers. In this paper, we propose a robust cross validation method. Instead of comparing the mean of the prediction error, we propose to compare the quantiles of the test error due to its skewness nature. We illustrate the necessity of rank-sum comparison through motivating examples, and demonstrate the advantage of the proposed robust cross validation method through extensive simulation and real data analysis. In order to study the limiting distribution of the evaluation criterion, we develop the Gaussian approximation theory for high dimensional two sample U-statistics, which may be of independent interest.**Discussant:**Morgane Austern (Harvard University)**Links:**[Relevant papers: ]

**Thursday, July****14****, 2022 (****100-th ISSI****seminar****)**[Link to join]**Speaker:**Yoav Benjamini (Tel Aviv University)

**Thursday, July 2****1****, 2022**[Link to join]**Speaker:**Dacheng Xiu (University of Chicago)**Title:**Prediction When Factors are Weak

**Thursday, Ju****ly****28****, 2022**[Link to join]**Speaker:**Trambak Banerjee (University of Kansas)

## Format

The seminars are held on Zoom and last 60 minutes:

45 minutes of presentation

15 minutes of discussion, led by an invited discussant

Moderators collect questions using the Q&A feature during the seminar.

## How to join

You can attend by clicking the link to join (there is no need to register in advance).

More instructions for attendees can be found here.

## Organizers

Rina Barber (University of Chicago)

Will Fithian (UC Berkeley)

Lihua Lei (Stanford University)

Daniel Yekutieli (Tel Aviv University)

## Contact us

If you have feedback or suggestions or want to propose a speaker, please e-mail us at selectiveinferenceseminar@gmail.com.

## What is selective inference?

Broadly construed, * selective inference* means searching for interesting patterns in data, usually with inferential guarantees that account for the search process. It encompasses:

**Multiple testing:**testing many hypotheses at once (and paying disproportionate attention to rejections)**Post-selection inference:**examining the data to decide what question to ask, or what model to use, then carrying out one or more appropriate inferences**Adaptive / interactive inference:**sequentially asking one question after another of the same data set, where each question is informed by the answers to preceding questions**Cheating:**cherry-picking, double dipping, data snooping, data dredging, p-hacking, HARKing, and other low-down dirty rotten tricks; basically any of the above, but done wrong!