Upcoming Seminar Presentations
All seminars take place Thursdays at 8:30 am PT / 11:30 am ET / 4:30 pm London / 6:30 pm Tel Aviv. Past seminar presentations are posted here.
Thursday, February 22, 2024 [Link to join]
Speaker: Nick Koning (Erasmus University)
Title: Post-hoc p-values
Abstract: A pervasive methodological error is the post-hoc interpretation of p-values. A p-value p is the smallest significance level at which we would have rejected the null had we chosen level p. It is not the smallest significance level at which we reject the null. We introduce post-hoc p-values, that do admit such a post-hoc interpretation. We show that p is a post-hoc p-value if and only if 1/p is an e-value, a recently introduced statistical object. The product of independent post-hoc p-values is a post-hoc p-value, making them easy to combine. Moreover, any post-hoc p-value can be trivially improved if we permit external randomization, but only (essentially) non-randomized post-hoc p-values can be arbitrarily merged through multiplication. In addition, we discuss what constitutes a `good' post-hoc p-value. Finally, we argue that post-hoc p-values eliminate the need of a pre-specified significance level, such as \alpha = .05 or \alpha = .005 Benjamin et al. (2018). We believe this may take away incentives for p-hacking and contribute to solving the file-drawer problem, as both these issues arise from using a pre-specified significance level.
Discussant: Ruodu Wang (University of Waterloo)
Links: [Relevant papers: paper #1]
Thursday, February 29, 2024 [Link to join]
Speaker: Livio Finos (University of Padova)
Title: Post-selection Inference in Multiverse Analysis (PIMA): an inferentialframework based on the sign flipping score test
Abstract: When analyzing data researchers make some decisions that are either arbitrary, based on subjective beliefs about the data generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused, and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that accounts for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e. pre-processing) and any generalized linear model; it provides strong control of the family-wise error rate such that it allows researchers to claim that the null hypothesis can be rejected for each specification that shows a significant effect. The inferential proposal is based on the sign-flip score test of Hemerik et al. (2020) and De Santis et al. (2022) that will be briefly introduced in the talk.
Discussant: Jesse Hemerik (Erasmus University Rotterdam)
Thursday, March 7, 2024 [Link to join]
Speaker: Guanxun Li (Texas A&M University)
Title: E-values, Multiple Testing and Beyond.
Abstract: We discover a connection between the Benjamini-Hochberg (BH) procedure and the recently proposed e-BH procedure [Wang and Ramdas, 2022] with a suitably defined set of e-values. This insight extends to a generalized version of the BH procedure and the model-free multiple testing procedure in Barber and Cand`es  (BC) with a general form of rejection rules. The connection provides an effective way of developing new multiple testing procedures by aggregating or assembling e-values resulting from the BH and BC procedures and their use in different subsets of the data. In particular, we propose new multiple testing methodologies in three applications, including a hybrid approach that integrates the BH and BC procedures, a multiple testing procedure aimed at ensuring a new notion of fairness by controlling both the group-wise and overall false discovery rates (FDR), and a structure adaptive multiple testing procedure that can incorporate external covariate information to boost detection power. One notable feature of the proposed methods is that we use a data-dependent approach for assigning weights to e-values, significantly enhancing the efficiency of the resulting e-BH procedure. The construction of the weights is non-trivial and is motivated by the leave-one-out analysis for the BH and BC procedures. In theory, we prove that the proposed e-BH procedures with data-dependent weights in the three applications ensure finite sample FDR control. Furthermore, we demonstrate the efficiency of the proposed methods through numerical studies in the three applications.
Discussant: Peter W. MacDonald (McGill University)
Links: [Relevant papers: paper #1]
Thursday, March 14, 2024 [Link to join]
Speaker: Lars van der Laan (University of Washington)
Title: Self-Consistent Conformal Prediction
Abstract: In decision-making guided by machine learning, decision-makers often take identical actions in contexts with identical predicted outcomes. Conformal prediction helps decision-makers quantify outcome uncertainty for actions, allowing for better risk management. Inspired by this perspective, we introduce self-consistent conformal prediction, which yields both Venn-Abers calibrated predictions and conformal prediction intervals that are valid conditional on actions prompted by model predictions. Our procedure can be applied post-hoc to any black-box predictor to provide rigorous, action-specific decision-making guarantees. Numerical experiments show our approach strikes a balance between interval efficiency and conditional validity.
Discussant: Tiffany Ding (UC Berkeley)
Links: [Relevant papers: paper #1]
Thursday, March 21, 2024 [Link to join]
Speaker: Etienne Roquain (Sorbonne Université)
Title: Selecting informative conformal prediction sets with false coverage rate control
Abstract: In supervised learning, including regression and classification, conformal methods provide distribution free prediction sets for the outcome/label with finite sample coverage for any machine learning predictors. We consider here the case where such prediction sets come after a selection process. The selection process requires that the selected prediction sets be informative in a well defined sense. We consider both the classification and regression settings where the analyst may consider as informative only the sample with prediction label sets or prediction intervals small enough, excluding a null value, or obeying other appropriate 'monotone' constraints. While this covers many settings of possible interest in various applications, we develop a unified framework for building such informative conformal prediction sets while controlling the false coverage rate (FCR) on the selected sample. This framework generalizes some recent results on selective conformal inference in the literature. We show the usefulness of our resulting procedures on real and simulated data.
Discussant: Zijun Gao (University of Southern California)
Links: [Relevant papers: ]
Thursday, March 28, 2024 [Link to join]
Speaker: Armeen Taeb (University of Washington)
Title: On False Positive Error
Abstract: Controlling the false positive error in model selection is a prominent paradigm for gathering evidence in data-driven science. In model selection problems such as variable selection and graph estimation, models are characterized by an underlying Boolean structure such as presence or absence of a variable or an edge. Therefore, false positive error or false negative error can be conveniently specified as the number of variables/edges that are incorrectly included or excluded in an estimated model. However, the increasing complexity of modern datasets has been accompanied by the use of sophisticated modeling paradigms in which defining false positive error is a significant challenge. For example, models specified by structures such as partitions (for clustering), permutations (for ranking), directed acyclic graphs (for causal inference), or subspaces (for principal components analysis) are not characterized by a simple Boolean logical structure, which leads to difficulties with formalizing and controlling false positive error. We present a generic approach to endow a collection of models with partial order structure, which leads to systematic approaches for defining natural generalizations of false positive error and methodology for controlling this error. (Joint work with Peter Bühlmann, Venkat Chandrasekaran, and Parikshit Shah)
Discussant: Peter Hansen (University of North Carolina, Chapel Hill)
Thursday, April 11, 2024 [Link to join]
Speaker: Wanteng Ma (Hong Kong University of Science and Technology)
Title: Multiple Testing of Linear Forms for Noisy Matrix Completion
Abstract: Many important tasks of large-scale recommender systems can be naturally cast as testing multiple linear forms for noisy matrix completion. These problems, however, present unique challenges because of the subtle bias-and-variance tradeoff and an intricate dependency among the estimated entries induced by the low-rank structure. Here, we develop a general approach to overcome these difficulties by introducing new statistics for individual tests with sharp asymptotics both marginally and jointly, and utilizing them to control the false discovery rate (FDR) via a data splitting and symmetric aggregation scheme. We show that valid FDR control can be achieved with guaranteed power under nearly optimal sample size requirements using the proposed methodology. Extensive numerical simulations and real data examples are also presented to further illustrate its practical merits.
Discussant: Yu Gui (University of Chicago)
Links: [Relevant papers: paper #1]
The seminars are held on Zoom and last 60 minutes:
45 minutes of presentation
15 minutes of discussion, led by an invited discussant
Moderators collect questions using the Q&A feature during the seminar.
How to join
You can attend by clicking the link to join (there is no need to register in advance).
More instructions for attendees can be found here.
What is selective inference?
Broadly construed, selective inference means searching for interesting patterns in data, usually with inferential guarantees that account for the search process. It encompasses:
Multiple testing: testing many hypotheses at once (and paying disproportionate attention to rejections)
Post-selection inference: examining the data to decide what question to ask, or what model to use, then carrying out one or more appropriate inferences
Adaptive / interactive inference: sequentially asking one question after another of the same data set, where each question is informed by the answers to preceding questions
Cheating: cherry-picking, double dipping, data snooping, data dredging, p-hacking, HARKing, and other low-down dirty rotten tricks; basically any of the above, but done wrong!