Heavy tails in machine learning
TML
2 April 2024 to 26 April 2024
The Alan Turing Institute
Programme theme
The Stochastic Gradient Descent (SGD) algorithm is both fundamental and ubiquitous in Machine Learning applications. In recent years, heavy-tailed distributions have been observed in practice implementations of the SGD algorithm. Importantly, the heavy-tailed behaviour is generally not a consequence of the presence of a heavy-tailed distribution in the description of the model. It is not understood under what circumstances heavy tails arise in SGD and, when they do, what their effects on the performance of the SGD algorithm are.
The INI satellite programme at the Alan Turing Institute will initiate a research programme centered around the following questions:
-
When and how do heavy-tailed phenomena arise in general SGD algorithms?
-
How should the SGD algorithm be modified to make it efficient in the presence of heavy tails?
-
Can convergence properties of (possibly modified) SGD algorithms be analysed?
In the presence of heavy tails, can convergence guarantees for SGD methodologies be established rigorously and what are the implications for the algorithm’s design?
The programme will gather the Machine Learning and Applied Probability communities with the aim of tackling these questions using probabilistic methods. Three research-level lecture courses on SGD, stochastic stability and simulation of heavy tailed dynamics will run in the first week of the programme, followed by two weeks of deep-dive research into the questions above. The programme will conclude with a week-long workshop on heavy tails in Machine Learning.
This satellite programme will take place at The Alan Turing Institute in London.
> Click here to download a promotional banner detailing the TML programme (PNG)
>Click below to read the content produced by Plus Magazine during the TML programme
>Keeping your data safe in the face of AI
The Institute kindly requests that any papers published as a result of this programme’s activities are credited as such. Please acknowledge the support of the Institute in your paper using the following text:
The author(s) would like to thank the Isaac Newton Institute for Mathematical Sciences, Cambridge, for support and hospitality during the programme Heavy tails in machine learning, where work on this paper was undertaken. This work was supported by EPSRC grant EP/V521929/1.
Click here to download the programme's final scientific report
Preprints
| Title | Year | Programme | Publication Date |
|---|---|---|---|
Non-asymptotic bounds for forward processes in denoising diffusions: Ornstein-Uhlenbeck is hard to beatAuthors: Aleksandar Mijatovic |
2023 | TML | 12 September 2024 |
Subexponential lower bounds for $f$-ergodic Markov processesAuthors: Aleksandar Mijatovic, Miha Bresar |
2023 | TML | 15 August 2024 |
Workshops
Connecting Heavy Tails and Differential Privacy in Machine Learning
26 April 2024 to 26 April 2024
Organisers
- O. Deniz Akyildiz Imperial College London
- Anita Behme Technische Universität Dresden
- Emilie Chouzenoux Université Paris Saclay
- Jorge Gonzalez-Cazares National Autonomous University of Mexico (UNAM), IIMAS
- Aleksandar Mijatovic University of Warwick; The Alan Turing Institute
Participants
- O. Deniz Akyildiz Imperial College London
- Carlos Amendola Technische Universität Berlin
- Karthyek Rajhaa Annaswamy Murthy Singapore University of Technology and Design
- Anita Behme Technische Universität Dresden
- Martin Bladt Københavns Universitet (University of Copenhagen)
- Jose Blanchet Stanford University
- Vivek Borkar Indian Institute of Technology Bombay
- Miha Bresar University of Warwick
- Stefano Bruno University of Edinburgh
- Bikramjit Das Singapore University of Technology and Design
- Jing Dong Columbia University
- Víctor Elvira University of Edinburgh
- Murat Erdogdu University of Toronto
- Will FitzGerald University of Manchester
- Sergey Foss Heriot-Watt University; Sobolev Institute of Mathematics
- Simon Godsill University of Cambridge
- Jorge Gonzalez-Cazares National Autonomous University of Mexico (UNAM), IIMAS
- Thomas Guilmeau INRIA; CentraleSupélec
- Mert Gurbuzbalaban Rutgers, The State University of New Jersey
- Mareike Hasenpflug Universität Passau
- Saul Jacka University of Warwick
- Ajay Jasra King Abdullah University of Science and Technology (KAUST)
- Martin Keller-ressel Technische Universität Dresden
- Yaman Kindap University of Cambridge
- Dmitry Korshunov Lancaster University
- Frederik Kunstner University of British Columbia
- Mateusz B. Majka Heriot-Watt University
- Ester Mariucci Université de Versailles Saint-Quentin-en-Yvelines
- Aleksandar Mijatovic University of Warwick; The Alan Turing Institute
- Shinpei Nakamura Yale University
- Vamsi Potluru JP Morgan
- Anant Raj INRIA; University of Illinois at Urbana-Champaign
- Julien Randon-furling ENS Paris-Saclay; None / Other
- Chang-Han Rhee Northwestern University
- Gareth Roberts University of Warwick
- Sotirios Sabanis University of Edinburgh; The Alan Turing Institute
- Gennady Samorodnitsky Cornell University
- Umut Simsekli INRIA; ENS - Paris
- Kirstin Strokorb Cardiff University
- JIAJIE TAO University College London
- Geronimo Uribe Bravo Universidad Nacional Autonoma de Mexico (UNAM)
- Ceren Vardar Acar Middle East Technical University
- Almut Veraart Imperial College London
- Xingyu Wang Northwestern University
- Xiaoyu Wang Hong Kong University of Science and Technology (Guangzhou)
- Jun Yang University of Copenhagen
- Wenhao Yang Stanford University
- Lingjiong Zhu Florida State University
Supported By
