Homepage of Aditya Krishna Menon

Staff Research Scientist

Google

New York, NY

Email: adityakmenon followed by google.com

Biography

I'm a research scientist at Google. I work on machine learning and its applications.

I completed my honours in Computer Science from the University of Sydney in 2006 under Sanjay Chawla. I completed my PhD in Computer Science from UC San Diego in 2013 under Charles Elkan. From 2013 - 2018, I held positions at NICTA, CSIRO Data61, and the Australian National University, working with Bob Williamson and other inimitable colleagues.

Here is a copy of my CV.

Research interests

I am broadly interested in the design and analysis of machine learning algorithms for (weakly-) supervised learning problems occurring in practice. Current areas of interest include:

Efficient inference for large (language) models (e.g., knowledge distillation, model cascading)
Retrieval and re-ranking (e.g., negative mining, loss function design)
Foundations of (weakly-)supervised learning (e.g. class-probability estimation, bipartite ranking)

I have also worked on (and remain interested in) long-tail learning, learning from label noise, and collaborative filtering.

Selected publications

Below are a few representative publications. For a full list, see here.

Inspired by Bernard Chazelle's wonderful idea of "liner notes" for his papers, I've included some of my own for a few papers. (Liner notes are not peer-, or even coauthor-reviewed.)

Post-hoc estimators for learning to defer to an expert.
Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar.
In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[pdf]
A statistical perspective on distillation.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar.
In International Conference on Machine Learning (ICML), 2021.
[pdf]
Long-tail learning via logit adjustment.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar.
In International Conference on Learning Representations (ICLR), 2021.
[pdf]
Fairness risk measures.
Robert C. Williamson and Aditya Krishna Menon.
In International Conference on Machine Learning (ICML), Long Beach, 2019.
[pdf]
Linking losses for density ratio and class-probability estimation.
Aditya Krishna Menon and Cheng Soon Ong.
In International Conference on Machine Learning (ICML), New York City, 2016.
[pdf] [slides] [poster] [code] [liner notes]

Long before being introduced to proper losses, I'd known about, and completely accepted, the basic premise as to why density ratio was different to class-probability estimation: the later only indirectly modelled the ratio, which is generally a sub-optimal strategy. Sometime in late 2015, I re-read the cool work on LSIF. This time, with link functions and partial losses fresh in my mind, I noticed that one could think of this as involving a particular proper loss plus link function. This was mildly interesting, but the point remained: using anything but the link that directly yields the density ratio must be a bad idea.

This was something of a defeat for my running theory at the time, that most everything can be attacked with some version of logistic regression. To better understand this failure mode, I was looking at Sugiyama's elegant unified Bregman view of density ratios. I noticed that logistic regression was mentioned, but didn't pay much attention to it; after all, the Bregman view of regret under proper losses immediately led to the KL minimisation view of logistic regression.

It was only on a re-read that I realised there was a bit more to this innocuous result: it was a statement of Bregman minimisation not to the true probability, but to the true density ratio. Now that I didn't know was true for logistic regression. Studying the surprisingly simple proof, I saw it could be viewed as a surprising equality between the KL divergence on probabilities and on ratios.

I was sure this couldn't be true in general; but why? I worked through the case of a general divergence, trying to find out the line where the proof would break.

Half an hour later, after staring at my working multiple times, I was still incredulous that everything seemed to work in general, too. And so was Lemma 2 born. [hide]