After four years, the Austin ML Journal Club has concluded. There will be no further sessions.
The club began in October 2022 as a small group of Austin-based ML practitioners who wanted to read papers closely to understand how “the sausages get made”. It later moved to a virtual format to include participants outside Austin. Over four years we ran 22 sessions under the Chatham House Rule, covering multi-agent systems, reasoning claims, evaluation methodology, and a number of papers whose headline findings did not survive a careful look at their results.
That last part was important to us, and we often asked critical questions such as:
- Do the measures actually support the claims?
- Were the annotations or judges validated?
- What is the benchmark confounded with?
- Is the reported phenomenon genuinely emergent or reducible to something simpler?
Those questions returned verdicts. Emergent abilities tracked the choice of evaluation metric rather than anything about the model. MAST’s 14 multi-agent failure modes were, by the paper’s own prevalence numbers, mostly single-agent pathologies in multi-agent dress. The Illusion of Thinking drew its easy, moderate, and hard regimes post-hoc from the very model performance they were meant to explain. And the multi-turn collapse was reported through two metrics that, on binary-scored tasks with ten runs, nearly collapse into each other. Asking these of well-regarded papers was more useful than chasing whatever was new that month. None of those questions required the paper to be bad, or in anyone’s subfield. They required reading the papers carefully, which is most of what a journal club is for.
Everything the club produced stays here. The session write-ups, the reading list, and the repository remain online.
Thanks to everyone who showed up on a weekday evening. Particular thanks to Anil Kamat, Satish K C, Athula Pudhiyidath, Ivan Perez Avellaneda, Joel Afriyie, Brian King, Kshitij Aggarwal, Meghann Agarwal, Saina Lajevardi, and Akshata Mohan.
Hongsup