When AI judges AI: the benchmark problem

Today’s most important AI news

Sunday closed a week marked by two tensions that define the ecosystem’s current moment: the fragility of the systems we use to measure AI progress, and the effort to bring advanced models to users who don’t know how to code. There were no big launches, but there were signals worth reading carefully.

A $25,000 prize nobody should have won

Google DeepMind and Kaggle organized a challenge called “Measuring Progress Toward AGI - Cognitive Abilities”, whose goal was for participants to design new benchmarks based on cognitive science to evaluate AI capabilities. This week they announced the grand-prize winner: $25,000 USD. The problem is that several members of the community analyzed the winning work and published evidence that it’s what’s known as AI slop: AI-generated content that sounds coherent but lacks real substance. According to the analysis published on r/MachineLearning, the winning work is essentially “a machine for generating meaningless numbers” backed by unfounded claims.

What makes this particularly serious isn’t just that someone won money with questionable work. The underlying problem is that the challenge had a critical purpose: improving how we measure whether AI is progressing toward general intelligence. If the judges of that process can’t distinguish a rigorous analysis from one superficially generated with AI, then we have a circular problem: we’re using broken tools to measure whether the tools are working.

For product managers and executives who make decisions based on AI benchmarks, this case is a practical warning. The rankings and awards that appear in papers and competitions aren’t a guarantee of quality. The next time someone on your team cites a benchmark to justify adopting a model, it’s worth asking: who designed that benchmark and under what criteria was it evaluated?

Spreadsheet predictions without writing a line of code

A developer published TabFM Studio, a web application that lets you run tabular foundation models — currently Google’s TabFM — directly on CSV or Excel files, with no code needed. The flow is as simple as dragging a file, clicking the column you want to predict and pressing a button. Rows with existing data are used as in-context examples, and empty rows receive predictions on the same grid.

Foundation models for tabular data are a category that has matured significantly over the last two years. Unlike LLMs, these models are optimized for the kind of structured data that lives in spreadsheets: sales, inventory, customer churn, operational data. Google’s TabFM can make useful predictions with few examples without prior training, which makes it a potentially powerful tool for business teams.

The practical implication is direct: if you have analysts, operators or sales teams who work in Excel and today depend on someone from data building them a model, TabFM Studio can close that gap. It won’t replace a custom-trained model for critical cases, but for quick exploration or low-risk predictions in non-technical teams, it’s worth trying. The repository is available on GitHub.

Fine-tuning visual models at scale with NVIDIA and Hugging Face

NVIDIA and Hugging Face published a guide to fine-tuning video and image models at scale using NeMo Automodel integrated with the Diffusers library. The proposal combines NVIDIA’s distributed training infrastructure with Hugging Face’s model ecosystem, aimed at teams that need to customize generative visual models without building their own infrastructure from scratch.

For teams working with image or video generation in production, this kind of integration solves one of the area’s most concrete problems: scaling the fine-tuning of diffusion models requires coordinating multiple GPUs, managing memory efficiently and maintaining compatibility between frameworks. NeMo Automodel abstracts away much of that complexity, and the fact that it’s integrated with Diffusers means the resulting models are directly compatible with the Hugging Face ecosystem.

If your company is building custom visual generation flows — whether for e-commerce, marketing, entertainment or any product that requires bespoke images — this stack deserves evaluation. The barrier to doing serious fine-tuning has dropped considerably.

In a nutshell

Today’s three stories point to the same underlying question: who controls the standards? The Kaggle challenge scandal shows that even the biggest organizations in the sector can fail to distinguish real quality from the appearance of quality as the volume of AI production increases. TabFM Studio and NVIDIA’s integration with Hugging Face, on the other hand, represent the opposite movement: democratizing access to powerful tools so more people can use them without depending on technical gatekeepers. The risk of that movement, of course, is that democratization without solid evaluation criteria produces exactly the kind of result we saw in DeepMind’s challenge. AI is becoming more accessible and harder to audit at the same time.

---

Sources used: https://www.reddit.com/r/MachineLearning/comments/1uzyf66/did_blatant_ai_slop_just_win_a_25k_usd_deepmind/, https://www.reddit.com/r/MachineLearning/comments/1uzx1el/tabfm_studio_pointandclick_predictions_on/, https://huggingface.co/blog/nvidia/scale-diffusers-finetuning-nemo-automodel