Browse
Reference
Directory
Company
Help & Legal
Follow
Subscribe
Quick Take
Meta researchers deployed AI agents to run 1,097 machine-learning experiments in 31 days, consuming approximately 360,000 GPU-hours to improve search activity by 0.23%.
The experiment, detailed in a Nov. 18 research paper, demonstrated how autonomous AI systems can develop and test improvements to large-scale recommendation models with limited human intervention.
However, the resulting model also slowed the rendering of Meta's search interface, exposing a trade-off between automated performance improvements and the resources required to achieve them.
AI agents take over hundreds of engineering experiments
The researchers developed a system that lets AI agents propose changes to machine-learning models, modify their code, train new versions, and evaluate performance.
The experiment targeted Meta's search-suggestion system, which determines the recommendations users see when opening the search bar before typing a query.
Traditionally, improving such systems requires engineers to identify potential changes, implement them and assess their performance through repeated training and testing.
Meta's approach automated much of that process, allowing agents to conduct multiple experiments simultaneously while engineers retained control over research priorities and final deployment decisions.
The system used Anthropic's Claude Opus 4.8 to explore architectural improvements and generate candidate changes. Across 31 days, the agents completed 1,097 candidate experiments and accepted 325, representing a 29.6% acceptance rate.
Meta’s live search test
Agents screened candidates offline. Meta’s engineers reviewed the combined model before testing it on live traffic.
The combined model drew more search sessions, and the search interface took longer to render.
The live test applies to one combined ranker. The paper leaves permanent rollout and a matched comparison with human-only engineering unresolved.
Sources
Supporting facts
Method
The blue share of the experiment bar is 325 completed candidates kept divided by 1,097 completed candidates. The live result is the relative difference in search-session volume for the combined model versus the control, with the authors’ 95% confidence interval. GPU-hours are Meta’s estimate for the campaign.
Scope
That compared with a 15.6% acceptance rate in an earlier version of the system, which evaluated changes sequentially rather than running multiple experiments in parallel.
The researchers also reported that the agents identified several architectural improvements that Meta's engineering team had not previously explored.
However, the campaign required substantial computing resources.
Meta estimated total consumption at about 360,000 GPU-hours, including completed experiments, additional tests measuring training variability, and failed or retried jobs.
The researchers said individual training and evaluation runs took several hours, making parallel execution necessary to complete the campaign within a month.
Small search gains expose limits of automated optimization
After combining the accepted improvements into a final model, Meta tested it against its existing search-ranking system using live traffic.
The results showed a 0.23% increase in search-session volume, with a 95% confidence interval of plus or minus 0.15 percentage points.
Other engagement measures, including participation and retention, moved in a positive direction.
For a platform handling more than a billion search-ranking requests daily, even a modest improvement in user activity could have considerable operational significance.
However, the experiment also produced an unexpected consequence.
The new model increased the time the search interface took to render, although the researchers did not disclose the size of the slowdown.
The problem arose because the agents optimized performance against offline measures of search engagement, while researchers monitored interface rendering speed separately.
The researchers acknowledged that the system could not independently identify every trade-off between improving its assigned performance targets and maintaining other aspects of the user experience.
The experiment also highlights the computing demands of using AI to improve existing AI systems.
Although the campaign produced a statistically significant increase in search activity, Meta did not disclose its financial cost or compare the resources consumed with those required by a conventional engineering team.
The researchers nevertheless argued that the results showed autonomous agents could perform meaningful machine-learning engineering work on production-scale systems.
Their findings leave Meta with a practical question of whether the additional search activity and engineering productivity justify the computing resources and slower interface performance before the model is adopted more broadly.
Mentioned in this article
Bitcoin crashes to $59,000 as Anthropic falls 25% from yesterday’s IPO-day high
Cookie Consent
We use cookies to improve your experience on our site. By using our site, you consent to cookies.
Cookie Preferences
Manage your cookie preferences below:
Essential cookies enable basic functions and are necessary for the proper function of the website.
CloudFlare provides web performance and security solutions, enhancing site speed and protecting against threats.
Service URL: developers.cloudflare.com (opens in a new window)
Google Analytics is a powerful tool that tracks and analyzes website traffic for informed marketing decisions.
Service URL: policies.google.com (opens in a new window)
Statistics cookies collect information anonymously. This information helps us understand how visitors use our website.
Clarity is a web analytics service that tracks and reports website traffic.
Service URL: clarity.microsoft.com (opens in a new window)
You can find more information in our Privacy Policy and .
Loading image…
This image could not be loaded.
The Catalyst



