Welcome to Trendster’s common AI publication! We’re happening hiatus for a bit, however you’ll find all our AI protection, together with my columns, our every day evaluation, and breaking information tales, at Trendster. If you would like these tales and way more in your inbox on daily basis, join our every day newsletters right here.
This week, billionaire Elon Musk’s AI startup, xAI, launched its newest flagship AI mannequin, Grok 3, which powers the corporate’s Grok chatbot apps. Educated on round 200,000 GPUs, the mannequin beats plenty of different main fashions, together with from OpenAI, on benchmarks for arithmetic, programming, and extra.
However what do these benchmarks actually inform us?
Right here at TC, we frequently reluctantly report benchmark figures as a result of they’re one of many few (comparatively) standardized methods the AI trade measures mannequin enhancements. Well-liked AI benchmarks have a tendency to check for esoteric data, and provides combination scores that correlate poorly to proficiency on the duties that most individuals care about.
As Wharton professor Ethan Mollick identified in a collection of posts on X after Grok 3’s unveiling Monday, there’s an “pressing want for higher batteries of checks and impartial testing authorities.” AI corporations self-report benchmark outcomes as a rule, as Mollick alluded to, making these outcomes even more durable to just accept at face worth.
“Public benchmarks are each ‘meh’ and saturated, leaving plenty of AI testing to be like meals critiques, primarily based on style,” Mollick wrote. “If AI is crucial to work, we want extra.”
There’s no scarcity of impartial checks and organizations proposing new benchmarks for AI, however their relative benefit is much from a settled matter throughout the trade. Some AI commentators and consultants suggest aligning benchmarks with financial affect to make sure their usefulness, whereas others argue that adoption and utility are the last word benchmarks.
This debate could rage till the top of time. Maybe we should always as an alternative, as X consumer Roon prescribes, merely pay much less consideration to new fashions and benchmarks barring main AI technical breakthroughs. For our collective sanity, that might not be the worst thought, even when it does induce some degree of AI FOMO.
As talked about above, This Week in AI is occurring hiatus. Thanks for sticking with us, readers, by way of this curler coaster of a journey. Till subsequent time.
Information
OpenAI tries to “uncensor” ChatGPT: Max wrote about how OpenAI is altering its AI improvement strategy to explicitly embrace “mental freedom,” irrespective of how difficult or controversial a subject could also be.
Mira’s new startup: Former OpenAI CTO Mira Murati’s new startup, Pondering Machines Lab, intends to construct instruments to “make AI work for [people’s] distinctive wants and objectives.”
Grok 3 cometh: Elon Musk’s AI startup, xAI, has launched its newest flagship AI mannequin, Grok 3, and unveiled new capabilities for the Grok apps for iOS and the online.
A really Llama convention: Meta will host its first developer convention devoted to generative AI this spring. Known as LlamaCon after Meta’s Llama household of generative AI fashions, the convention is scheduled for April 29.
AI and Europe’s digital sovereignty: Paul profiled OpenEuroLLM, a collaboration between some 20 organizations to construct “a collection of basis fashions for clear AI in Europe” that preserves the “linguistic and cultural range” of all EU languages.
Analysis paper of the week
OpenAI researchers have created a brand new AI benchmark, SWE-Lancer, that goals to guage the coding prowess of highly effective AI techniques. The benchmark consists of over 1,400 freelance software program engineering duties that vary from bug fixes and have deployments to “manager-level” technical implementation proposals.
In response to OpenAI, the best-performing AI mannequin, Anthropic’s Claude 3.5 Sonnet, scores 40.3% on the total SWE-Lancer benchmark — suggesting that AI has fairly a methods to go. It’s price noting that the researchers didn’t benchmark newer fashions like OpenAI’s o3-mini or Chinese language AI firm DeepSeek’s R1.
Mannequin of the week
A Chinese language AI firm named Stepfun has launched an “open” AI mannequin, Step-Audio, that may perceive and generate speech in a number of languages. Step-Audio helps Chinese language, English, and Japanese and lets customers alter the emotion and even dialect of the artificial audio it creates, together with singing.
Stepfun is certainly one of a number of well-funded Chinese language AI startups releasing fashions underneath a permissive license. Based in 2023, Stepfun reportedly lately closed a funding spherical price a number of hundred million {dollars} from a number of buyers that embrace Chinese language state-owned non-public fairness corporations.
Seize bag
Nous Analysis, an AI analysis group, has launched what it claims is without doubt one of the first AI fashions that unifies reasoning and “intuitive language mannequin capabilities.”
The mannequin, DeepHermes-3 Preview, can toggle on and off lengthy “chains of thought” for improved accuracy at the price of some computational heft. In “reasoning” mode, DeepHermes-3 Preview, much like different reasoning AI fashions, “thinks” longer for more durable issues and exhibits its thought course of to reach on the reply.
Anthropic reportedly plans to launch an architecturally comparable mannequin quickly, and OpenAI has stated such a mannequin is on its near-term roadmap.
