For well over a year, California has been fighting against Silicon Valley’s powerful lobbying efforts to thwart A.I. regulations. On Monday, the state achieved a milestone when Gov. Gavin Newsom signed into law SB 53—an incremental win for the pro-regulation camp. The Transparency in Frontier Artificial Intelligence Act, authored by state Sen. Scott Wiener, is an overhauled version of his infamous SB 1047, which was passed by California’s legislature last year before being vetoed by Newsom for being “well-intentioned” but too strict. Afterward, Newsom convened a group of “world-leading A.I. academics and experts” to suggest a more even-handed approach, and SB 53 was born.

The new law has a couple of core objectives. Among the most significant are expanded whistleblower protections and safety-related transparency and reporting requirements, which must be shared on company websites. While promising in theory, the law doesn’t outline any clear enforcement methods, and might not be much better than the voluntary reporting regime the industry has been pushing for. The bill also lays the groundwork for a public compute cluster, which would make A.I. more available to researchers and institutions that don’t have billions of dollars in funding. Some recommendations—such as third-party evaluation requirements—didn’t make it into the final version of the bill.

A MESSAGE FROM OUR SPONSOR

Meet Rovo, your new AI teammate

Rovo is the AI-powered teammate connected to your organization's knowledge. With Rovo, you can streamline your workflow and power up your team's productivity.

Find what you need in a snap with Rovo Search. From brainstorming to complex requests, Rovo Chat delivers insights in the context of your work. Team up with out-of-the-box Rovo Agents – or build the perfect agent for your project.

Rovo is powered by Atlassian and included in Jira, Confluence, or Jira Service Management Enterprise or Premium Cloud subscriptions.

Know when AI turns from tool to teammate? If you Rovo, you know.

Discover Rovo

Unlike SB 1407, which had been quickly (and publicly!) lambasted by the anti-regulation crowd, SB 53 moved more quietly through the legislature. Teri Olle, the director of Economic Security California Action, a co-sponsor of the bill, attributed the legislation’s relatively easy passage to the working group whose recommendations shaped its scope—and to Anthropic, of course, which offered its public support. Olle also conceded that the legislation was a compromise—a “turn in your homework bill” that’s as “light touch as you can get,” relying on transparency rather than liability. But, she added, the bill still embraces a core principle of SB 1047: that “innovation and guardrails are not incompatible.”

Many of the industry’s biggest players have welcomed the law. Meta, which recently launched a PAC to fight against strict A.I. regulation in California, called it a “positive step” toward balanced A.I. regulation. OpenAI—whose co-founder Greg Brockman recently funded a $100 million super PAC focused on opposing A.I. policies that “stifle innovation”—said it was “pleased to see that California has created a critical path toward harmonization with the federal government.” Jack Clark, Anthropic’s co-founder, agreed that “SB 53 establishes meaningful transparency requirements for frontier A.I. companies without imposing prescriptive technical mandates.” (Nvidia declined to comment, and Microsoft, IBM, and Google didn’t return requests for comment.)

Meanwhile, a number of other A.I.-related bills are sitting on Newsom’s desk. The No Robo Bosses Act, or SB 7, would require human oversight of A.I. systems in the workplace; the LEAD for Kids Act is intended to protect children from dangerous companion chatbots; AB 325 would prohibit pricing algorithms; and a handful of other bills are still under review. Newsom has until October 12 to either sign or veto these.

And now for the main event…

Earlier this year, a handful of A.I. models achieved gold medal performances in the International Math Olympiad—the ne plus ultra of math competitions, which draws the brightest high-school students from around the globe for its two-day test. The competition forces participants to tackle maddeningly difficult problems that demand creativity, critical thinking, and a deep knowledge of multiple branches of mathematics—all of which also makes it an ideal forum for testing model capabilities. This year, OpenAI and Google DeepMind tested advanced, unreleased systems, and both scored 35 out of 42, correctly answering five out of six questions—worse than the best teenage mathletes in the world, but better than just about everyone else.

One other A.I. achieved a gold medal performance: Aristotle, the first model from Harmonic, whose creators are betting on formal mathematical verification to eliminate the problem of A.I. hallucinations. Co-founded in 2023 by Vlad Tenev, the billionaire co-founder and C.E.O. of the trading platform Robinhood, and Tudor Achim, a wunderkind computer scientist, Harmonic recently closed a $100 million Series B funding round at an $875 million valuation with the goal of achieving “mathematical superintelligence.”

This week, Harmonic agreed to give me an exclusive first look at the model architecture behind Aristotle. When I connected with Achim, Harmonic’s C.E.O., he said that the decision to speak with me was simple. “The team just felt very excited about what we built—you know, we think it’s a great system,” he told me, adding that the official release of their research paper, which will be published soon on Arxiv, proves that chain-of-thought reasoning is not, in fact, the only way to elicit “reasoning” in an A.I. model. Harmonic’s alternative approach is wonky, but worth exploring in detail.

It’s widely accepted that large language models are “brittle” and unreliable—prone to spitting out incorrect statements with the same confidence as correct ones. Automatically, or mathematically, verifying a given model’s output, however, would render them safe for much broader use by effectively eliminating hallucinations. That’s essentially what Harmonic claims to have done with Aristotle, although the company’s forthcoming paper doesn’t disclose training data, source code, etcetera. (Achim said they are “definitely not ruling out” a more open-source release.)

A MESSAGE FROM OUR SPONSOR

Meet Rovo, your new AI teammate

Rovo is the AI-powered teammate connected to your organization's knowledge. With Rovo, you can streamline your workflow and power up your team's productivity.

Find what you need in a snap with Rovo Search. From brainstorming to complex requests, Rovo Chat delivers insights in the context of your work. Team up with out-of-the-box Rovo Agents – or build the perfect agent for your project.

Rovo is powered by Atlassian and included in Jira, Confluence, or Jira Service Management Enterprise or Premium Cloud subscriptions.

Know when AI turns from tool to teammate? If you Rovo, you know.

Discover Rovo

The Harmonic paper, which I’ve reviewed, explains that Aristotle consists of three major subsystems: a Monte Carlo tree search algorithm; an informal reasoning system that breaks down generated proofs into lemmas (already proven mathematical statements) before formalizing them in Lean (a machine-verifiable programming language); and a geometry solver system. Achim explained that Aristotle’s biggest differentiator is the search algorithm, which “explores many more branches of reasoning in parallel than standard chain-of-thought approaches.”

While chain-of-thought reasoning explores one thing at a time, he said, the tree search surfaces “maybe 10 things in parallel, and then for each of those, you might try 10 more things in parallel.” The “exponentially growing tree” is kept under control by “using a value function that prioritizes what to explore.” A proprietary language model guides the tree search toward its next steps, though the paper notes that the search algorithm alone can prove “many challenging university-level and Olympiad-level math problems.”

For a gut check on Harmonic’s approach, I reached out to UCLA professor Terence Tao, a Fields medalist who is widely regarded as one of the greatest living mathematicians. “Blending the creative (but error-prone) ability of large language models with deterministic, but extremely reliable, formal verification software such as Lean is a very promising way forward,” he told me. “Particularly in research-level mathematics, where we seek completely reliable proofs of statements.” You heard it here first.

I’m not a mathematician, so I asked Achim to explain some of the possible uses and impacts of Aristotle. To start, he said, Harmonic has seen very strong results with its ability to formally verify code for software engineering. “You embed code in Lean, and then if you have a specification about what that code in the other language should do, you convert that specification into Lean, and then prove that that specification is met in the Lean embedding of that language,” he explained. In other words, turning code into verifiable math—a surefire way to ensure that the original code is not just correct, but also less buggy and more robust. “It turns out this is very scalable,” Achim said, because “you only have to write [that embedded algorithm] once, and then you can reuse that over and over in the same form on arbitrary code that’s passed in.”

Another benefit, Achim said, is that if you have a Lean proof of a piece of code, you can be sure it’s hallucination free. That doesn’t mean Aristotle is incapable of hallucinations, per se—it’s still built atop an L.L.M., which is error-prone by nature. But the requirement that any output needs to be validated in Lean means that any problems are immediately flagged to its human operator.

The upshot, in theory, is that software engineers using Aristotle can have much greater confidence that the code it produces won’t need to be debugged—and that mathematicians like Tao will be able to greatly accelerate the pace of theoretical breakthroughs. “We envision a future where, instead of humanity solving one giant open problem per decade, they’re solving one every day with the help of A.I.,” Achim said. “And these proofs are going to be extremely sophisticated, way beyond what an individual human can check in one day. It’s going to be important that, at that scale, the computer provides some certificate that the proof is correct, because no matter how smart the system is, you won’t be able to trust it 100 percent.”

Of course, as exciting as formal verification for A.I. output may be, Harmonic’s approach has limitations, too. Selmer Bringsjord, the director of the Rensselaer A.I. and Reasoning Lab, pointed out that verification tends to be time-consuming, so it’s less than ideal for scenarios that demand rapid action, like aerospace or combat missions. But Achim argued that it’s just a matter of time (and more compute) before models can engage in fast, highly advanced, hallucination-free problem solving. Moreover, he said, there’s no real “upper bound” to how far scaled reinforcement learning can advance an A.I. system, even though many researchers are skeptical that systems can endlessly improve their capabilities in this manner. Still, Achim acknowledged that the system—running on “pools” of cloud-based G.P.U.s and C.P.U. clusters—is “costly to maintain.”

The company presumably has some cash to burn, having raised a $75 million Series A in 2024 before securing its $100 million B round this summer. But all that reinforcement learning is expensive. Achim told me that a subscription product is coming after Aristotle finishes its beta test, though he declined to share any timeline for a full product release. In the meantime, he hopes that “a system that’s very good at math will generalize to other reasoning tasks” beyond STEM problems. At least, that’s the idea.

Harmonic is chasing even more ambitious goals, too. To his credit, Achim offered me a definition of what the company has been referring to as “mathematical superintelligence”—describing it as “a system that’s able to solve math faster and better than … the collection of all human mathematicians.” As for a specific benchmark, Achim argued that “the first time an A.I. is able to solve, let’s say, a Millennium Prize [problem] by itself, without human intervention, I would say we’ve achieved mathematical superintelligence.” Of course, it’s as easy to make these sorts of grandiose promises as it is to become cynical about them. Achim, for his part, said he’s expecting Harmonic to crack the code by 2028, if not sooner.

The Information got its hands on OpenAI’s latest P&L. According to the report, OpenAI earned $4.3 billion in sales for the first half of this year, and burned $2.5 billion in cash. That sets the company on track to rake in less than the $13 billion that was previously reported. [The Information]

Ed Zitron just published an 18,000-word piece on why the A.I. industry’s financial structures simply don’t work—and won’t work. It’s a novella-length, 71-minute read (clearly, he doesn’t have my editors). In one paragraph worth highlighting, Zitron reported that Microsoft has only 8 million active Copilot users—less than 2 percent of Microsoft’s total 440 million Office 365 subscribers. Those numbers align with qualitative stories about generative A.I. adoption generally, and Copilot more specifically. [Where’s Your Ed At]

Richard Sutton, a Turing Award–winning pioneer in the field of A.I. and “the father of reinforcement learning,” recently argued on Dwarkesh Patel’s podcast that L.L.M.s have reached a dead end on the path to achieving general intelligence. The appearance sparked a fair bit of backlash from the “L.L.M.s are one step away from God” crowd. [Dwarkesh]

That’s all for today. I’ll see you next week.

Ian

Need help? Review our FAQ page or contact us for assistance. For brand partnerships, email ads@puck.news.

You received this email because you signed up to receive emails from Puck, or as part of your Puck account associated with {{customer.email}}. To stop receiving this newsletter and/or manage all your email preferences, click here.

Puck is published by Heat Media LLC. 107 Greenwich St, New York, NY 10006