A new AI test built from real expert work: the best model scores 12%

On 3 November SemiAnalysis, the research firm best known for its work on AI chips and datacentres, launched GDPx, a new benchmark (a standard test used to compare AI models). It was built with 14 small companies that make training material for AI labs, each an expert in one field. The result is 206 tasks across 11 areas where SemiAnalysis expects advanced AI to earn the most money in the next year or two: chip design, cybersecurity, biology, finance, law, quant trading (computer-driven stock trading), AI research itself, hardware design, materials science, alignment (whether a model behaves safely and stays within its instructions) and software engineering.

The headline: the best model scores 12%. That is a striking number in a year when the top models keep scoring near 100% on the industry's hardest exams. In September, OpenAI's GPT-6 Astra scored 100%, 99.9% and 97.6% on three of them, as we covered at the time. So which number is right?

The leaderboard. Averaging all 11 areas equally over three attempts per task: Anthropic's Claude Opus 5.5 12.0%, Google's Gemini 4 Carbon 10.9%, OpenAI's GPT-6 Astra 9.6%, Meta's Muse Wonder Full 9.0%, Claude Fable 5.1 8.5%, GPT 6.1 Sol 8.4% and Claude Sonnet 5.5 8.2%. The best free-to-download model, DeepSeek 4.1 Flash, scored 5.4%; China's GLM 5.3 scored 2.7%. Cost varies wildly: Opus spent about $51 per task, Gemini 4 Carbon about $16, and DeepSeek and OpenAI's cheap GPT-6 Luna under $1, for roughly half the top score.

Why the tasks are hard. Most popular tests hand the model everything it needs in one tidy prompt. GDPx drops it into a realistic workplace and lets it dig. One finance task is built from a real private-equity deal (an investment firm buying about 62% of a snack company for $880 million, with all names changed): the model must navigate a deal room of roughly 4,000 files, emails and spreadsheets, as a junior accountant on the real deal did. Cyber tasks are built on real, previously unknown security holes in running software. Materials tasks put the model inside a simulated laboratory. Each task allows up to 8 hours, and each model works in its own maker's tool where one exists, such as Claude Code for Opus.

The zeros. SemiAnalysis grades finance and law pass-or-fail by design, because a single wrong cell in a spreadsheet can sink a deal. On that grading every model scored 0% in both. The leaderboard therefore shows partial credit for those two. The failures are instructive: in the deal task, 15 runs counted the banker's success fee as debt when the purchase agreement deducts it separately, and 17 kept a family member who logged no hours on the payroll as a real business cost. In quant trading, 70 of 111 runs did worse than the simple strategy they were asked to beat.

Where each model shines. By SemiAnalysis's account Opus 5.5 wins quant trading, hardware design and chip design; Gemini 4 Carbon wins biology and narrowly software engineering; GPT-6 Astra wins AI research and materials science. SemiAnalysis also says the safety filters on Opus and Astra pull their scores down in cyber, biology and alignment, where they sometimes refuse, and that Meta's Muse Wonder Full is currently the best cyber model 'by a meaningful margin'.

Who built it. These are SemiAnalysis's own claims about its own test. Its partners sell training data to AI labs, and the two SemiAnalysis authors previously co-founded such a company. SemiAnalysis says every partner had to pass its quality checks on the first try and that it dropped several well-known ones that did not. When it ran the same checks on Terminal-Bench 4.0, a popular coding-agent test, nearly half the tasks were flagged as broken.

Think of a driving test. Most AI benchmarks have been the written theory exam: clear questions, one right answer, everything you need on the page. The best models now ace it. GDPx is closer to driving through Oslo at rush hour in an unfamiliar car, with roadworks, a confusing sat-nav and a passenger who gives half the directions. Passing the theory exam with full marks is real knowledge. It just does not tell you whether someone can do the drive.

A 12% score does not mean the AI is useless. It means that on long, messy, expert-level jobs, where one mistake fails the whole task, today's best model gets the equivalent of about one in eight right on its own. Partial credit and human checking change that picture a lot, which is how most people actually use these tools.

Hard benchmarks are a recurring cycle. A test is built to be difficult, models catch up, and within a year or two it is 'saturated' (top models score so high it can no longer tell them apart). Humanity's Last Exam, launched in early 2025 as a set of questions even experts struggle with, began with top models scoring in single digits; in September we wrote about GPT-6 Astra maxing out three other hard exams. Benchmarks with low headline scores are also not new; every fresh test starts that way.

What is newer is the shape. The closest predecessor is OpenAI's GDPval from September 2025, which measured AI on tasks drawn from real jobs and is still quoted on model launch pages. SemiAnalysis argues GDPval is now inadequate because each task spoon-feeds the context in a short prompt, and because its rubric grading is noisy: OpenAI's own human experts agreed with each other only 71% of the time. GDPx's bet is that the hard part of real work is finding the right information in a pile of files, not writing the final report. SemiAnalysis also openly picked the areas by where it thinks AI money will be made, and left out fields such as government, real estate and insurance. That makes GDPx a test of the frontier, not of the average job.

The fair caution: GDPx is one week old, built by one firm and its partners, and not yet reproduced by anyone else. SemiAnalysis plans a new version every three to six months.

For anyone choosing an AI tool, the useful lessons are the uneven ones. No model wins everywhere: SemiAnalysis's charts show sharp spikes by field, so the best choice for a chip designer may not be the best for a biologist. Safety filters matter in practice: a model that refuses part of a security job scores lower even if it is more capable. And price is a real variable: Gemini 4 Carbon reached about 90% of the top score at about a third of Opus's cost per task.

For the bigger 'will AI take expert jobs' question, GDPx is a sober data point. On end-to-end deal accounting, contract redlining and trading research, today's best models still fail the pass-or-fail version of the job. That is the honest picture this week: very strong assistants, not yet independent professionals. The measured caveat runs the other way too. These are the hardest tasks SemiAnalysis's partners could build, and earlier 'impossible' tests did not stay hard for long. The number to watch is not 12% itself but how fast it moves in the next version.