Subscribe to Updates

    Get the latest creative news from FooBar about art, design and business.

    What's Hot

    The impressive new Nothing Headphone (1) Pro are on sale at their first great discount

    Motorola updates the status of its phones receiving Android 17

    The most misunderstood graph in AI just got a major update

    Facebook Twitter Instagram
    • Tech
    • Gadgets
    • Spotlight
    • Gaming
    Facebook Twitter Instagram
    circuitthoughtscircuitthoughts
    Subscribe
    • Home
    • Gadgets
    • Insights
    • Apps

      Google Uses AI Searches To Detect If Someone Is In Crisis

      Gboard Magic Wand Button Will Covert Your Text To Emojis

      Android 10 & Older Devices Now Getting Automatic App Permissions Reset

      Spotify Blend Update Increases Group Sizes, Adds Celebrity Blends

      Samsung May Improve Battery Significantly With Galaxy Watch 5

    • Gear
    • Mobiles
      1. Tech
      2. Gadgets
      3. Insights
      4. View All

      The impressive new Nothing Headphone (1) Pro are on sale at their first great discount

      Motorola updates the status of its phones receiving Android 17

      The most misunderstood graph in AI just got a major update

      Amazon finally drops a decent Moto G Power (2026) deal for Black Friday

      March Update May Have Weakened The Haptics For Pixel 6 Users

      Project 'Diamond' Is The Galaxy S23, Not A Rollable Smartphone

      The At A Glance Widget Is More Useful After March Update

      Pre-Order The OnePlus 10 Pro For Just $1 In The US

      Motorola Edge+ Review: It Checks A Lot Of Boxes

      This Smartphone Concept Design Is Different… In A Good Way

      Twitter Just Made Searching Your Direct Messages Better

      That Netflix Price Hike Is Starting To Take Place

      Latest Huawei Mobiles P50 and P50 Pro Feature Kirin Chips

      Samsung Galaxy M62 Benchmarked with Galaxy Note10’s Chipset

      9.1

      Review: T-Mobile Winning 5G Race Around the World

      8.9

      Samsung Galaxy S21 Ultra Review: the New King of Android Phones

    • Computing
    circuitthoughtscircuitthoughts
    Home»Tech»The most misunderstood graph in AI just got a major update
    Tech

    The most misunderstood graph in AI just got a major update

    adminBy No Comments3 Mins Read
    Facebook Twitter Pinterest LinkedIn Tumblr Email
    Share
    Facebook Twitter LinkedIn Pinterest Email

    Even if the models do lag behind human performance on open-ended tasks right now, the new report suggests that they might be getting better. Some of the newest models, such as Opus 5.5, tend to achieve what METR calls “borderline above the bar” performance on the evaluation design tasks—not yet usable for METR’s research, but promising. Just a year ago, every model was scoring in the “clearly below the bar” category.

    But because these open-ended and judgment-heavy tasks are so tough to design and score, METR includes few of them in its report, and that makes it difficult to predict whether and how models might continue to progress in these areas. “The improvement we observe, I think that does tell us something for the current moment,” says Najoung Kim, a linguist and computer scientist at Boston University. “I’m not sure just yet how to think about how it would extrapolate.”

    Matt Putz, a member of the technical staff at METR, says that predicting progress on these kinds of tasks is likely to remain difficult. “By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify,” he said in an email. “Ultimately, I don’t think we’ll be able to establish trends as clean and robust as for the easy-to-specify [tasks].”

    In general, METR is clearly butting up against the practical difficulties of testing AI performance on difficult and underspecified tasks—precisely the ones most relevant to automating AI R&D. METR’s time horizon plots score task difficulty by evaluating how long it would take a human to complete the task, and for the original plot they measured this quantity by actually having people try each task. 

    In the new report, however, the completion times are mostly estimates, because paying people for tasks that might take hundreds of hours, or that only a few experts are qualified to complete, is prohibitively expensive and time-consuming. Other tasks have more specific shortcomings—for example, for the philosophical argument task, just one person rated most of the arguments, and agents are judged by how well they agree with that particular individual. It’s possible that the scores would change if more experts were surveyed—but that, too, is time-consuming.

    It’s still early days for evaluating models on tasks that are hard to validate or require judgment. And while the experts I spoke with see the inclusion of these tasks in METR’s newest time horizon plot as a clear step in the right direction, they agree that we are still far from achieving any definitive answers about whether, and how quickly, automated AI R&D might be coming. “This move toward open-ended tasks is promising,” Kapoor says. But, he adds, “there is just such a lot more to be done to actually get a fuller understanding.”

    #misunderstood #graph #major #update

    graph major misunderstood Update
    Share. Facebook Twitter Pinterest LinkedIn Tumblr Email

    Related Posts

    The impressive new Nothing Headphone (1) Pro are on sale at their first great discount

    Motorola updates the status of its phones receiving Android 17

    Amazon finally drops a decent Moto G Power (2026) deal for Black Friday

    Add A Comment

    Leave A Reply Cancel Reply

    Editors Picks
    8.5

    Apple Planning Big Mac Redesign and Half-Sized Old Mac

    Autonomous Driving Startup Attracts Chinese Investor

    Onboard Cameras Allow Disabled Quadcopters to Fly

    Top Reviews
    9.1

    Review: T-Mobile Winning 5G Race Around the World

    By
    8.9

    Xiaomi Mi 10: New Variant with Snapdragon 870 Review

    By
    8.9

    Samsung Galaxy S21 Ultra Review: the New King of Android Phones

    By
    circuitthoughts
    Facebook Twitter Instagram Pinterest Vimeo YouTube
    • Home
    • Tech
    • Gadgets
    • Mobiles
    • Our Authors
    © 2026 ThemeSphere. Designed by WPfastworld.

    Type above and press Enter to search. Press Esc to cancel.