GPT-3 can reason about as well as a college student, psychologists report

Researchers from UCLA have conducted a study showing that the artificial intelligence language model GPT-3 performs remarkably well in analogical reasoning tasks, similar to college undergraduates. Analogical reasoning is a cognitive process that was previously believed to be a uniquely human ability.

The study, published in Nature Human Behaviour, evaluated GPT-3’s ability to solve reasoning problems inspired by Raven’s Progressive Matrices, a test that involves predicting the next image in a complex arrangement of shapes. Surprisingly, GPT-3 performed about as well as human participants, solving 80% of the problems correctly, compared to the human average score of just under 60%. The AI model also outperformed humans in solving SAT analogy questions.

However, the researchers noted that GPT-3 still faces significant limitations. It struggled with problems that required understanding physical space, such as using tools to complete a physical task. In these cases, the solutions suggested by GPT-3 were nonsensical, even though they were problems that children can typically solve quickly.

The UCLA researchers raised questions about the nature of GPT-3’s reasoning abilities. They wondered whether the AI’s performance is a result of mimicking human reasoning due to its massive language training dataset or if it employs a fundamentally new cognitive process.

GPT-3’s inner workings are proprietary and guarded by OpenAI, the company that developed it, making it challenging for the researchers to ascertain the exact cognitive processes it uses. Despite its impressive performance in some reasoning tasks, the AI tool still falls short in other areas.

The researchers have developed their own computer model inspired by human cognition and have been comparing its performance to that of commercial AI systems. GPT-3’s capabilities have raised questions about whether language learning models are beginning to “think” like humans or if they are merely mimicking human thought in a different way. The researchers hope to gain a deeper understanding of AI models’ underlying cognitive processes to determine their true potential.

Ultimately, gaining access to the software and training data of AI models like GPT-3 would be crucial for cognitive researchers to better understand the AI’s decision-making processes and potentially unlock new breakthroughs in artificial intelligence.