AI's Take on Work: Decoding Occupations through GPT-4's Eyes

As AI becomes a larger part of our daily lives, its influence extends beyond technological convenience and functional tasks. When used for research and information it can also affect how we understand the world around us, touching on deeper social aspects such as how we see the world of work.

New research looks at how the large language model (LLM) GPT-4 evaluates occupations and compares this with how humans evaluate occupations. We provide an in-depth look at how AI may contribute to our understanding of occupational hierarchies, a topic traditionally explored through human surveys.

AI mimics human judgments quite well

The study systematically compares GPT-4's assessments of the prestige and social value of 580 occupation titles against those obtained from a high-quality human respondents survey in the UK. The findings reveal a high correlation between GPT-4 and human evaluations across all occupational categories , showcasing AI's potential as a useful research tool.

GPT-4 was particularly strong in predicting the prestige and social value within the major occupational groups of managers (e.g., bank manager, government minister, IT manager and head teacher), technicians and associate professionals (e.g., aircraft pilot, insurance broker, real estate agent and footballer) as well as elementary occupations (e.g., office cleaner, box packer, garden labourer and street newspaper vendor).

However, there are also critical considerations and limitations of such technology. For example, GPT-4 overestimates the prestige and social value of emerging occupations in the digital economy and sales-oriented jobs.

Examples where the accuracy of the predictions was particularly low include email marketer, chatbot conversation trainer, marketing manager and online video content creator. Here the average social value scores of GPT-4 were more than 30 points higher than the human scores on a 0-100 scale (e.g., the AI model assigned email marketer and online video content creator a social value of around 60 but humans only scored these occupations around 28). The scores of GPT-4 also align more closely with white and female respondents than black and male ones, indicating some biases.

The importance of this research lies not only in its comparative approach but also in its exploration of AI's role in constructing social realities. Traditional measures of occupational prestige and social value have relied on human perception, shaped by societal norms and media narratives.

By introducing AI as a non-human evaluator, this research opens new avenues for analyzing how occupations are valued and perceived, offering insights into the changing landscape of work in the digital age.

The need for a balanced approach

While LLMs have clear advantages in terms of efficiency and cost-effectiveness, their application in occupational evaluation raises questions about the alignment of AI-generated data with human perspectives.

A careful consideration of AI's limitations and biases is needed. A balanced approach that leverages AI's strengths while being mindful of and addressing its shortcomings will yield the best outcomes.

Taken together, the study presents a nuanced view of AI's potential to mirror and inform our understanding of occupational realities, especially how they are seen. By comparing GPT-4's evaluations with human judgments, the research not only shows the usefulness of AI in supporting researchers but also prompts a critical examination of the ethical challenges posed by AI's growing role in society.