Exploring responsible uses of AI in environmental science

Evidence syntheses are considered the gold standard by everyone from scientists to policymakers, who use the pool of data provided to plan new projects, compare research findings, or make real-world decisions.

These big picture summaries of the state of science reveal clear, reliable insights into complex questions from a variety of sources. But the time and resources required to create evidence syntheses is a significant investment.

PiNC Lab’s Luyanda Mazibuko recognised this challenge, specifically for identifying the evidence base underpinning nature-based solutions for climate change, where the volume of material was overwhelming. 

Embarking on an Honours research project, Mazibuko explored whether generative AI model ChatGPT-4o, could reliably support systematic reviews of nature-based solutions (NbS) for climate change adaptation.

Now celebrating the award of her degree, she reflects on the Honour’s research journey, the impact of her work, and where she hopes to go next.

Assessing AI-generated responses

Mazibuko’s research builds on an existing systematic review of 119 nature-based solutions papers, using a structured codebook designed to extract comparable information across diverse studies. From this dataset, 10 peer-reviewed studies were selected to reflect geographic and thematic diversity across South Africa, Kenya, Ghana, Madagascar, and Tanzania.

Each paper was assessed using 440 AI-generated responses spanning meta-data, binary, open-ended, and closed-ended multi-category formats against a human-coded reference standard. ChatGPT-4o’s responses were evaluated using a three-level validity rubric: Correct, Partially Correct, and Incorrect or False.

“What became clear is that AI performs best when information is standardised, well structured and easy to locate in papers. Where reporting is unstructured, narrative, ambiguous or categories overlap, errors increase,” says Mazibuko. 

Overall, 87% of responses were fully or partially correct, with especially strong performance on tasks such as identifying meta-data and answering binary questions.

“These are the kinds of tasks that take up a lot of time in systematic reviews,” Mazibuko notes. “Seeing such high accuracy here suggests real potential for improving efficiency.” 

That said, the thesis did not statistically compare performance across question types, and binary questions (with only two response options) can inflate apparent accuracy.

However, performance declined for questions that required interpretive synthesis or classification across multiple categories. “When the evidence was nuanced or implicitly reported, the AI might capture the general idea, but would miss important contextual details or misclassify outcomes,” she explains.

Scaling up without sacrificing quality

Supervised by PiNC Lab Lead Petra Holden and lab postdoctoral researcher Tsikai Chinembiri, Mazibuko’s research adds to a growing body of work exploring responsible and transparent uses of AI in environmental science.

“Luyanda’s thesis highlights the need for larger, more detailed studies on how AI can support evidence synthesis, especially when evidence is diverse in format and methods,” says Holden. “Dedicated funding is needed to test transparent, reproducible, real-time AI-enabled synthesis for researchers, practitioners, and policymakers.”

While future research could expand Mazibuko’s study to larger datasets, refine codebooks to reduce ambiguity, and test how different prompt designs affect AI performance, she doesn’t think AI will replace climate scientists anytime soon.

Instead, she suggests a hybrid AI-human workflow, where it supports researchers to streamline repetitive processes, while leaving interpretation, context and critical judgment firmly in human hands.

“At a time when climate decisions need to be made quickly, we must think carefully about tools that can help scale up evidence synthesis without sacrificing quality,” says Mazibuko. “If we want evidence to meaningfully inform climate decisions, we need tools that help us work faster, but also more carefully.”