From linguistic evaluation to mechanistic verification: testing LLM-generated farm recommendations

João Serra, Franca Giannini-Kurina, David Kraus, Meshach Ojo Aderele, Michal Antala, Sheng Wang, Jaber Rahimi, Claire Treat, Sevval Frank, Mary Grace Barbacias, Tom Cripps, Morten Graversgaard, Klaus Butterbach-Bahl, Diego Abalos, Vasilis Michailidis

Large language models (LLM) are increasingly used to generate farm-management advice, but their biophysical consequences remain largely unverified. We introduce a process-based verification framework that combines management portfolios generated by ChatGPT and Claude with the process-based model LandscapeDNDC across 11 contrasting agroecosystems. The LLMs produced agronomically plausible interventions, privileging changes in fertiliser timing, splitting and rates, and generally preserved crop yields. However, their agreement was markedly weaker for environmental targets like nitrogen losses and soil organic carbon. LLMs predicted the direction of agri-environmental change more reliably than its magnitude: directional agreement averaged 86%, whereas only 49% of simulated responses fell within the expected ranges. Portfolios that targeted several targets simultaneously rarely performed consistently across sites. Our results show that process-based models can screen AI-generated farm recommendations for environmental burden shifting before they are used in practice.