Evaluating policy interventions for displaced populations requires authentic stakeholder perspectives, yet traditional methods prove inadequate during humanitarian crises. This paper presents a meta-evaluation of an AI-powered tool utilizing Retrieval-Augmented Generation technology to process 521,089 Telegram messages from Ukrainian and Russian-speaking populations. Our central finding challenges initial design assumptions: while intended to simulate stakeholder perspectives, the tool proved most valuable as a document exploration platform for qualitative data analysis. This shift from simulation to exploration represents a significant methodological insight for evaluation practice. The technical architecture successfully implemented multi-stage filtering, hierarchical clustering into 249 thematic groups, and transparent retrieval mechanisms. We argue that AI technologies offer greatest promise not in replacing stakeholder engagement, but in enhancing evaluators’ capacity to systematically process qualitative data. This research contributes to debates on responsible AI integration in evaluation methodology.
