Beyond chat demos: evaluating LLM systems at scale with DeepEval
World Bank Open Data blog post on an automated, persona-driven evaluation framework for multi-turn LLM assistants, built on DeepEval.
Ideas in public
Writing
Papers, methods, and field notes on AI, research, and building useful systems.
7 pieces Newest first
armada: orchestrating AI agent teams with evidence gates
Introducing armada, an open-source tool that splits coding work into evidence-gated phases and dispatches specialist subagents.
Field Order Should Not Matter: Permutation-Invariant Embedding Model Fine-Tuning for Structured Metadata Retrieval
Permutation-invariant fine-tuning approach for embedding models applied to structured metadata retrieval.
AI for Monitoring and Classifying Data Used in Research Literature
Methods for AI-assisted monitoring of how datasets appear in published research, including classification of usage context.
Tracking Dataset Use at Scale with AI: How Synthetic Data Helps
World Bank Open Data blog post on using synthetic data and AI to track dataset citations at scale.
Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers
Workshop paper at ICLR 2025 using LLMs and synthetic data to detect and classify dataset citations in research literature.
Climate Change AI track paper (ICLR 2025)
Climate Change AI track paper presented at ICLR 2025, addressing dataset monitoring in climate research.