LLM Evaluation & Alignment

Three threads from Microsoft AI. Rubric-based LLM-judge evaluation with human-in-the-loop metrics for Copilot's retrieval-augmented generation; post-training for multimodal retrieval models (Qwen-VL, CLIP-style); and an LLM teacher distilled into a lightweight E5 retriever that shipped after A/B testing.

2023–2026 · Microsoft AI evaluation · RAG · multimodal post-training · distillation
+15.6%
Recall@10 on Copilot RAG retrieval
+6%
overall answer quality
+28–34%
AUC from distilled student, shipped after A/B

1. Evaluation and alignment loop for Copilot RAG

The problem. A RAG-based Copilot can go wrong at every link in the chain. The generator can produce a wrong or ungrounded answer even when it is handed the right documents. The retriever can miss the right documents, or surface the wrong ones with confidence. And the source documents themselves can be wrong or out of date, so the fix is to update the knowledge, not the model. Each of these fails quietly and looks like the others from the outside. You need an evaluator that can tell them apart and run on every change, and you need humans in the loop to tell you when the evaluator itself is wrong.

What I built.

  • Rubric-based LLM-judge evaluation. GPT-based reward-proxy evaluation with chain-of-thought prompting, decomposed into modular rubrics so groundedness, instruction following, and hallucination are each scored on their own and can gate a release.
  • Human-in-the-loop metrics and eval sets. Curated high-quality, human-reviewed evaluation sets, defined the metrics that human review and the judge share, and wired the core metrics into real-time dashboards so regressions showed up the same day. After launch, user feedback from production fed the same loop and drove many further iterations of both the system and the evaluation sets.
  • Retrieval and product improvements. Using that evaluation loop, improved the retrieval algorithm (+15.6% Recall@10, +6% overall answer quality) and contributed production LLM features: multi-turn conversation, user-intent rewriting, and frustration detection.
Human-reviewed eval sets define and calibrate the rubrics RAG candidate retriever / prompt change Rubric judges CoT reward proxies Dashboards gate the release Ship to users User feedback from production iterate the system refresh the eval sets
Human-reviewed eval sets set up the rubric judges; every change is judged and gated before it ships; user feedback from production then iterates both the system and the eval sets.

2. Post-training for multimodal retrieval

Built distributed PyTorch fine-tuning and evaluation pipelines for Qwen-VL and CLIP-style models (CLIP, SigLIP, Perception Encoder) across five multimodal retrieval and reranking tasks, and led the post-training that improved AUC by up to 12%.

3. Teacher-to-student distillation for text retrieval

Post-trained an LLM teacher for text-to-text retrieval and distilled it into a lightweight E5 student: +28–34% AUC over the production baseline, and −6.2% defect rate in production A/B tests, after which the student shipped.

What I did. The multimodal retrieval work and the distillation work were my projects end to end. On Copilot RAG, I owned the evaluation pipeline, from judge design to human-in-the-loop metrics and dashboards, while other teammates developed the product features.