Hi, my name is Robert!

Introduction

Senior Software Engineer @ Microsoft, Azure AI Search — building the vector search, agentic retrieval, and RAG systems behind enterprise AI.


Hi, my name is Robert! I'm a Senior Software Engineer on Microsoft's Azure AI Search team, where I build the retrieval engine behind enterprise AI. My work sits at the intersection of information retrieval, vector search, and applied machine learning: improving relevance for complex global queries, grounding LLMs and multi-agent workflows in governed enterprise knowledge, and turning research prototypes into production-grade agentic retrieval and RAG capabilities.

Earlier, as a Software Engineer II, I delivered the core of our vector engine — including quantization techniques that cut customer costs by 8–32× and latency by up to 20×, a hybrid-search relevance stack, and an extensible facet-aggregation engine built on formal grammar and parsing. I care deeply about performance, correctness, and distributed systems, and I'm often the engineer who root-causes the gnarliest production incidents.

Beyond shipping code, I love building communities. I graduated with a B.Eng in Electrical & Computer Engineering (97% CGPA, 20+ awards worth over $100K), drove a 200+ person conference from vision to reality, and grew a Senior's Program to 180+ volunteers and 650+ participants while delivering technical talks to 250+ engineering students. I also led a student team to train a deep neural network for human pose estimation from randomly initialized weights — check it out on Streamlit.

Please reach out if you'd like to connect!

Portrait of Robert Lee

Work Experience

Senior Software Engineer, Microsoft Azure AI Search

Mar. 2025 - Present | Redmond, WA

C++, C#, Java, Python | Vector Search, Agentic Retrieval, Information Retrieval, RAG, Distributed Systems

- Driving a novel vector-search diversity capability end-to-end: with no existing blueprint, owning scoping, surfacing engine-level constraints, and proposing and refining the design - aimed at improving relevance for complex global queries through diversity-aware sampling that broadens retrieved evidence for higher-quality answer synthesis across enterprise indexes holding billions of vectors.
- Led a cross-team effort to bridge research and production, defining feature scope, data-driven ship criteria, and validation methodology to move multiple agentic-retrieval quality improvements from prototype to shipped Public Preview features.
- Integrated Azure AI Search retrieval into agent workflows - grounding LLM agents in governed, indexed enterprise knowledge across tool calling, multi-agent orchestration, and Retrieval Augmented Generation.
- Architected a benchmarking system from scratch to profile highly variable agentic-retrieval workloads across CPU, memory, and throughput - modeling latency distributions, tool-calling iterations, and dependency patterns to produce the production billing model that enabled a new serverless enterprise search offering to launch; surfaced and resolved memory optimizations along the way.
- Serve as a technical leader at the intersection of information retrieval, vector search, Azure OpenAI, and agent orchestration - mentoring and onboarding engineers, leading design reviews, and driving deeply technical architecture decisions.

Microsoft logo

Software Engineer II, Microsoft Azure AI Search

Jun. 2022 - Feb. 2025 | Redmond, WA

C++, C#, Java, Python | Vector Search, Vector Quantization, Information Retrieval, RAG, Distributed Systems

Drove vector storage, quantization, and relevance for Azure AI Search's vector retrieval engine, spanning billions of embeddings.

- Drove vector quantization from Public Preview through to GA - now widely adopted by customers - delivering 8-32x cost savings and up to 20x latency reduction through binary vectors, scalar and binary quantization, and SIMD-accelerated distance computation.
- Strengthened the relevance stack by designing hybrid-search subscore fusion and score thresholding to improve result quality across blended vector and keyword retrieval.
- Designed and shipped a data-driven quota-enforcement mechanism for HNSW indexes tied to physical resource utilization, cutting limit overshoot by 100x through cross-team design collaboration.
- Owned end-to-end delivery of an extensible facet-aggregation engine from spec, building a custom lexer-parser-evaluator that applies BNF grammar, the shunting-yard algorithm, and Reverse Polish Notation to parse, simplify, and validate expressions - backed by extensive A/B test coverage.
- Hardened the vector search engine with a new test suite that caught a critical bug in a new quantization algorithm before release.
- Acted as a subject-matter expert and key reviewer - root-causing deeply technical production incidents across teams to rapidly restore customer service and drive durable fixes, while raising reliability through rigorous review of distributed-systems and vector-algorithm changes.

Microsoft logo

Software Engineer, Microsoft Azure AI Search

Jul. 2021 - May 2022 | Redmond, WA

- Delivered the new index alias feature to Public Preview, letting customers repoint applications to a different underlying index with no code changes or downtime.
- Modernized a core telemetry-database table through a phased, zero-downtime migration that preserved data integrity and delivered a 50-100x query speedup.

Microsoft logo

Projects

Human pose estimation predicting skeleton keypoints on a skateboarder

Human Pose Estimation using Deep Neural Networks

Jan. 2021 - May 2021

Project and Team Lead for a student team to develop a deep neural network for human pose estimation (HPE) on the COCO-2017 dataset.

I architected our cloud training pipeline, model architecture, and data augmentation method. I led work in model visualization and deployment.

Our model achieves very good performance on almost all images, provided the person is relatively centered and about 60-95% of the vertical image height (these are limitations we chose due to the tight time constraints of the project). It struggles in some extremely difficult images, typically with highly overlapped people or heavily occluded joints.

The model performs single-person single-image 2D HPE with a kinematic model and a heatmap-based approach for joint prediction rather than a regression on joint locations. The model was trained from scratch with random weight initializations, and our team developed much of the plumbing from scratch as well.

Semantic road segmentation model output overlaid on a street scene

Semantic Road Segmentation using Convolutional Neural Networks

Jul. 2020 - Sep. 2020

- Successfully trained a U-Net CNN on the KITTI Road dataset using Keras, achieving up to 99.1% F1 score and 91% in the worst case.
- Designed and implemented the full training pipeline, including data generator, training/testing scripts, custom loss functions, and cloud training setup using Google Colab.
- Tuned hyperparameters and used data augmentation to improve model generalization under varied conditions.
- Researched and designed the network architecture by analyzing state-of-the-art computer vision literature.
- Resolved architectural and training bugs in the decoder and segmentation logic to ensure stable performance.
- Evaluated model using pixel-based metrics and addressed edge cases like shadows, occlusions, and road artifacts.

Monocular depth estimation output as a colorized depth map

Computer Vision Project on Monocular Depth Estimation

Jan. 2020 - May 2020

- Implemented a limited version of Monodepth2, a self-supervised monocular depth estimation algorithm, in TensorFlow 1.13.1, replacing the original PyTorch implementation to improve project feasibility.
- Adapted the model to the DrivingStereo dataset, a large-scale dataset (~136k training image pairs), to enhance depth prediction generalization compared to the smaller KITTI dataset.
- Designed and trained a fully convolutional U-Net with a ResNet-18 encoder, optimized for stereo image inputs at 640×192 resolution.
- Integrated self-supervised learning using stereo photometric reconstruction and edge-aware smoothness loss without requiring ground truth depth data.
- Implemented multi-scale training loss, auto-masking for static pixels, and per-pixel minimum reprojection to improve depth estimation near occlusions and motion boundaries.

Battlesnake game board

Battlesnake Reinforcement Learning AI Design

Jan. 2019 - May 2019

- Challenge: Classic snake game mechanics, but with up to 7 opponent snakes. Collisions with longer snakes or starvation result in death. Design algorithm to control snake in real-time game. Goal: Survive the longest.
- Trained keras-rl reinforcement learning model with a combination of self-play and publicly available snakes.

Contact Me

View of an airplane wing above the clouds

I'd love to connect, so please feel free to reach me on any of these platforms.





© Robert Lee | Made with ❤ with HTML, CSS & JavaScript. Template from Rosacodes