Haoyu Zhao

Continual learning is the ability of LLM agents to improve over time by acquiring, refining, and applying knowledge from experience while retaining useful prior knowledge. It is essential for agents learning over long time horizons, yet current evaluations largely assess individual task performance rather than improvement through accumulated experience. We introduce Continual Learning Exam (CLE), a large-scale benchmark developed by 50+ domain experts to measure experience-driven improvement in realistic professional workflows. CLE comprises 120 human-authored tasks, each organised as a five-level ladder, yielding 600 levels spanning 89 professional roles across 12 occupational domains. Agents must pass each level to unlock the next, with later levels building on experience accumulated during earlier assignments. An evaluation agent assesses each submission against human-authored rubrics. We compare a single-submission setting with one that provides diagnostic feedback after failure and allows one revision. Across 17 harness-model configurations using frontier LLMs, feedback and revision yield a median completion gain of 5.3 percentage points. Yet the best-performing system clears only 47.7% of levels and completes all five levels in just 23.3% of tasks, highlighting the difficulty of sustained learning from experience for existing frontier LLMs. As such, the dataset can also be used to post-train LLMs to strengthen their continual learning capabilities.

@misc{zhao2026continuallearningexambenchmarking,
  title={Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents},
  author={Haoyu Zhao and Sitian Tu and Yuxuan Huang and Huichi Zhou and Zhangyue Yin and Jiahao Huang and Jingyuan Xu and Tianmiao Shi and Jin Shi and Nianxi Zhang and Yangmiao Zhang and Zhuolang Chan and Huiyan Hu and Chenchao Sheng and Yaqing Luo and Yue Chen and Huaxiu Yao and Meng Fang and Jun Wang},
  year={2026},
  eprint={2610.33956},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2610.33956},
}

Couldn’t copy automatically. Select the citation and press ⌘C or Ctrl+C.

Learning to act in unfamiliar environments requires agents to infer how the world works and revise that understanding as new evidence arrives. Yet limited observations can support multiple world models that explain past interactions but predict different outcomes in unseen states. We introduce Memento 3, building on the Memento series to enable frozen LLM agents to continually learn explicit world models through external memory. The agent maintains a natural-language rulebook as persistent semantic memory, recording revisable hypotheses about environment dynamics while leaving unknown aspects underspecified. It compiles this rulebook into executable code for prediction and planning. Through a continual loop of observation, reflection, rule revision, compilation, and verification, the agent uses prediction errors to refine both the rulebook and its code. Updated code is accepted only when the LLM judges it faithful to the rulebook and cell-exact replay reproduces the observed transitions. We investigate this process as a model-based route to recursive self-improvement (RSI): the agent autonomously explores the environment, revises its world model, and uses verified updates to guide subsequent interaction and learning, while the underlying LLM remains fixed. A population extension maintains multiple world models in parallel, sharing interaction evidence and using their predictions to guide exploration. On ARC-AGI-3, the single-model agent clears every level of all 25 public games, achieves a mean Relative Human Action Efficiency (RHAE) of 100.0, and uses 44% of the human action count. In an Atari Pong case study, a learned feedback controller wins 21:0 in each of three evaluated episodes with different openings, without further LLM calls.

@misc{zhao2026memento3modelbasedrecursive,
  title={Memento 3: Model-Based Recursive Self-Improvement through Reflective Rulebooks},
  author={Haoyu Zhao and Zhengxu Yu and Zhiyuan He and Meng Fang and Rasul Tutunov and Haitham Bou-Ammar and Weilin Luo and Jun Wang},
  year={2026},
  eprint={2610.11794},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2610.11794},
}

Couldn’t copy automatically. Select the citation and press ⌘C or Ctrl+C.

Recent advances in Multimodal Large Language Models (MLLMs) have enabled their use as intelligent agents for smartphone operation. However, existing methods depend on the Android Debug Bridge (ADB) for data transmission and action execution, limiting their applicability to Android devices. In this work, we introduce the novel Embodied Smartphone Operation (ESO) task and present See-Control, a framework that enables smartphone operation via direct physical interaction with a low-DoF robotic arm, offering a platform-agnostic solution. See-Control comprises three key components: (1) an ESO benchmark with 155 tasks and corresponding evaluation metrics; (2) an MLLM-based embodied agent that generates robotic control commands without requiring ADB or system back-end access; and (3) a richly annotated dataset of operation episodes, offering valuable resources for future research. By bridging the gap between digital agents and the physical world, See-Control provides a concrete step toward enabling home robots to perform smartphone-dependent tasks in realistic environments.

@misc{zhao2025seecontrolmultimodalagentframework,
  title={See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm},
  author={Haoyu Zhao and Weizhong Ding and Yuhao Yang and Zheng Tian and Linyi Yang and Kun Shao and Jun Wang},
  year={2025},
  eprint={2512.08629},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2512.08629},
}

Couldn’t copy automatically. Select the citation and press ⌘C or Ctrl+C.

Rapid advances in multimodal models demand benchmarks that rigorously evaluate understanding and reasoning in safety-critical, dynamic real-world settings. We present AccidentBench, a large-scale benchmark that combines vehicle accident scenarios with Beyond domains, safety-critical settings in air and water that emphasize spatial and temporal reasoning (e.g., navigation, orientation, multi-vehicle motion). The benchmark contains approximately 2000 videos and over 19000 human-annotated question--answer pairs spanning multiple video lengths (short/medium/long) and difficulty levels (easy/medium/hard). Tasks systematically probe core capabilities: temporal, spatial, and intent understanding and reasoning. By unifying accident-centric traffic scenes with broader safety-critical scenarios in air and water, AccidentBench offers a comprehensive, physically grounded testbed for evaluating models under real-world variability. Evaluations of state-of-the-art models (e.g., Gemini-2.5 Pro and GPT-5) show that even the strongest models achieve only about 18% accuracy on the hardest tasks and longest videos, revealing substantial gaps in real-world temporal, spatial, and intent reasoning. AccidentBench is designed to expose these critical gaps and drive the development of multimodal models that are safer, more robust, and better aligned with real-world safety-critical challenges. The code and dataset are available at: https://github.com/SafeRL-Lab/AccidentBench

@misc{gu2025accidentbenchbenchmarkingmultimodalunderstanding,
  title={AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond},
  author={Shangding Gu and Xiaohan Wang and Donghao Ying and Haoyu Zhao and Runing Yang and Ming Jin and Boyi Li and Marco Pavone and Serena Yeung-Levy and Jun Wang and Dawn Song and Costas Spanos},
  year={2025},
  eprint={2509.26636},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2509.26636},
}

Couldn’t copy automatically. Select the citation and press ⌘C or Ctrl+C.

Estimating physical properties for visual data is a crucial task in computer vision, graphics, and robotics, underpinning applications such as augmented reality, physical simulation, and robotic grasping. However, this area remains under-explored due to the inherent ambiguities in physical property estimation. To address these challenges, we introduce GaussianProperty, a training-free framework that assigns physical properties of materials to 3D Gaussians. Specifically, we integrate the segmentation capability of SAM with the recognition capability of GPT-4V(ision) to formulate a global-local physical property reasoning module for 2D images. Then we project the physical properties from multi-view 2D images to 3D Gaussians using a voting strategy. We demonstrate that 3D Gaussians with physical property annotations enable applications in physics-based dynamic simulation and robotic grasping. For physics-based dynamic simulation, we leverage the Material Point Method (MPM) for realistic dynamic simulation. For robot grasping, we develop a grasping force prediction strategy that estimates a safe force range required for object grasping based on the estimated physical properties. Extensive experiments on material segmentation, physics-based dynamic simulation, and robotic grasping validate the effectiveness of our proposed method, highlighting its crucial role in understanding physical properties from visual data.

@InProceedings{Xu_2025_ICCV,
    author    = {Xu, Xinli and Ge, Wenhang and Qiu, Dicong and Chen, ZhiFei and Yan, Dongyu and Liu, Zhuoyun and Zhao, Haoyu and Zhao, Hanfeng and Zhang, Shunsi and Liang, Junwei and Chen, Ying-Cong},
    title     = {GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2025},
    pages     = {7231-7240}
}

Couldn’t copy automatically. Select the citation and press ⌘C or Ctrl+C.

Visual grounding is an essential tool that links user-provided text queries with query-specific regions within an image. Despite advancements in visual grounding models, their ability to comprehend complex queries remains limited. To overcome this limitation, we introduce LLM-Optic, an innovative method that utilizes Large Language Models (LLMs) as an optical lens to enhance existing visual grounding models in comprehending complex text queries involving intricate text structures, multiple objects, or object spatial relationships, situations that current models struggle with. LLM-Optic first employs an LLM as a Text Grounder to interpret complex text queries and accurately identify objects the user intends to locate. Then a pre-trained visual grounding model is used to generate candidate bounding boxes given the refined query by the Text Grounder. After that, LLM-Optic annotates the candidate bounding boxes with numerical marks to establish a connection between text and specific image regions, thereby linking two distinct modalities. Finally, it employs a Large Multimodal Model (LMM) as a Visual Grounder to select the marked candidate objects that best correspond to the original text query. Through LLM-Optic, we have achieved universal visual grounding, which allows for the detection of arbitrary objects specified by arbitrary human language input. Importantly, our method achieves this enhancement without requiring additional training or fine-tuning. Extensive experiments across various challenging benchmarks demonstrate that LLM-Optic achieves state-of-the-art zero-shot visual grounding capabilities. Project Page: https://haoyu-zhao.github.io/LLM-Optic.github.io/ .

@misc{zhao2024llmopticunveilingcapabilitieslarge,
  title={LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding},
  author={Haoyu Zhao and Wenhang Ge and Ying-cong Chen},
  year={2024},
  eprint={2405.17104},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2405.17104},
}

Couldn’t copy automatically. Select the citation and press ⌘C or Ctrl+C.

Neural implicit surface learning has shown significant progress in multi-view 3D reconstruction, where an object is represented by multilayer perceptrons that provide continuous implicit surface representation and view-dependent radiance. However, current methods often fail to accurately reconstruct reflective surfaces, leading to severe ambiguity. To overcome this issue, we propose Ref-NeuS, which aims to reduce ambiguity by attenuating the effect of reflective surfaces. Specifically, we utilize an anomaly detector to estimate an explicit reflection score with the guidance of multi-view context to localize reflective surfaces. Afterward, we design a reflection-aware photometric loss that adaptively reduces ambiguity by modeling rendered color as a Gaussian distribution, with the reflection score representing the variance. We show that together with a reflection direction-dependent radiance, our model achieves high-quality surface reconstruction on reflective surfaces and outperforms the state-of-the-arts by a large margin. Besides, our model is also comparable on general surfaces.

@InProceedings{Ge_2023_ICCV,
    author    = {Ge, Wenhang and Hu, Tao and Zhao, Haoyu and Liu, Shu and Chen, Ying-Cong},
    title     = {Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {4251-4260}
}

Couldn’t copy automatically. Select the citation and press ⌘C or Ctrl+C.

Team composition is one of the most important and challenging directions in the recommendation problem. Compared with a single person, the advantage of a team is mainly reflected in the synergy of team members’ complementary collaboration. To build a high-efficiency team, how to choose the team members has become a tricky problem. However, there is a lack of quantitative algorithms and validation methods for team member selection. In this paper, we put forward three indicators to measure a team’s ability and formulate the selection of football team members as a multi-objective optimization problem. Subsequently, an evolutionary player selection algorithm based on the genetic algorithm is proposed to solve the team composition problem. We verify the effectiveness of the team member recommendation algorithm via data analysis, football game simulation under different budget constraints and provide comparisons with existing methods.

@article{Zhao_2021,
  title={Multi-Objective Optimization for Football Team Member Selection},
  volume={9},
  ISSN={2169-3536},
  url={http://dx.doi.org/10.1109/ACCESS.2021.3091185},
  DOI={10.1109/access.2021.3091185},
  journal={IEEE Access},
  publisher={Institute of Electrical and Electronics Engineers (IEEE)},
  author={Zhao, Haoyu and Chen, Haihui and Yu, Shenbao and Chen, Bilian},
  year={2021},
  pages={90475--90487}
}

Couldn’t copy automatically. Select the citation and press ⌘C or Ctrl+C.