Weicai Ye 叶伟才

I am currently a Senior Researcher at Visual Generation Group (a.k.a. Kling Team), Kuaishou Technology, working on Multimodal Generative Foundation Models and World Models.

Previously, I was a visiting researcher at ETH Zurich, advised by Prof. Marc Pollefeys. I received my Ph.D. in 2024 from ZJU3DV, State Key Lab of CAD&CG, Zhejiang University, advised by Prof. Hujun Bao and Prof. Guofeng Zhang. I obtained my bachelor's degree from UESTC in 2018.

My research goal is to make computers (robotics) learn to perceive, localize, reconstruct, reason, and interact with the real world like humans, that is AGI. I'm interested in Multimodal Video Generation, World Models, 3D Vision Foundation Models, and Embodied AI, especially correspondence, 3D/4D reconstruction, rendering, generation, and robotics manipulation.

We are actively looking for research interns who have strong backgrounds in MLLM, Audio, Video Generation, 3D Vision, etc., to work on cutting-edge research topics. Feel free to email me if you are interested.

Email  /  Google Scholar  /  Twitter  /  Github

• [2025-04] NeRF-Det++ accepted to TIP 2025.
• [2025-03] Invited by VALSE Webinar, I give a talk "Multi-modal Video Generative Foundation Models" about our recent work FullDiT and Any2Caption.
• [2025-03] StreetSurfGS accepted to TCSVT 2025 (corresponding author).
• [2025-02] Three papers (PhysFlow, SketchVideo, Splatter-360) accepted to CVPR 2025 (one corresponding author).
• [2025-01] DGTR accepted to ICRA 2025 and VIPeR accepted to RA-L 2025.
• [2025-01] Five papers (Where Am I and What Will I See, MeshAnything, Lumina-T2X, ND-SDF, HiSplat) (including 2 Spotlight) accepted to ICLR 2025 (100% acceptance rate, two corresponding author). Congratulations to all my collaborators!
• [2025-01] Welcome to check out our Elements feature (that is: multi-reference images to video generation) in the Kling app/web.
• [2024-12] GigaGS accepted to AAAI 2025 (corresponding author).
• [2024-11] PGSR accepted to TVCG 2024.
• [2024-10] Excited to join Kling Team, Kuaishou Technology, as a senior researcher.
• [2024-09] Defended my Ph.D. thesis with 5A rating.
• [2024-09] Three papers (DiffPano, NeuRodin, Point Cloud Matters) accepted to NeurIPS 2024 (1 first-author and 1 corresponding author).

* denotes equal contribution, † denotes corresponding author, ‡ denotes project lead. Representative works are highlighted.

A unified video generative foundation model that seamlessly integrates multiple conditions, significantly reduces parameter overhead, avoids conflicts common in adapter-based methods, and shows scalability and emergent ability in combining diverse, previously unseen modalities.

To address the bottleneck of accurate user intent interpretation within current video generation community, present a novel framework that interprets diverse condition into dense, structured captions for controllable video generation from any condition. Significantly improve video quality across various sota video generation models. Propose large-scale high- quality benchmark dataset for any-condition-to-caption task.

Propose scalable MLMM framework tamed for sign language translation.

Achieved sketch-based spatial and motion control for video generation and support fine-grained editing of real or synthetic videos.

Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation with a differentiable Material Point Method (MPM) and optical flow guidance.

Proposed Generalizable 360∘ Gaussian Splatting for Wide-baseline Panoramic Images

First cloud-edge-device hierarchical framework with Distributed Learning for large-scale high-fidelity surface reconstruction, achieving balance between high-precision reconstruction and low-cost memory.

Proposed a novel auto-regressive framework that jointly addresses spatial localization and view prediction.

MeshAnything mimics human artist in extracting meshes from any 3D representations. It can be combined with various 3D asset production pipelines, such as 3D reconstruction and generation, to convert their results into Artist-Created Meshes that can be seamlessly applied in 3D industry.

Proposed flow-based large diffusion transformers foundation model for transforming text into any modality (image, video, 3D, Audio, music, etc.), resolution, and duration.

Proposed a hierarchical manner in generalizable 3D Gaussian Splatting to construct hierarchical 3D Gaussians via a coarse-to-fine strategy, which significantly enhances reconstruction quality and cross-dataset generalization.

Proposed Normal Deflection fields to represent the angle deviation between the scene normals and the prior normals, achieving smooth surfaces with fine-grained structures, outperforming MonoSDF.

Based on PGSR, proposed photorealistic rendering and efficient high-fidelity large surface reconstruction in a divide-and-conquer manner with LOD structure, outperforming Neuralangelo.

Proposed photorealistic rendering and efficient high-fidelity surface reconstruction model without any pretrained priors, outperforming 3DGS-based (Sugar, 2DGS, Gaussian Opacity Fields, etc.) and SDF-Based methods on T&T, DTU, etc. with faster training. e.g. (ours: only 1 hour vs Neuralangelo 128+ hours)

Proposed scalable and consistent text-to-panorama generation with spherical epipolar-aware diffusion. Established large-scale panoramic video-text datasets with corresponding depth and camera poses. Achieved long-term, consistent, and diverse panoramic scene generation given unseen text and camera poses with SOTA performance.

Identified two main factors of the SDF-based approach that degrade surface quality and proposed a two-stage neural surface reconstruction framework without any pretrained priors, achieving faster training (only 18 GPU hours) and high-fidelity surface reconstruction with fine-grained details, outperforming Neuralangelo on T&T, ScanNet++, etc.

Implied that point cloud observation, or explicit 3D information, matters for robot learning. With point cloud as input, the agent achieved higher mean success rates and exhibited better generalization ability.

Proposed a novel distributed framework for efficient Gaussian reconstruction for sparse-view vast scenes, leveraging feed-forward Gaussian model for fast inference and a global alignment algorithm to ensure geometric consistency.

Based on PGSR, proposed the DynaSurfGS framework, which can facilitate real-time photorealistic rendering and dynamic high-fidelity surface reconstruction, achieving smooth surfaces with meticulous geometry.

Based on PGSR, proposed photorealistic rendering and efficient high-fidelity Large Scene Surface Reconstruction for Urban Street Scenes with Free Camera Trajectories, outperforming F2NeRF.

Proposed a concise, elegant, and robust SfM pipeline with point tracking for smooth camera trajectories and dense pointclouds from casual monocular videos.

Proposed self-supervised dynamic SLAM with Flow Motion Decomposition and DINO Guidance, outperforming DROID-SLAM.

Proposed a visual incremental place recognition method with adaptive mining and lifelong learning

Proposed efficient point cloud backbone with Mamba framework and achieved SOTA performance.

Incorporating semantic cues and perspective-aware depth supervision, NeRF-Det++ outperforms NeRF-Det by +1.9% in mAP@0.25 and +3.5% in mAP@0.50 on ScanNetV2.

Proposed plug-and-play and iterative diffusion refinement framework for robust scene flow estimation. Achieved unprecedented millimeter level accuracy on KITTI, and with 6.7% and 19.1% EPE3D reduction respectively on FlyingThings3D and KITTI 2015.

Introduced intrinsic decomposition into the NeRF-based rendering and performed editable novel view synthesis in room-scale scenes.

Introduced panoptic visual odometry framework to achieve comprehensive modeling of the scene motion, geometry, and panoptic segmentation information.

Proposed a novel dual-flow representation of selfsupervised scene motion decomposition for dynamic dense SLAM

Proposed a novel end-to-end RGB-D SLAM, which adopts a feature-based deep neural tracker as frontend and a NeRF-based neural implicit mapper as the backend.

As a main solution for feature matching and visual localization, integrated into OpenXRLab.

Achieved SOTA performance on video panoptic segmentation from two perspectives: feature space (Instance Tracker) and spatial location (Pixel Tracker).

Proposed an efficient system named Coxgraph for multi-robot collaborative dense reconstruction in real-time. To facilitate transmission, we propose a compact 3D representation which transforms the SDF submap to mesh packs.

Proposed a multi-device integrated cargo loading management system with AR, which monitors cargoes by fusing perceptual information from multiple devices in real-time.

Introduced robust plane matching in texture-less scenes and achieved SOTA performance in image-based localization.

Proposed a novel saliency guided subdivision method to achieve the trade-off between detail generation and memory consumption. Our method can both produce visually pleasing mesh reconstruction results with fine details and achieve better performance.

Proposed bipartite graph network with Hungarian pooling layer to deal with 2D-3D matching, which can find more correct matches and improves localization on both the robustness and accuracy.

Design and source code from Jon Barron's website