Heng (Henry) Yu


I am a master student (MSR) at CMU Robotics Institute, where I work on 3D vision advised by Prof. Laszlo Jeni . I also closely collaborated with Prof. Berkin Bilgic from Harvard Medical School on MRI reconstruction topics and Prof. Cheng Jin from Institute of Medical Robotics, Shanghai Jiao Tong University on medical vision topics.

I obtained my Bachelor's degree from Tsinghua University with a major in Department of Automation , and a second major in School of Economics and Management . During my time at Tsinghua, I worked on computer vision and medical image analysis research under the supervision of Prof. Jie Zhou and Prof. Jianjiang Feng in Intelligent Vision Group . After that, I co-founded Tsingh Technology focused on intelligent logistics with three PhDs. I used to have a few good times at Nebula Link Technology and Sangfor Technologies Inc.

I also worked with Prof. Kris Kitani from CMU RI, Prof. Kawin Setsompop and Prof. Ruijiang Li from Stanford Medicine. My research interest is computer vision and its applications.

CV  /  Google Scholar  /  Github  /  LinkedIn

profile photo
  • Feb 2023: DyLiN paper is accepted by CVPR2023, see you in Vancouver!
  • Feb 2023: SubZero abstract is accepted by ISMRM2023 as a power pitch, see you in Toronto!
  • Dec 2022: CoNFies paper has been nominated as a best paper candidate!
  • Nov 2022: Non-pooling Network paper won Best Paper Award at MICAD 2022!
  • Nov 2022: Won gold medal at the 8th China International College Students' 'Internet+' Innovation and Entrepreneurship Competition!
  • Sep 2022: CoNFies paper is accepted by FG2023, see you in Hawaii!
  • May 2022: I am joining Fujitsu this summer as a machine learning intern, working on controllable NeRF.
  • Feb 2022: One paper is accepted by Magnetic Resonance in Medicine!
  • Aug 2021: Start my graduate study at CMU RI!
  • May 2021: I am joining Sangfor this summer as a machine learning engineer, working on evading web application firewalls with reinforcement learning.
  • Mar 2021: One paper is accepted by Nature Communications!
  • Feb 2021: eRAKI abstract is accepted by ISMRM2021 as an oral!
  • Apr 2020: MixModule paper is accepted by ISBI2020!
  • Nov 2019: One paper is accepted by Annals of Surgery!


I have rich experiences in computer vision, MRI reconstruction and medical image analysis. I would like to explore the possibility of AI technology and its applications (e.g. 3D Scene Understanding , Healthcare, etc).


Reviewer: CVPR, ECCV, NeurIPS, MICCAI, ISBI, Computer Graphics Forum, ISMRM

Selected Publications

* refers to co-first author. Please refer to my google scholar for more details.

clean-usnob CoGS: Controllable Gaussian Splatting
Heng Yu, Joel Julin, Zoltan Adam Milacski, Koichiro Niinuma, László A. Jeni,
In Submission
paper / project page / code

We present CoGS, a method for Controllable Gaussian Splatting, that enables the direct manipulation of scene elements, offering real-time control of dynamic scenes without the prerequisite of pre-computing control signals.

clean-usnob DyLiN: Making Light Field Networks Dynamic
Heng Yu, Joel Julin, Zoltan Adam Milacski, Koichiro Niinuma, László A. Jeni,
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023
paper / project page / code / CMU RI News

We propose propose the Dynamic Light Field Network (DyLiN) method that can handle non-rigid deformations, including topological changes, which outperformed state-of-the art methods in terms of visual fidelity and compute complexity.

clean-usnob Unsupervised Style-based Explicit 3D Face Reconstruction from Single Image
Heng Yu, Zoltan Adam Milacski, László A. Jeni,
CVPR workshop, 2023

We propose a general adversarial learning framework for solving Unsupervised 2D to Explicit 3D Style Transfer.

clean-usnob CoNFies: Controllable Neural Face Avatars
Heng Yu, Koichiro Niinuma László A. Jeni,
International Conference on Automatic Face and Gesture Recognition (FG), 2023 - Best Paper Award Finalist
paper / project page / code

We propose a fully-automatic controllable neural representation for face self-portraits.

clean-usnob SubZero: Subspace Zero-Shot MRI Reconstruction
Heng Yu, Yamin Arefeen, Berkin Bilgic
Proceedings of the 31th Annual Meeting of ISMRM, 2023 - Power Pitch
paper / code

We propose a parallel network framework and introduce an attention mechanism to improve subspace-based zero-shot self-supervised learning and enable higher acceleration factors.

clean-usnob Scan-specific artifact reduction in k-space (SPARK) neural networks synergize with physics-based reconstruction to accelerate MRI
Yamin Arefeen, Onur Beker, Jaejin Cho, Heng Yu, Elfar Adalsteinsson, Berkin Bilgic
Magnetic Resonance in Medicine (MRM), 2022
paper / code

We develop a scan-specific model that estimates and corrects k-space errors made when reconstructing accelerated MRI data.

clean-usnob eRAKI: Fast Robust Artificial neural networks for K-space Interpolation (RAKI) with Coil Combination and Joint Reconstruction
Heng Yu, Zijing Dong, Yamin Arefeen, Congyu Liao, Kawin Setsompop, Berkin Bilgic
Proceedings of the 29th Annual Meeting of ISMRM, 2021 - Oral Presentation
paper / code

We accelerate RAKI by more than 200 times by directly learning a coil-combined target.

clean-usnob Predicting treatment response from longitudinal images using multi-task deep learning
Cheng Jin*, Heng Yu*, Jia Ke*, Peirong Ding*, Yongju Yi, Xiaofeng Jiang, Xin Duan, Jinghua Tang, Daniel T. Chang, Xiaojian Wu, Feng Gao, Ruijiang Li
Nature Communications, 2021
paper / code

We present a multi-task deep learning approach that allows simultaneous tumor segmentation and response prediction of pathologic complete response after neoadjuvant chemoradiotherapy.

clean-usnob MixModule: Mixed CNN Kernel Module for Medical Image Segmentation
Heng Yu, Xue Feng, Ziwen Wang, Hao Sun
IEEE 17th International Symposium on Biomedical Imaging (ISBI), 2020
paper / code

We use mixed kernels to improve the performance of existing medical image segmentation networks.

clean-usnob Development and validation of a deep learning CT signature to predict survival and chemotherapy benefit in gastric cancer: a multicenter, retrospective study.
Yuming Jiang*, Cheng Jin*, Heng Yu*, Jia Wu*, Chuanli Chen, Qingyu Yuan, Weicai Huang, Yanfeng Hu, Yikai Xu, Zhiwei Zhou, George A. Fisher Jr., Guoxin Li, Ruijiang Li
Annals of surgery, 2020
paper / code

We propose a novel deep neural network (S-net) to construct a CT signature for predicting disease-free survival (DFS) and overall survival.

clean-usnob Deep learning analysis of the primary tumour and the prediction of lymph node metastases in gastric cancer.
C Jin*, Y Jiang*, H Yu*, W Wang, B Li, C Chen, Q Yuan, Y Hu, Y Xu, Z Zhou, G Li, R Li
British Journal of Surgery, 2020
paper / code

We develop a deep learning system for predicting lymph node metastasis in multiple nodal stations based on preoperative CT images in patients with gastric cancer.

clean-usnob SmartPartNet: Part-Informed Person Detection for Body-Worn Smartphones.
Heng Yu, Eshed Ohn-Bar, Donghyun Yoo, Kris Kitani
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2018

We develop an image-based person detection algorithm for wearable computing using commodity smartphones.

clean-usnob Left atrial appendage segmentation using fully convolutional neural networks and modified three-dimensional conditional random fields.
Cheng Jin, Jianjiang Feng , Lei Wang, Heng Yu, Jiang Liu, Jiwen Lu, Jie Zhou
IEEE Journal of Biomedical and Health Informatics (JBHI), 2018

We propose a robust method for automatic left atrial appendage segmentation on computed tomographic angiography data using fully convolutional neural networks with 3D conditional random fields.

clean-usnob Detection of Substances in the Left Atrial Appendage by Spatiotemporal Motion Analysis Based on 4D-CT.
Cheng Jin, Heng Yu, Jianjiang Feng , Lei Wang, Jiwen Lu, Jie Zhou
MICCAI workshop, 2017 - Oral Presentation

we present a new approach for the detection of substances in the left atrial appendage by spatiotemporal motion analysis and make a detailed judgment and analysis of spatial distribution and classification of most objects in the left atrial appendage.

clean-usnob Left atrial appendage neck modeling for closure surgery.
Cheng Jin, Heng Yu, Jianjiang Feng , Lei Wang, Jiwen Lu, Jie Zhou
MICCAI workshop, 2017

We propose a robust method for automatic left atrial appendage segmentation on computed tomographic angiography data using fully convolutional neural networks with 3D conditional random fields.

Website template from Jon Barron.