publications

2022

1. NeurIPS

   

   Learnable Polyphase Sampling for Shift Invariant and Equivariant Convolutional Networks

   Renán A. Rojas-Gómez, Teck-Yian Lim, Alexander G. Schwing, Minh N. Do, and Raymond A. Yeh

   Advances in Neural Information Processing Systems, 2022

   Abs Project Page Code

   We propose learnable polyphase sampling (LPS), a pair of learnable down/upsampling layers that enable truly shift-invariant and equivariant convolutional networks. LPS can be trained end-to-end from data and generalizes existing handcrafted downsampling layers. It is widely applicable as it can be integrated into any convolutional network by replacing down/upsampling layers. We evaluate LPS on image classification and semantic segmentation. Experiments show that LPS is on-par with or outperforms existing methods in both performance and shift consistency. For the first time, we achieve true shift-equivariance on semantic segmentation (PASCAL VOC), i.e., 100% shift consistency, outperforming baselines by an absolute 3.3%.

2021

1. RaDICaL: A synchronized FMCW radar, depth, IMU and RGB camera data dataset with low-level FMCW radar signals

   Teck-Yian Lim, Spencer A Markowitz, and Minh N. Do

   IEEE Journal of Selected Topics in Signal Processing, 2021

   Abs Project Page Code

   Within the autonomous driving community, millimetre-wave frequency-modulated continuous-wave (FMCW) radars are not used to their fullest potential. Classical, hand-designed target detection algorithms are applied in the signal processing chain and the rich contextual information is discarded. This early discarding of information limits what can be applied in algorithms further downstream. In contrast with object detection in camera images, radar has thus been unable to benefit fully from data-driven methods. This work seeks to bridge this gap by providing the community with a diverse, minimally processed FMCW radar dataset that is not only RGB-D (color and depth) aligned but also synchronized with inertial measurement unit (IMU) measurements in the presence of ego-motion. Moreover, having time-synchronized measurements allow for verification, automated or assisted labelling of the radar data, and opens the door for novel methods of fusing the data from a variety of sensors. We present a system that could be built with accessible, off-the-shelf components within a $1000 budget and an accompanying dataset consisting of diverse scenes spanning indoor, urban and highway driving. Finally, we demonstrated the ability to go beyond classical radar object detection with our dataset with a classification accuracy of 85.1% using the low-level radar signals captured by our system, supporting our argument that there is value in retaining the information discarded by current radar pipelines.

2019

1. 

   Radar and Camera Early Fusion for Vehicle Detection in Advanced Driver Assistance Systems

   Teck-Yian Lim, Amin Ansari, Bence Major, Daniel Fontijne, Michael Hamilton, and 2 more authors

   In NeuRIPS2019 ML4AD Workshop, 2019

   Abs Project Page PDF Poster

   The perception module is at the heart of modern Advanced Driver Assistance Systems (ADAS). To improve the quality and robustness of this module, especially in the presence of environmental noises such as varying lighting and weather conditions, fusion of sensors (mainly camera and LiDAR) has been the center of attention in the recent studies. In this paper, we focus on a relatively unexplored area which addresses the early fusion of camera and radar sensors. We feed a minimally processed radar signal to our deep learning architecture along with its corresponding camera frame to enhance the accuracy and robustness of our perception module. Our evaluation, performed on real world data, suggests that the complementary nature of radar and camera signals can be leveraged to reduce the lateral error by 15% when applied to object detection.

2018

1. ICASSP

   Time-frequency networks for audio super-resolution

   Teck-Yian Lim, Raymond A. Yeh, Yijia Xu, Minh N. Do, and Mark Hasegawa-Johnson

   In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018

   Abs Project Page PDF Code Poster

   Audio super-resolution (a.k.a. bandwidth extension) is the challenging task of increasing the temporal resolution of audio signals. Recent deep networks approaches achieved promising results by modeling the task as a regression problem in either time or frequency domain. In this paper, we introduced Time-Frequency Network (TFNet), a deep network that utilizes supervision in both the time and frequency domain. We proposed a novel model architecture which allows the two domains to be jointly optimized. Results demonstrate that our method outperforms the state-of-the-art both quantitatively and qualitatively.

2. ICASSP

   

   Image restoration with deep generative models

   Raymond A. Yeh, Teck-Yian Lim, Chen Chen, Alexander G. Schwing, Mark Hasegawa-Johnson, and 1 more author

   In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018

   Abs Project Page PDF Code Slides

   Many image restoration problems are ill-posed in nature, hence, beyond the input image, most existing methods rely on a carefully engineered image prior, which enforces some local image consistency in the recovered image. How tightly the prior assumptions are fulfilled has a big impact on the resulting task performance. To obtain more flexibility, in this work, we proposed to design the image prior in a data-driven manner. Instead of explicitly defining the prior, we learn it using deep generative models. We demonstrate that this learned prior can be applied to many image restoration problems using an unified framework.

2017

1. CVPR

   

   Semantic image inpainting with deep generative models

   Raymond A. Yeh, Chen Chen, Teck-Yian Lim, Alexander G. Schwing, Mark Hasegawa-Johnson, and 1 more author

   In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2017

   Abs Project Page PDF Code Poster

   Semantic image inpainting is a challenging task where large missing regions have to be filled based on the available visual data. Existing methods which extract information from only a single image generally produce unsatisfactory results due to the lack of high level context. In this paper, we propose a novel method for semantic image inpainting, which generates the missing content by conditioning on the available data. Given a trained generative model, we search for the closest encoding of the corrupted image in the latent image manifold using our context and prior losses. This encoding is then passed through the generative model to infer the missing content. In our method, inference is possible irrespective of how the missing content is structured, while the state-of-the-art learning based method requires specific information about the holes in the training phase. Experiments on three datasets show that our method successfully predicts information in large missing regions and achieves pixel-level photorealism, significantly outperforming the state-of-the-art methods.