publications

2024

1. 

   Human Gaussian Splatting: Real-time Rendering of Animatable Avatars

   A Moreau, J Song, H Dhamo, R Shaw, Y Zhou, and E Pérez-Pellitero

   CVPR, 2024

   Abs arXiv PDF Website

   This work addresses the problem of real-time rendering of photorealistic human body avatars learned from multi-view videos. While the classical approaches to model and render virtual humans generally use a textured mesh, recent research has developed neural body representations that achieve impressive visual quality. However, these models are difficult to render in real-time and their quality degrades when the character is animated with body poses different than the training observations. We propose the first animatable human model based on 3D Gaussian Splatting, that has recently emerged as a very efficient alternative to neural radiance fields. Our body is represented by a set of gaussian primitives in a canonical space which are deformed in a coarse to fine approach that combines forward skinning and local non-rigid refinement. We describe how to learn our Human Gaussian Splatting (HuGS) model in an end-to-end fashion from multi-view observations, and evaluate it against the state-of-the-art approaches for novel pose synthesis of clothed body. Our method presents a PSNR 1.5dbB better than the state-of-the-art on THuman4 dataset while being able to render at 20fps or more.

   
   
2. 

   NCRF: Neural Contact Radiance Fields for Free-viewpoint Rendering of Hand-Object Interactions

   Z Zhang, J Song, E Pérez-Pellitero, Y Zhou, HJ Chang, and A Leonardis

   3DV, 2024

   Abs arXiv PDF

   Modelling hand-object interaction is a fundamental challenging task in 3D computer vision. Despite remarkable progress has been achieved in this field, existing methods still fail to synthesize the hand-object interaction photo-realistically, suffering from degraded rendering quality caused by the heavy mutual occlusions between hand and object, and inaccurate hand-object pose estimation. To tackle these challenges, we present a novel free-viewpoint rendering framework, Neural Contact Radiance Field (NCRF), to reconstruct hand-object interactions from a sparse set of videos. In particular, the proposed NCRF framework consists of two key components: (a) A contact optimization field that predicts an accurate contact field from 3D query points for achieving desirable contact between hand and object. (b) A hand-object neural radiance field to learn an implicit hand-object representation in a static canonical space, in concert with the specifically designed hand-object motion field to produce observation-to-canonical correspondences. We jointly learn these key components where they mutually help and regularize each other with visual and geometric constraints, producing a high-quality hand-object reconstruction with photo-realistic novel view synthesis. Extensive experiments on HO3D and DexYCB datasets show that our approach outperforms the current state-of-the-art in terms of both rendering quality and pose estimation accuracy.

   
   

2023

1. 

   HeadGaS: Real-Time Animatable Head Avatars via 3D Gaussian Splatting

   H Dhamo, Y Nie, A Moreau, J Song, R Shaw, Y Zhou, and E Pérez-Pellitero

   arXiv:2312.02902 [cs.CV], 2023

   Abs arXiv PDF

   3D head animation has seen major quality and runtime improvements over the last few years, particularly empowered by the advances in differentiable rendering and neural radiance fields. Real-time rendering is a highly desirable goal for real-world applications. We propose HeadGaS, the first model to use 3D Gaussian Splats (3DGS) for 3D head reconstruction and animation. In this paper we introduce a hybrid model that extends the explicit representation from 3DGS with a base of learnable latent features, which can be linearly blended with low-dimensional parameters from parametric head models to obtain expression-dependent final color and opacity values. We demonstrate that HeadGaS delivers state-of-the-art results in real-time inference frame rates, which surpasses baselines by up to  2dB, while accelerating rendering speed by over x10.

   
   
2. 

   VSCHH 2023: A Benchmark for the View Synthesis Challenge of Human Heads

   YK Jang, J Zheng, J Song, H Dhamo, E Pérez-Pellitero, et al.

   ICCV Workshops, 2023

   Abs PDF Website

   This manuscript presents the results of the "A View Synthesis Challenge for Humans Heads (VSCHH)", which was part of the ICCV 2023 workshops. This paper describes the competition setup and provides details on replicating our initial baseline, TensoRF. Additionally, we provide a summary of the participants’ methods and their results in our benchmark table. The challenge aimed to synthesize novel camera views of human heads using a given set of sparse training view images. The proposed solutions of the participants were evaluated and ranked based on objective fidelity metrics, such as PSNR and SSIM, computed against unseen validation and test sets. In the supplementary material, we detailed the methods used by all participants in the VSCHH challenge, which opened on May 15th, 2023, and concluded on July 24th, 2023.

   
   

2022

1. 

   FlexHDR: Modelling Alignment and Exposure Uncertainties for Flexible HDR Imaging

   S Catley-Chandar, T Tanay, L Vandroux, A Leonardis, G Slabaugh, and E Pérez-Pellitero

   IEEE Trans. on Image Processing, 2022

   Abs arXiv Publisher PDF Website

   High dynamic range (HDR) imaging is of fundamental importance in modern digital photography pipelines and used to produce a high-quality photograph with well exposed regions despite varying illumination across the image. This is typically achieved by merging multiple low dynamic range (LDR) images taken at different exposures. However, over-exposed regions and misalignment errors due to poorly compensated motion result in artefacts such as ghosting. In this paper, we present a new HDR imaging technique that specifically models alignment and exposure uncertainties to produce high quality HDR results. We introduce a strategy that learns to jointly align and assess the alignment and exposure reliability using an HDR-aware, uncertainty-driven attention map that robustly merges the frames into a single high quality HDR image. Further, we introduce a progressive, multi-stage image fusion approach that can flexibly merge any number of LDR images in a permutation-invariant manner. Experimental results show our method can produce better quality HDR images with up to 0.8dB PSNR improvement to the state-of-the-art, and subjective improvements in terms of better detail, colours, and fewer artefacts.

   
   
2. 

   NTIRE 2022 Challenge on High Dynamic Range Imaging: Methods and Results

   E Pérez-Pellitero, S Catley-Chandar, R Shaw, A Leonardis, R Timofte, et al.

   CVPR Workshops, 2022

   Abs arXiv HTML Publisher PDF

   This paper reviews the challenge on constrained high dynamic range (HDR) imaging that was part of the New Trends in Image Restoration and Enhancement (NTIRE) workshop, held in conjunction with CVPR 2022. This manuscript focuses on the competition set-up, datasets, the proposed methods and their results. The challenge aims at estimating an HDR image from multiple respective low dynamic range (LDR) observations, which might suffer from under- or over-exposed regions and different sources of noise. The challenge is composed of two tracks with an emphasis on fidelity and complexity constraints: In Track 1, participants are asked to optimize objective fidelity scores while imposing a low-complexity constraint (i.e. solutions can not exceed a given number of operations). In Track 2, participants are asked to minimize the complexity of their solutions while imposing a constraint on fidelity scores (i.e. solutions are required to obtain a higher fidelity score than the prescribed baseline). Both tracks use the same data and metrics: Fidelity is measured by means of PSNR with respect to a ground-truth HDR image (computed both directly and with a canonical tonemapping operation), while complexity metrics include the number of Multiply-Accumulate (MAC) operations and runtime (in seconds).

   
   
3. 

   HDR Reconstruction from Bracketed Exposures and Events

   R Shaw, S Catley-Chandar, A Leonardis, and E Pérez-Pellitero

   BMVC (spotlight), 2022

   Abs arXiv Publisher PDF Poster Website

   Reconstruction of high-quality HDR images is at the core of modern computational photography. Significant progress has been made with multi-frame HDR reconstruction methods, producing high-resolution, rich and accurate color reconstructions with high-frequency details. However, they are still prone to fail in dynamic or largely over-exposed scenes, where frame misalignment often results in visible ghosting artifacts. Recent approaches attempt to alleviate this by utilizing an event-based camera (EBC), which measures only binary changes of illuminations. Despite their desirable high temporal resolution and dynamic range characteristics, such approaches have not outperformed traditional multi-frame reconstruction methods, mainly due to the lack of color information and low-resolution sensors. In this paper, we propose to leverage both bracketed LDR images and simultaneously captured events to obtain the best of both worlds: high-quality RGB information from bracketed LDRs and complementary high frequency and dynamic range information from events. We present a multi-modal end-to-end learning-based HDR imaging system that fuses bracketed images and event modalities in the feature domain using attention and multi-scale spatial alignment modules. We propose a novel event-to-image feature distillation module that learns to translate event features into the image-feature space with self-supervision. Our framework exploits the higher temporal resolution of events by sub-sampling the input event streams using a sliding window, enriching our combined feature representation. Our proposed approach surpasses SoTA multi-frame HDR reconstruction methods using synthetic and real events, with a 2dB and 1dB improvement in PSNR-L and PSNR-mu on the HdM HDR dataset, respectively.

   
   
4. 

   Self-supervised HDR Imaging from Motion and Exposure Cues

   M Nazarczuk, S Catley-Chandar, A Leonardis, and E Pérez-Pellitero

   arXiv:2203.12311 [cs.CV], 2022

   Abs arXiv PDF

   Recent High Dynamic Range (HDR) techniques extend the capabilities of current cameras where scenes with a wide range of illumination can not be accurately captured with a single low-dynamic-range (LDR) image. This is generally accomplished by capturing several LDR images with varying exposure values whose information is then incorporated into a merged HDR image. While such approaches work well for static scenes, dynamic scenes pose several challenges, mostly related to the difficulty of finding reliable pixel correspondences. Data-driven approaches tackle the problem by learning an end-to-end mapping with paired LDR-HDR training data, but in practice generating such HDR ground-truth labels for dynamic scenes is time-consuming and requires complex procedures that assume control of certain dynamic elements of the scene (e.g. actor pose) and repeatable lighting conditions (stop-motion capturing). In this work, we propose a novel self-supervised approach for learnable HDR estimation that alleviates the need for HDR ground-truth labels. We propose to leverage the internal statistics of LDR images to create HDR pseudo-labels. We separately exploit static and well-exposed parts of the input images, which in conjunction with synthetic illumination clipping and motion augmentation provide high quality training examples. Experimental results show that the HDR models trained using our proposed self-supervision approach achieve performance competitive with those trained under full supervision, and are to a large extent superior to previous methods that equally do not require any supervision.

   
   
5. 

   Model-Based Image Signal Processors via Learnable Dictionaries

   M V Conde, S McDonagh, M Maggioni, A Leonardis, and E Pérez-Pellitero

   AAAI (oral), 2022

   Abs arXiv HTML PDF

   Digital cameras transform sensor RAW readings into RGB images by means of their Image Signal Processor (ISP). Computational photography tasks such as image denoising and colour constancy are commonly performed in the RAW domain, in part due to the inherent hardware design, but also due to the appealing simplicity of noise statistics that result from the direct sensor readings. Despite this, the availability of RAW images is limited in comparison with the abundance and diversity of available RGB data. Recent approaches have attempted to bridge this gap by estimating the RGB to RAW mapping: handcrafted model-based methods that are interpretable and controllable usually require manual parameter fine-tuning, while end-to-end learnable neural networks require large amounts of training data, at times with complex training procedures, and generally lack interpretability and parametric control. Towards addressing these existing limitations, we present a novel hybrid model-based and data-driven ISP that builds on canonical ISP operations and is both learnable and interpretable. Our proposed invertible model, capable of bidirectional mapping between RAW and RGB domains, employs end-to-end learning of rich parameter representations, i.e. dictionaries, that are free from direct parametric supervision and additionally enable simple and plausible data augmentation. We evidence the value of our data generation process by extensive experiments under both RAW image reconstruction and RAW image denoising tasks, obtaining state-of-the-art performance in both. Additionally, we show that our ISP can learn meaningful mappings from few data samples, and that denoising models trained with our dictionary-based data augmentation are competitive despite having only few or zero ground-truth labels.

   
   

2021

1. 

   NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results

   E Pérez-Pellitero, S Catley-Chandar, A Leonardis, R Timofte, et al.

   CVPR Workshops, 2021

   Abs arXiv HTML PDF

   This paper reviews the first challenge on high-dynamic range (HDR) imaging that was part of the New Trends in Image Restoration and Enhancement (NTIRE) workshop, held in conjunction with CVPR 2021. This manuscript focuses on the newly introduced dataset, the proposed methods and their results. The challenge aims at estimating a HDR image from one or multiple respective low-dynamic range (LDR) observations, which might suffer from under- or over-exposed regions and different sources of noise. The challenge is composed by two tracks: In Track 1 only a single LDR image is provided as input, whereas in Track 2 three differently-exposed LDR images with inter-frame motion are available. In both tracks, the ultimate goal is to achieve the best objective HDR reconstruction in terms of PSNR with respect to a ground-truth image, evaluated both directly and with a canonical tonemapping operation.

   
   

2019

1. 

   Perceptual Video Super Resolution with Enhanced Temporal Consistency

   E Pérez-Pellitero, M S M Sajjadi, M Hirsch, and B Schölkopf

   arXiv:1807.07930v2 [cs.CV], 2019

   Abs arXiv PDF

   With the advent of perceptual loss functions, new possibilities in super-resolution have emerged, and we currently have models that successfully generate near-photorealistic high-resolution images from their low-resolution observations. Up to now, however, such approaches have been exclusively limited to single image super-resolution. The application of perceptual loss functions on video processing still entails several challenges, mostly related to the lack of temporal consistency of the generated images, i.e., flickering artifacts. In this work, we present a novel adversarial recurrent network for video upscaling that is able to produce realistic textures in a temporally consistent way. The proposed architecture naturally leverages information from previous frames due to its recurrent architecture, i.e. the input to the generator is composed of the low-resolution image and, additionally, the warped output of the network at the previous step. Together with a video discriminator, we also propose additional loss functions to further reinforce temporal consistency in the generated sequences. The experimental validation of our algorithm shows the effectiveness of our approach which obtains images with high perceptual quality and improved temporal consistency.

   
   

2018

1. 

   Photorealistic Video Super Resolution

   E Pérez-Pellitero, M S M Sajjadi, M Hirsch, and B Schölkopf

   ECCV Workshops (PIRM), 2018

   Abs arXiv PDF

   In this work, we present a novel adversarial recurrent network for video upscaling that is able to produce realistic textures in a temporally consistent way. The proposed architecture naturally leverages information from previous frames due to its recurrent architecture, i.e. the input to the generator is composed of the low-resolution image and, additionally, the warped output of the network at the previous step.

   
   

2017

1. 

   Manifold Learning for Super Resolution

   Eduardo Pérez-Pellitero

   PhD Dissertation
   Leibniz Universität Hannover, 2017

   Abs HTML PDF

   The development pace of high-resolution displays has been so fast in the recent years that many images acquired with low-end capture devices are already outdated or will be shortly in time. Super Resolution is central to match the resolution of the already existing image content to that of current and future high resolution displays and applications. This dissertation is focused on learning how to upscale images from the statistics of natural images.

   
   

2016

1. 

   PSyCo: Manifold Span Reduction for Super Resolution

   E Pérez-Pellitero, J Salvador, J Ruiz-Hidalgo, and B Rosenhahn

   CVPR, 2016

   Abs PDF Code

   In this paper we present a novel regression-based SR algorithm that benefits from an extended knowledge of the structure of both manifolds. We propose a transform that collapses the 16 variations induced from the dihedral group of transforms (i.e. rotations, vertical and horizontal reflections) and antipodality (i.e. diametrically opposed points in the unitary sphere) into a single primitive. The key idea of our transform is to study the different dihedral elements as a group of symmetries within the high-dimensional manifold. The experimental validation of our algorithm shows the effectiveness of our approach, which obtains competitive quality with a dictionary of as little as 32 atoms (reducing other methods’ dictionaries by at least a factor of 32) and further pushing the state-of-the-art with a 1024 atoms dictionary.

   
   
2. 

   Antipodally Invariant Metrics For Fast Regression-based Super-Resolution

   E Pérez-Pellitero, J Salvador, J Ruiz-Hidalgo, and B Rosenhahn

   IEEE Trans. on Image Processing, 2016

   Abs PDF Code

   In this paper we present a very fast regression-based algorithm which builds on densely populated anchored neighborhoods and sublinear search structures. Even though we validate the benefits of using antipodally invariant metrics, most of the binary splits use Euclidean distance, which does not handle antipodes optimally. In order to benefit from both worlds, we propose a simple yet effective Antipodally Invariant Transform (AIT) that can be easily included in the Euclidean distance calculation. We modify the original Spherical Hashing algorithm with this metric in our Antipodally Invariant Spherical Hashing scheme, obtaining the same performance as a pure antipodally invariant metric. We round up our contributions with a novel feature transform that obtains a better coarse approximation of the input image thanks to Iterative Back Projection.

   
   
3. 

   Half Hypersphere Confinement for Piecewise Linear Regression

   E Pérez-Pellitero, J Salvador, J Ruiz-Hidalgo, and B Rosenhahn

   WACV, 2016

   Abs PDF Code Poster

   In this paper we study the characteristics of the metrics best suited for the piecewise regression algorithms, in which comparisons are usually made between normalized vectors that lie on the unitary hypersphere. Even though Euclidean distance has been widely used for this purpose, it is suboptimal since it does not handle antipodal points (i.e. diametrically opposite points) properly. Therefore, we propose the usage of antipodally invariant metrics and introduce the Half Hypersphere Confinement (HHC), a fast alternative to Multidimensional Scaling (MDS) that allows to map antipodally invariant distances in the Euclidean space with very little approximation error. The performance of our method, which we named HHC Regression (HHCR), applied to Super-Resolution (SR) improves both in quality (PSNR) and it is faster than any other state-of-the-art method. Additionally, under an application-agnostic interpretation of our regression framework, we also test our algorithm for denoising and depth upscaling with promising results.

   
   

2015

1. 

   Naive Bayes Super-Resolution Forest

   J Salvador, and E Pérez-Pellitero

   ICCV, 2015

   Abs PDF Code

   This paper presents a fast, high-performance method for super resolution with external learning. The first contribution leading to the excellent performance is a bimodal tree for clustering, which successfully exploits the antipodal invariance of the coarse-to-high-res mapping of natural image patches and provides scalability to finer partitions of the underlying coarse patch space. During training an ensemble of such bimodal trees is computed, providing different linearizations of the mapping. The second and main contribution is a fast inference algorithm, which selects the most suitable mapping function within the tree ensemble for each patch by adopting a Local Naive Bayes formulation. The resulting method is beyond one order of magnitude faster and performs objectively and subjectively better than the current state of the art.

   
   
2. 

   Accelerating Super-Resolution for 4K Upscaling

   E Pérez-Pellitero, J Salvador, J Ruiz-Hidalgo, and B Rosenhahn

   ICCE, 2015

   Abs PDF

   This paper presents a fast Super-Resolution (SR) algorithm based on a selective patch processing. Motivated by the observation that some regions of images are smooth and unfocused and can be properly upscaled with fast interpolation methods, we locally estimate the probability of performing a degradation-free upscaling. Our proposed framework explores the usage of supervised machine learning techniques and tackles the problem using binary boosted tree classifiers.

   
   

2014

1. 

   Fast Super-Resolution via Dense Local Training and Inverse Regressor Search

   E Pérez-Pellitero, J Salvador, I Torres-Xirau, J Ruiz-Hidalgo, and B Rosenhahn

   ACCV, 2014

   Abs PDF

   Under the locally linear embedding assumption, SR can be properly modeled by a set of linear regressors distributed across the manifold. In this paper we propose a fast inverse-search approach for regression-based SR. Instead of performing a search from the image to the dictionary of regressors, the search is done inversely from the regressors’ dictionary to the image patches. Additionally, we propose an improved training scheme for SR linear regressors which improves perceived and objective quality. By merging both contributions we improve both speed and quality compared to the state-of-the-art.

   
   
2. 

   Fast Approximate Nearest-Neighbor Field by Cascaded Spherical Hashing

   I Torres-Xirau, J Salvador, and E Pérez-Pellitero

   ACCV, 2014

   Abs PDF

   We present an efficient and fast algorithm for computing approximate nearest neighbor fields between two images. Our method builds on the concept of Coherency-Sensitive Hashing (CSH), but uses a recent hashing scheme, Spherical Hashing (SpH), which is known to be better adapted to the nearest-neighbor problem for natural images. Cascaded Spherical Hashing concatenates different configurations of SpH to build larger Hash Tables with less elements in each bin to achieve higher selectivity. Our method amply outperforms existing techniques like PatchMatch and CSH, and the experimental results show that our algorithm is faster and more accurate than existing methods.

   
   
3. 

   Robust Single-Image Super-Resolution using Cross-Scale Self-Similarity

   J Salvador, E Pérez-Pellitero, and A Kochale

   ICIP, 2014

   Abs PDF

   We present a noise-aware single-image super-resolution (SISR) algorithm, which automatically cancels additive noise while adding detail learned from lower-resolution scales. In contrast with most SI-SR techniques, we do not assume the input image to be a clean source of examples. Instead, we adapt the recent and efficient in-place cross-scale self-similarity prior for both learning fine detail examples and reducing image noise. The experimental results show a promising performance, despite the relatively simple algorithm. Both objective evaluations and subjective validations show clear quality improvements when upscaling noisy images.

   
   
4. 

   An Epipolar-Constrained Prior for Efficient Search in Multi-View Scenarios

   I Bosch, J Salvador, E Pérez-Pellitero, and J Ruiz-Hidalgo

   EUSIPCO, 2014

   Abs PDF

   In this paper we propose a novel framework for fast exploitation of multi-view cues with applicability in different image processing problems. An epipolar-constrained prior is presented, onto which a random search algorithm is proposed to find good matches among the different views of the same scene. This algorithm includes a generalization of the local coherency in 2D images for multi-view wide-baseline cases. Experimental results show that the geometrical constraint allows a faster initial convergence when finding good matches.

   
   

2013

1. 

   Bayesian region selection for adaptive dictionary-based Super-Resolution

   E Pérez-Pellitero, J Salvador, J Ruiz-Hidalgo, and B Rosenhahn

   BMVC, 2013

   Abs HTML PDF

   This paper presents a novel sparse SR method, which focuses in adaptively selecting the optimal patches for the dictionary training. The method divides the training images into sub-image regions of sizes that preserve texture consistency. The best-representing region for each input LR patch is found through a Bayesian selection stage. In this selection process, SIFT descriptors are extracted densely from both input LR patches and regions and a local NBNN approach is used in order to efficiently handle the high number of different regions in the training dataset.