Ko Nishino, Ph.D.Assistant ProfessorDepartment of Computer Science College of Engineering Drexel University I am an assistant professor of Computer Science at Drexel University. Prior to joining Drexel in fall 2005, I was a postdoctoral research scientist at Columbia University. I received all my degrees from The University of Tokyo: both BE and ME in Information and Communication Engineering (CE) in 1997 and 1999, respectively, and PhD in Information Science (CS) in 2002. My research interests primarily lie in computer vision. In particular, I strive to develop novel models and computational algorithms to better extract, understand, and regenerate visual information from photographs and videos. To this end, my main focus centers on leveraging intrinsic structures of visual data -- the latent structures that can be found in the geometry, radiometry, and motion of real-world scenes that are not necessarily apparent to our naked eyes. ContactDepartment of Computer Science Group MembersProspective students: If you are interested in studying towards a PhD degree under my guidance, please just go ahead and submit your application through the official channel (see this page). If you are eager to work with me, simply state so in your essay/statement and explain your motivation, credentials, plan, etc. There is no need to contact me and unfortunately I will not have the time to respond to individual inquiries. Also, please note that I do not hire interns.
Teaching
ResearchMy research spans the areas of Computer Vision and Computer Graphics. I mainly focus on developing physically-driven image-based representations and algorithms for efficently analyzing and synthesizing natural appearance. My current research interests include image-based rendering/modeling, photometric material analysis/synthesis, lighting estimation and relighting, recognition and surveillance especially under varying illumination.The following are some projects I have worked on in the past. If you are looking for VisualEyes, please go HERE.
PublicationsHere's a list of selected publications. A complete list can be found here.
Codes and Presentations
|
drexel.edu
In this work, we show that, with canonical geometric scale-space
analysis, the inherent scale-variability encoded in a range image
can be exploited as a rich source of discriminative information
regarding the captured geometry. We extend our previous work on
geometric scale-space analysis of 3D models to analyze the
scale-variability of a range image and to detect scale-dependent 3D
features - geometric features with their inherent scales. We derive
novel local 3D shape descriptors that encode the local shape
information within the inherent support region of each feature. We
show that the resulting set of scale-dependent local shape
descriptors can be used in an efficient hierarchical registration
algorithm for aligning range images with the same global scale. We
also show that local 3D shape descriptors invariant to the scale
variation can be derived and used to align range images with
significant differences in their global scales. Finally, we
demonstrate that the scale-dependent/invariant local 3D shape
descriptors can even be used to fully automatically register
multiple sets of range images with varying global scales
corresponding to multiple objects. This is the first work to
demonstrate such capability.
Three-dimensional geometric data play increasingly vital roles in
various computer vision applications, ranging from navigation to
inverse rendering. Despite their ubiquitous use, little attention
has been given to the fact that real-world 3D geometric data can
contain significant scale variation in their local geometric
structures. For instance, in a 3D human face model, both the tip of
the nose and dimples are discriminative geometric features suitable
for representing the underlying surface. The spatial extents of such
geometric features, however, significantly differ from one another
-- they lie at entirely different scales. While this geometric scale
variability can be deemed to be another cause of error in subsequent
processing, it can in turn be exploited as an additional source of
information to enrich the representation of the actual object/scene
geometry. In this paper we focus on extracting scale-dependent 3D
geometric features, a unified set of geometric features detected at
their own intrinsic scales, in geometric data given as mesh models.
(with J. Novatnack)
Face recognition under varying illumination remains a challenging
problem. Much progress has been made toward a solution through
methods that require multiple gallery images of each subject under
varying illumination. Yet for many applications, this requirement
is too severe. We propose a novel method that
requires only a single gallery image per subject taken under unknown
lighting. The method builds upon two contributions. We first
estimate the lighting from its reflection in the eyes. This allows
us to explicitly recover the illumination in the single gallery
images as well as the probe image. Next, we exploit the local
linearity of face appearance variation across different people. We
represent the gallery images as locally linear montages of images of
many different faces taken under the same lighting (bootstrap
images). Then, we transfer the estimated combination of bootstrap
images to synthesize each subject's face under the probe lighting to
accomplish recognition. We show through tests on the CMU
PIE database that we can achieve better recognition results using
our lighting estimation method and locally linear montages than the
current state-of-the-art.
(with 
In order to relight real objects and scenes, it is important to know
their lighting conditions. Obtaining the lighting of a scene from a
single image remains a difficult and open problem. However, this
problem can be solved when we have a face (and hence an eye) in the
image, which is often the case with images and videos. We show how
an image of an eye taken in a completely unstructured setting can be
used to extract a dense illumination distribution. In concrete, we
show that an environment map of the scene with a large field of view
can be computed from the eye which, in turn, represents the
illumination distribution of the scene with respect to the eye. In
other words, the eye can be used as a natural light probe which
gives us not just the directions of a few point sources but the
complete distribution of the frontal illumination incident on the
eye and hence the face. We demonstrate the use of an eye in a number
of relighting scenarios including virtual object insertion, casual
face relighting, and face replacement in already taken images. These
results show that the eye not only serves as a useful tool for
relighting but also make relighting possible in situations where
other techniques are hard to use. (with
Eyes in images convey very rich visual information. It turns out
that the cornea of an eye and a camera viewing the eye form a
catadioptric imaging system. Unlike conventional catadioptric
systems, this corneal imaging system is an uncalibrated one. We use
a geometric model of the cornea based on anatomical studies to
self-calibrate the system. Once this is done, a wide-angle view of
the environment of the person can be obtained from the image. In
addition, we can compute the projection of the environment onto the
retina with its center aligned with the gaze direction. This
foveated retinal image reveals what the person is looking at. We
present a detailed analysis of the characteristics of the corneal
imaging system including field of view, resolution and locus of
viewpoints. When both eyes of a person are captured in an image, we
have a stereo corneal imaging system. We analyze the epipolar
geometry of this stereo system and show how it can be used to
compute 3D structure. The visual information extracted from eyes can
be used to understand the circumstance and intent of the person in
the image. Such information can be of significant use in various
fields including psychology, HCI, and computer graphics. (with
We present a method to accomplish photorealistic
rendering of real-world objects from their sparsely
sampled appearance variation. Using a 3D model and a
small set of images of an object, we recover all the
necessary photometric information for subsequent
rendering. Unlike previous
approaches, we require less input images and we do not
assume anything to be known about the three photometric
attributes, namely the diffuse and specular reflection
parameters and the lighting condition. (
with 
Eigen-Texture method samples apppearances of a real
object under various illumination and viewing
conditions, and compresses them in the 2D coordinate
system defined on the 3D model surface, which is
generated from a sequence of range images. It does not
require any analysis of reflectance properties of the
object surface, as model-based methods did. Furthermore,
as it generates and uses a 3D model of the object, it
has great advantage in application to mixed reality
systems; such as casting shadows, which was difficult
with image-based methods. (with 
Cast shadows produce troublesome effects for video
surveillance systems. To robustly eliminate these shadows
from image sequences as a preprocessing stage for robust
video surveillance, we propose a framework based on the
idea of intrinsic images. Unlike previous methods to
derive intrinsic images, we derive time-varying
reflectance images and corresponding illumination images
from a sequence of images. Using the illumination
images, we normalize the input image sequence in terms of
distribution of the incident lighting to eliminate shadow
effects. We also propose an approach which can potentially
run in realtime by introducing illumination eigenspace,
which captures the illumination variation due to weather,
time of day, etc., and a shadow interpolation method based
on shadow hulls. (with 
We present a new method of color constancy to
estimate the illumination chromaticity from a
single/multi-colored surface. Unlike existing
dichromatic-based methods, our method requires only
rough highlight regions, without segmenting the colors
inside them. We show that by analyzing the highlights of
an image, we can obtain a direct correlation between
illumination chromaticity and image chromaticity. This
correlation can be clearly characterized in the
"inverse-intensity chromaticity space", a new
two-dimensional space we introduce. Then, using Hough
transform and histogram analysis in this space, we can
robustly estimate the illumination chromaticity from
even a highly textured surface. (with 
I co-led a 5 year project to develop
techniques to digitally preserve existing ancestral
objects mainly in Japan and other countries in Asia. My
main contributions were the development of a robust
simultaneous registration algorithm to align multiple
range images output from a large scale laser range
finder, a robust merging algorithm to integrate those
registered range images into one mesh model which can
adaptively construct the meshes with regards to its
additional attributes such as curvature and surface
texture and a robust algorithm to align 2D images to the
resulting 3D mesh model taking into account the
multi-view epipolar geometry. (with 