IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007 1481
An Image Morphing Technique Based
on Optimal Mass Preserving Mapping
Abstract—Image morphing, or image interpolation in the time
domain, deals with the metamorphosis of one image into another.
In this paper, a new class of image morphing algorithms is proposed
based on the theory of optimal mass transport. The P mass
moving energy functional is modified by adding an intensity penalizing
term, in order to reduce the undesired double exposure effect.
It is an intensity-based approach and, thus, is parameter free.
The optimal warping function is computed using an iterative gradient
descent approach. This proposed morphing method is also
extended to doubly connected domains using a harmonic parameterization
technique, along with finite-element methods.
Index Terms—Image interpolation, image morphing, image
warping, mass preserving mapping, Monge–Kantorovich flow,
optimal transport.
I. INTRODUCTION
IMAGE morphing, sometimes referred to as “image interpolation
in the time domain,” deals with the metamorphosis of
one image to another [21]. It is a technique widely used in television
commercials, music videos and motion pictures. Image
morphing has also been used for facial recognition [37]. Given a
pair of images, the goal of image morphing is to find a sequence
of intermediate images, such that the first image in the sequence
is equal to the first given image (starting image) and the last
image is equal to the second given image (ending image). The
process begins with finding a reasonable warping function between
the two images, and this warping function is then used
to interpolate the position of pixels through the in-between sequence.
Finally, intensity or color interpolation (i.e., cross dissolving)
is performed to generate the intermediate images.
There have been many algorithms proposed for image morphing.
Some of the most popular approaches are mesh warping,
Manuscript received September 7, 2006; revised January 22, 2007. This work
was supported in part by the National Science Foundation, in part by the Air
Force Office of Scientific Research, in part by ARO, in part by MURI, in part
by MRI-HEL, and in part by the National Institutes of Health (NIH) under Grant
NAC P41 RR-13218 through Brigham and Women’s Hospital. This work is part
of the National Alliance for Medical Image Computing (NAMIC), funded by the
NIH through the NIH Roadmap for Medical Research, Grant U54 EB005149.
The associate editor coordinating the review of this manuscript and approving
it for publication was Prof. Stanley J. Reeves.
L. Zhu, Y. Yang, and A. Tannenbaum are with the Department of Biomedical
Engineering and the School of Electrical and Computer Engineering, Georgia
Institute of Technology, Atlanta, GA 30332 USA (e-mail: zlzl@ece.gatech.edu;
zhulei1976@hotmail.com; yan.yang@gatech.edu; tannenba@ece.gatech.edu).
S. Haker is with the Surgical Planning Laboratory, Brigham and Women’s
Hospital and Harvard Medical School, Boston, MA 02115 USA (e-mail:
haker@bwh.harvard.edu).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TIP.2007.896637
Lei Zhu, Yan Yang, Steven Haker, and Allen Tannenbaum
1057-7149/$25.00 © 2007 IEEE
field warping, and energy-based warping. In mesh warping [35],
features are specified by a nonuniform control mesh, and the
warping function is usually generated by a spline interpolation.
This class of mesh warping algorithm usually shows good distortion
behavior, but it has a critical drawback in specifying features
since the features on the control mesh may have an arbitrary
structure. It is also time consuming to define the feature
correspondence via a user interface. In field morphing [2],
a pair of corresponding lines is specified. The mapping of a
point in the vicinity of a line can be determined by its distance
from the line. In the case of multiple line pairs, the warping of a
given point is calculated by a weighted sum of mappings of all
line pairs. This method is easy to use to specify corresponding
features. However, sometimes a part of the image may appear
in unrelated regions in the in-between images (often referred
to as “ghosts”). Energy minimization-based warpings usually
guarantee the one-to-one mapping property, which prevents the
warped image from folding back upon itself. For example, in
Lee et al.’s work [21], points, polylines, and curves are sampled
and reduced to a collection of points. These points are then
used to generate the warping function by minimizing an energy
functional. A similar method has been applied to facial morphing
based on Navier elastic body spline [15]. All of the above
approaches fall into the landmark-based category and require
user inputs of the corresponding features.
The approach proposed in the present work is intensity based.
Here, one uses only information given by the images such as
pixel intensities to perform the warping. A successful intensity-based
algorithm can achieve automatic morphing without
user inputs or prior assumptions on special shapes or features of
the objects in the image. See [31] for a review of the literature
and an extensive list of references on this subject. We should
also mention the nice work of Iwanowski [17] in which an image
morphing approach is proposed that combines morphological
interpolation and linear filtering and does not require control
points or landmarks. In this paper, we present an automatic morphing
algorithm using a completely different approach which is
based on the theory of optimal mass transport. It is important to
note that we do not claim that our method is optimal in any sense
for morphing, but does throw some new light on the morphing
problem, and seems reasonable for certain types of images in
which there is an elastic deformation as demonstrated in some
of our experiments given below.
Optimal mass transport problem was first formulated by
Gaspar Monge in 1781. It concerned finding an optimal way,
in the sense of minimal transportation cost, of moving a pile of
soil from one site to another. This problem was given a modern
formulation by Kantorovich in 1948 [19], and is also known
1482 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007
as the Monge–Kantorovich problem (MKP). The optimal mass
transport problem has been extensively studied in the fields of
econometrics, fluid dynamics, automatic control, transportation,
statistical physics, shape optimization, expert systems,
and meteorology [26]. Recently, this problem has been studied
within the context of content-based image retrieval [22], [27],
[28]. Pixels in an image are divided into several bins according
to their color and spatial locations. The Earth Mover’s distance
(EMD) is then calculated between the bins of two images and
used for image retrieval.
Our interest in MKP arose from our visualization work in
medical applications. For example, a flattened representation of
colon surface is helpful for the detection of colon polys [13]
and a flattened representation of vessel surface is useful for
the study of the correlation between wall shear stress and the
development of atherosclerosis [39]. Among various flattening
techniques, a mapping that preserves the area is of special interests.
In this approach, an angle preserving flattening is first
constructed through harmonic analysis, and then the optimal
mass preserving mapping is applied in order to correct area distortions.
The resulting mapping is an area preserving mapping
with minimal distortion. We have also proposed an MKP-based
image registration algorithm [14], in which a pseudo-mass (intensity-weighted
area) is preserved. Our algorithm can handle
area preserving registration [12] naturally by simply assigning
the pseudo mass density to unity on the entire domain of the
image. Other approaches to using various classes of diffeomorphisms
for registration and warping may be found for example
in [25], [31], [32], and the references therein.
In this paper, we present an improved approach for image
morphing, with special efforts taken to reduce the double exposure
effect (also referred to as the “fade-in and fade-out” effect).
An improved three-step gradient descent approach (as opposed
to the two-step approach used in [14] and [40]) is employed here
for rectangular domains. We also explain how to extend this approach
to irregular multiconnected domains.
We now outline the contents of this paper. In Section II, we
give a brief review of the optimal mass transport problem. In
Section III, we present our approach for image morphing between
two rectangular images, using the improved gradient descent
method. Two types of comparison terms are studied in this
section. In Section IV, we extend our morphing algorithm to
doubly connected domains, based on a harmonic parameterization
technique. In Section V, we illustrate the proposed algorithms
using real images as examples. Finally, in Section VI,
we summarize the contributions of this paper and discuss possible
future directions. An Appendix is provided at the end of
the paper to give more mathematical details of our methods.
II. MONGE–KANTOROVICH PROBLEM
We now give a modern formulation of the MKP. Assume that
and are two domains in with smooth boundaries,
each with a positive density, and , respectively. Further,
we assume that both domains contain same total amount of mass
(1)
A mapping is called mass preserving (MP), if
satisfies
We denote this as . Equation (2) is called the Jacobian
equation, where denotes the determinant of the Jacobian of
, and denotes the composition of two functions. Equation (2)
implies, for example, that if a small region in is mapped onto
a larger region in , there must be a corresponding decrease in
density in order for the mass to be preserved.
There exist infinite number of such mappings. A criterion, or
a metric, must be defined in order to obtain an optimal mapping.
In this work, we employ the Kantorovich–Wasserstein
metric defined as follows:
This metric places a penalty on the mass weighted distance
of each bit of material moved by mapping .
In this paper, we take . One can show that in this case
one gets a unique minimizer given by the gradient of a convex
function; see [3], [11], [20], and the references therein. Note
that the Kantorovich–Wasserstein metric defines the distance
between two mass densities, by seeking the “cheapest” way to
transport the mass from one domain to another with respect to
the metric (3). The optimal MP mapping thus defined is symmetric,
the optimal mapping from to being the inverse of
the optimal mapping from to .
III. IMAGE MORPHING ON RECTANGULAR REGIONS
In the context of image morphing, we can assume that the
mass density is the image intensity and directly apply the
optimal mass transport algorithm on two related images to
generate a deformed grid. The in-between image sequence
can be obtained using cross dissolving. However, a mapping
that maps a small high intensity region to a large low intensity
region is not desirable since it will cause double exposure effect
in the in-between images. The Kantorovich–Wasserstein
metric imposes a penalty only on the work spent on moving
mass from one shape to another, but not on the change of mass
densities or image intensities. Hence, we add a comparison
term to the distance metric (3) to penalize the change of intensity
in the image. In this section, we only consider the image
morphing problem between two rectangular domains, and leave
the discussion of more general domains to Section IV.
The idea is to minimize a functional of the following form
over an MP mappings
where is a fixed positive number. The first term controls
the “goodness of fit” between the intensity images
and , and when and are identical,
this term reaches its minimum. The second Monge–Kantorovich
term controls the warping of the map. The function is the
mass density defined on , which could be the same as or a
smoothed version of . could also be any scalar field that is
appropriate for the underlying physical model. Similarly, is
(2)
(3)
(4)
ZHU et al.: IMAGE MORPHING TECHNIQUE 1483
the mass density defined on . By adjusting , we control the
tradeoff between minimal mass transport and minimal intensity
change. It must be pointed out that by adding the comparison
term , the energy functional (4) may have multiple
local minima, and the resulting optimal mapping is no longer a
curl-free field as in the classical MKP.
We note that the comparison term may be taken
to be any metric that measures the similarity between the transformed
source image and the target image, e.g., the sum of
squared difference (SSD), likelihood measurement, correlation
ratio, normalized correlation, or mutual information (MI) [34].
In this paper, SSD and MI are adopted as the similarity measures,
based on the characteristics of our testing images.
If SSD is used as the similarity measure [14], we are minimizing
the following energy functional:
and if MI is used as the similarity measure [16], the energy functional
has the form
Note that there is a minus sign before the MI term, since the
more similar the two images are, the larger the MI measure is.
It should also be pointed out that the first integral is taken over
the domain of , where and are intensity levels in
the histograms of the two images. The probability density functions
and can be estimated by the 1-D nonparametric
Parzen–Rozenblatt density model [9]
and
where is a 1-D Gaussian window, whose standard deviation
can be chosen to be 10% of the standard deviation of and
for (7) and (8), respectively. is the area of , or the number
of pixels inside for the discrete case. From another point of
view, this Parzen-window method is equivalent to smoothing the
image histogram using a Gaussian filter.
In a similar way, can be estimated using a 2-D
nonparametric Parzen–Rozenblatt density model as follows:
(9)
where is a 2-D Gaussian window, whose covariance is decided
by the covariance matrix of the paired random variables
. For simplicity, we write as in the remaining part
of this paper.
(5)
(6)
(7)
(8)
There have been a number of algorithms considered for the
computation of an optimal transport map. For example, methods
have been proposed based on Lagrangian mechanics that are
closely related to the ideas in fluid dynamics [3], and geometric
methods have also been employed as in Culler-Purser [7]. Perhaps
the most common approach is to reduce the optimal
transport problem to a linear programming problem [26]. However,
a fundamental difficulty of linear programming is the computational
complexity [18].
We propose to use a different approach to solve the modified
optimal mass transport problem (4). The basic idea is to
start from an initial MP mapping from to , and based
on the fact that the composition of two MP mappings is still
an MP mapping, the second step is to update the mapping iteratively
in a gradient descent way using MP mapping that
maps onto itself and satisfies the MP property to decrease the
pure MKP energy functional without the comparison term.
It can be proved that the curl of the mapping is also decreasing
in this process [1]. The resulting MP mapping is a curl-free
field. Finally, we start from this curl-free field and use a similar
gradient descent approach to minimize the modified energy
functional (4). As we mentioned earlier, the functional (4) may
have multiple local minima. Starting with a curl-free MP mapping
will make the final MP mapping look more natural
by putting most curl into the dark region of the images, as shown
in the results described in Section V.
A. The Initial MP Mapping
For arbitrary domains, the initial mapping can be solved using
the method proposed by Dacorogna and Moser [8]. Since we are
working on more regular domains (specifically, two rectangular
domains in 2-D or two cubical domains in 3-D), a simpler algorithm
is implemented here. To further simplify the problem,
we assume we are working in with , the
generalization to higher dimensions being straightforward. The
idea of the proposed method is that we solve a 1-D mass transport
problem in one direction and then solve a family of 1-D
mass transport problems in the other direction. In 1-D, the optimal
transport map can then be found by simple quadrature.
We can let the mass first be transported along the lines parallel
to the axis, and then transported along the lines parallel to
the axis. Accordingly, we assume that the initial MP mapping
has the form , where function is
defined by equation
By differentiating (10) with respect to ,wehave
We may now define function by equation
(10)
(11)
(12)
1484 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007
Since , , by differentiating
(12) with respect to , we get
(13)
which is the MP condition. In practice, and can
be found using numerical integration techniques. Given our assumption
that and are positive everywhere, is a
monotonically increasing function. is also monotonically
increasing with respect to from (12), given that is
always positive. Hence, there is no space folding problem with
the initial mapping .
B. Curl-free MP Mapping
Once an initial MP mapping is found, we need to discover
the curl-free MP mapping by finding the polar factorization
[11], [23] of . Rather than finding directly, we
solve an energy minimizing problem iteratively via a gradient
descent method. If the energy functional is defined by the
Kantorovich–Wasserstein metric, this process is equivalent to
finding the polar factorization and is guaranteed to converge to
a global optimum [1]. We give the key steps of this method in
this section, and more mathematical details can be found in [1],
[14], and in the Appendix.
Assume that is updated by an MP mapping from to
itself, i.e., . Based on the facts that the inverse of an
MP mapping is an MP mapping and the composition of two MP
mappings is an MP mapping, is also an MP mapping from
to . We take to be a function of “time” (i.e., gradient descent
parameter), initially being the identity map. In order to maintain
the MP property, the update of should have the following form:
(14)
for some vector field on , with and
on , being the normal to the boundary of . Accordingly,
should satisfy
(15)
In the gradient descent approach, we take the derivative of the
pure MKP energy functional with respect to . It can be
proved that in 2-D, should satisfy , where represents
a rotation by counter clockwise and is given by
(16)
Here, indicates an identity mapping. Considering (15) and
, the updating equation for is given by
(17)
where stands for the solution to the Poisson equation (16).
When the algorithm converges, we have a curl-free vector field
, which will serve as the initial mapping for the next step.
C. inimizer
We use a similar gradient descent approach to find a minimizer
to the functional (4). As above, the idea is to modify
an initial mapping by an MP mapping from to itself in
order to decrease the energy (4).
More specifically, the updating equation is given by
If SSD is used as the comparison term, has the form of
and if MI is used as the comparison term, is given by
(18)
(19)
(20)
where stands for 2-D convolution, and is the derivative of
with respect to its first variable.
Once the algorithm converges, we have the final warping
function . Then a cross-dissolving method is performed
to generate a sequence of in-between images , such that
and . It is assumed that when time varies
from 0 to 1, the starting image continuously changes to the
ending image . We further require that the same transition
rate is applied to all points on the in-between images [21].
Hence, the image warping map at any time is
simply given by
(21)
and the corresponding cross-dissolved image at time is given
by
(22)
and can also be color images and (22) can be applied
to three color components individually. The warping function
(21) guarantees the continuous transformation from the source
image to the target image, being the transition rate. One can
also guarantee that the intermediate frames are mass preserving
simply by shading the pixels in the in-between images according
to .
D. Implementation
As a summary, our proposed algorithm takes the following
steps.
1) Construct an initial MP mapping using (10) and (12).
ZHU et al.: IMAGE MORPHING TECHNIQUE 1485
2) Starting with , update the MP mapping iteratively
using (17) and obtain .
3) Starting with , update the MP mapping iteratively
using (18).
4) Calculate in-between images using the cross-dissolving
method by (21) and (22).
An upwinding scheme is used when computing in (17),
and all other spacial derivatives (including and div) are computed
using standard central differences. Matlab solver poicalc
is used to solve the Poisson equation (16). The time step can
be chosen to be less than
for the nonlocal flow, where stands for the component of the
vector. The curl-free mapping is obtained as .In
practice, the procedure iterates until the mean absolute curl is
sufficiently small. More details about the implementation of
Monge–Kantorovich flows may be found in [14].
The numerical implementation of (18) is exactly the same as
that for solving using (17), except is substituted by
. The iteration stops when the energy functional (4) decreases
sufficiently slowly. The optimal mapping is obtained when
the iteration stops.
IV. IMAGE MORPHING ON DOUBLY CONNECTED REGIONS
In the previous section, we presented an MP morphing algorithm
between two simply connected domains, or more specifically,
two rectangular regions. It is assumed that mass is preserved
over the entire domain. In some images, however, the
MP condition is valid only between parts of the two images
rather than the entire domain. For example, if we are given a
sequence of magnetic resonance (MR) images of a human heart
taken at different times in the cardiac cycle at the same spatial
location, the MP condition is only valid in the myocardium region,
but invalid in the heart ventricles since the volume of the
blood varies from time to time. A natural approach would be to
perform image segmentation first to separate the myocardium
and the ventricles, and then perform image morphing only on
the myocardial region of interest.
In this section, we extend the MP mapping algorithm to
doubly connected domains [41]. The main difficulty comes
from the construction of the initial MP mapping. We propose
an algorithm that finds by using harmonic parameterization.
In this approach, the two domains are first harmonically
parameterized, and then can be obtained in the harmonic
coordinate system. A finite-element method (FEM) is applied
in conjunction with the gradient descent method to account for
the irregularity of the domains.
A. Harmonic Parameterization
We first show how to construct an analytic function
for the harmonic parameterization of a doubly connected
domain (here denotes a square root of 1). Similar
techniques have been applied for colon surface visualization
Fig. 1. Dubly onnected domain 6 with inner boundary ' and outter boundary
' .
[13], tissue thickness measurement [36], and defining orthogonal
curves for template matching [30].
Assume we have a doubly connected domain , which has
two boundaries: the inner boundary and the outer boundary
, as shown in Fig. 1.
The real component of is given by
with and (23)
A cut is then defined from to starting from an arbitrary
point on along the negative gradient direction of and
meet the inner boundary at . Cut , and form a new
closed and oriented boundary of domain
with the constraint that is on the left hand side of . The
boundary condition of the imaginary component on is
given by
(24)
according to the Cauchy–Riemann equations. is any given
point on that satisfies .
Finally, another Laplace equation
(25)
is solved to give the value of inside domain .
Numerically, we are working on a triangulated domain and
the Laplace equations are solved by a standard FEM technique
as we used for colon visualization [13]. Once the analytic function
is obtained, a curvilinear harmonic polar coordinate
system can be defined by taking as one coordinate and as
another. can be thought of as a curvilinear “radius” and as
the “angle”. By scaling and using a constant, we can make
run from 0 to . Thus, the annulus domain is mapped
onto a rectangular region in an angle preserving manner. Fig. 11
shows such a parameterization on a MR heart image with the
ventricle area excluded.
1486 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007
B. Optimal Mapping Between Doubly Connected Domains
We follow exactly the same steps as we used for rectangular
domains to find the optimal mapping. The first step is to construct
an initial MP mapping . By applying harmonic parameterization,
is mapped onto a rectangle region
via conformal mapping in an angle-preserving
manner. If we define as
(26)
the mapping from to is a MP mapping. Similarly, domain
is mapped onto another rectangular region
via , where is
(27)
Since and are both rectangular regions after the harmonic
parameterization, it is easy to define an MP mapping between
them using the method presented in Section III-A.
The whole process can be illustrated by the diagram in the
equation shown at the bottom of the page.
The resulting initial mapping between and is a composition
of , and , such that
(28)
Obviously, is an MP mapping since , and are
all MP mappings.
Next, we evolve according to (17) and then (18) to find
the optimal mapping . Due to the irregularity of the domains,
an FEM method is used to solve the Poisson equation (16) on
a triangulated surface. An upwind scheme is applied for computing
. For all other derivatives, we use a least mean squares
(LMS) method to numerically calculate the first-order spatial
derivatives. For example, assume that a given point has
neighbors , and a function is defined
such that for , it is easy to see that
the derivatives of should satisfy
where is the position difference matrix given by
(29)
(30)
When applying harmonic parameterization, different selection
of cutting point may lead to different initial mapping
. However, the gradient descent method allows every point
on the boundary to move along the boundary. Each point will
Fig. 2. Two given flame images. (a) Starting image; (b) ending image.
arrive at its optimal position eventually no matter how it is positioned
initially.
V. RESULTS
In this section, we show the results of applying the proposed
methods to various types of imagery. We first test the modified
MP morphing with SSD and MI as the comparison terms on
images of fire flames. The second example is an ocean wave
video. By selecting several key frames from the video, we interpolate
the in-between frames using our proposed method and
generate a continuous video. Our method seems well-suited to
these applications because the images lack obvious landmarks,
require nonrigid mappings for good registration. Further, the intensity-based
mass-preservation constraint is reasonable as it reflects
physical reality. Registration and interpolation of smoke,
clouds, flames and fluids is an active area of research in the computer
graphics and image processing community. See [10], [24],
and [33], for example, for related methods.
This can serve as an image compression method by storing
the selected frames and generating the omitted frames on-the-fly
when the video is played. Our extension of the MP morphing
algorithm to doubly connected domains is tested over two MR
heart images acquired at the systolic and diastolic phases of
the cardiac cycle. By applying the proposed method to the
images with heart ventricle regions excluded, we are able to
generate a natural heart beat through the cardiac cycle using
only two given images. The results as images are listed in this
paper, and the results as videos can be found on our webpage at
ZHU et al.: IMAGE MORPHING TECHNIQUE 1487
Fig. 3. Morphing sequence of the flame imagery without comparison terms.
Fig. 4. Morphing sequence of the flame imagery using SSD as the comparison term.
http://www.bme.gatech.edu/groups/minerva/publications/papers/zhu-extra-index.html.
We now explain the results in more detail. The first example
is image morphing between two flame images. The starting
image [Fig. 2(a)] and the ending image [Fig. 2(b)] are from
a flame video sequence in the Artbeats Digital Film Library
(http://www.artbeats.com). The two images are the 24th and
the 29th frames in the video. Fig. 3 shows the result of applying
pure MKP without a comparison term. Figs. 4 and 5 are
the results of using SSD and MI as the comparison term,
respectively. Fig. 6 compares the results generated by the
three different methods at time (the starting frame
is at and the ending frame is at ). In the pure
MKP result [Fig. 6(a)], the double exposure effect is
obvious. By adding SSD and MI as the comparison term,
the double exposure effect has been reduced effectively as
shown in Fig. 6(b) and (c). The results of SSD and MI are
very similar, however, since both of the image frames come
from the same imaging modality. MI has greater flexibility
in handling the case where the two image frames come from
different imaging modalities, such as PET and MRI in medical
imaging, but a better function is needed to be defined to
relate the image pixel intensity with the mass in each of the
modalities, which is beyond the scope of this paper.
As we have mentioned earlier, adding a comparison term in
the energy functional leads to the presence of curl in the final
deformed grids. Although SSD is able to reduce the double exposure
effect slightly better than MI, it causes broken effects at
some edges between the bright and the dark regions due to the
high value of curl remaining in these regions. Hence, there is
a tradeoff between reducing the double exposure effect and reducing
curl. In our previous two-step approach [14], [40], we
evolved directly from according to energy functional (4).
As a result, the remaining curl is mostly in the “bright” region of
the image and it causes unnatural effects as a result. In this work,
by starting with a curl-free mapping , the algorithm makes
corrections mainly close to the edges of high intensity regions
and low intensity regions. Since less energy is required to move
pixels within low density regions, the remaining curl is mostly
in the “dark” region of the image and it is almost unnoticeable.
These effects are more obvious when observed dynamically in
videos. Fig. 7 shows the deformed grids for these three cases.
In the second example, a few frames are selected from an
ocean wave video, with about 300-ms intervals between two
consecutive frames. These frames are used as key frames for
image morphing. Our algorithm is applied to two consecutive
key frames to interpolate the motion of the wave. In this example,
SSD is used as the comparison term. The images in the
1488 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007
Fig. 5. Morphing sequence of the flame imagery using MI as the comparison term.
Fig. 6. Comparing results generated by pure v MKP, SSD, and MI at � aHXS. (a) Pure v MKP; (b) SSD; (c) MI.
leftmost column of Fig. 8 are the key frames selected from the
original video sequence, and an image other then those in the
leftmost column comes from the morphing result between the
first image in its row and the first image in the next row. The
last key frame is omitted and not shown here.
The third example illustrates image morphing over two
doubly connected domains. Two 256 256 MR images of a
human’s heart were acquired using a GE MRI scanner at the
diastolic [Fig. 9(a)] and the systolic [Fig. 9(b)] phases of the
cardiac cycle. Dark regions in Fig. 10(a) and (b) are two doubly
connected domains, corresponding to the myocardium and
other tissues other than the heart ventricle. We also use image
intensity as the mass density in this case, given the fact that MR
image intensity is a function of photon density and, thus, related
to tissue mass density. The dark regions in Fig. 10 are chosen as
natural candidates to apply MP morphing to (in contrast to the
heart ventricle region in which the change in mass is too drastic
to satisfy the MP condition). Harmonic parameterization is
first performed over each of the two doubly connected domains
(Fig. 11 shows the parameterization over the distolic phase
image), and an FEM-based MKP is solved between the
two domains to find the correspondence. In this example, pure
MKP without comparison terms is adopted. Fig. 12 shows
the final deformed grids. The in-between images can then be
generated by cross dissolving over the entire domain, where
the deformation inside the heart ventricle is obtained by 2-D
spatial interpolation. Fig. 13 shows selected frames from the
morphing result.
VI. CONCLUSIONS AND DISCUSSION
In this paper, we applied a modified Monge–Kantorovich
flow to the problem of image morphing by adding a comparison
term to the optimal mass transport energy functional.
Other than the morphing examples presented in this paper,
this technique can also be used for the problem of medical
image registration, if the underlying physics model satisfies the
mass preserving condition. We are currently using the image
intensity or a smoothed version of image intensity as the mass
ZHU et al.: IMAGE MORPHING TECHNIQUE 1489
Fig. 7. Comparing deformed grids generated by pure v MKP, SSD, and MI. (a) Pure v MKP; (b) SSD; (c) MI.
density, which is the simplest mapping function from intensity
to mass. This requires the initial and end image to be similar
in the sense that they have the same amount of mass in order
to satisfy the mass-preserving condition. For images with only
certain regions that satisfy this condition, we perform image
segmentation first and apply the morphing method only on the
mass-preserving regions. This can be done through solving the
MKP between two doubly connected domains. If the inner
boundary is small enough, it can be regarded as a landmark,
and, hence, we have solved the problem of MP morphing with
a single landmark. This technique can be extended to multiconnected
domains (corresponding to multiple landmarks),
simply by dividing each domain into several doubly connected
domains to build the initial mapping [38].
The mass moving penalty studied in this paper may be
sometimes too severe. It tends to favor changes in density over
moving the mass around. In fact, this is the main reason that
causes the double exposure effects commonly seen in image
morphing applications. type mass moving penalty is much
less severe for large movement, but it may produce nonsmooth
results. To solve this problem, we are considering the use of
penalty in our future research, with being a value between
0 and 1.
The MP constraint can also be combined with the concept of
harmonic mapping to provide a new approach of MP diffeomorphisms,
i.e. to consider the minimization of the Dirichlet integral
over all MP maps
(31)
The underlying physical assumption of MP mapping is
exactly the same as that of extended optical flow constraint
(EOFC) [4]. Developing new optical flow algorithms within the
framework of optimal mass preserving mapping can also be an
interesting future direction.
APPENDIX
In this Appendix, we give mathematical details omitted in the
main sections of this paper. We will mainly focus on the proof
of MP mapping properties and the derivation of our improved
gradient descent method.
A. Properties of Mass Preserving (MP) Mappings
We have stated in Section III-B that in order to preserve the
MP condition, the evolution of and must satisfy
(32)
(33)
for some vector field on , with and
on , being the normal to the boundary of .Nowwe
give the proof to this statement. Since is an MP mapping from
to itself, according to the Jacobian constraint we have
. By differentiating it with respect to time ,we
get
(34)
1490 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007
Fig. 8. Morphing sequence of a wave imagery. Leftmost column: selected key frames. Others: Interpolated images. SSD used as the comparison term.
Hence, must have the following form:
(35)
for some vector field on , with and
on , being the normal to the boundary of . Since and
satisfy
by taking the derivative of (36) with respect to ,wehave
(36)
From (35), we get
(37)
B. Gradient Descent Methods for Pure MKP
In the following discussion, we intend to minimize a more
general form of the energy functional
(38)
where is a non-negative function. When ,
it is exactly the Kantorovich–Wasserstein functional. The
first step is to find an initial MP mapping as stated in Section
III-A. The second step is to minimize the energy functional
ZHU et al.: IMAGE MORPHING TECHNIQUE 1491
Fig. 9. Two MR heart images. (a) Diastolic phase; (b) systolic phase.
Fig. 10. Segmentation results of the heart images in Fig. 9. (a) Diastolic phase;
(b) systolic phase.
over by varying over MP mappings from to
, starting with equal to the identity map.
A change of variable is applied here by substituting in (38)
with . Due to the MP property of and , the
following is obvious:
(39)
and also . Hence, functional (38) equals
By taking the derivative of (40) with respect to , we get
(40)
(41)
Then we do another change of variable by substituting back
with and get
(42)
Clearly, were it not for the constraint ,we
could take to decrease energy.
However, considering this MP constraint, should be
a divergence-free vector field. Hence, we define
(43)
can be decomposed into a curl-free part and a
divergence-free part (Helmholtz decomposition [29]), i.e.,
(44)
Fig. 11. Harmonic parameterization of the heart image in the diastolic phase.
Fig. 12. Deformed grids over the heart image in the systolic phase.
where and on . Then, (42) can be
rewritten as
(45)
where is chosen to be , and can be found through the
Helmholtz decomposition.
Now we give the evolving equation for in the general
case, as well as in the 2-D case which has a simpler expression
and will be used in our algorithm.
1) Gradient Descent in : By taking the divergence of (44)
on both sides, it is easy to see that is a solution of the following
Neumann-type boundary problem
on (46)
1492 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007
Fig. 13. Morphing sequence of the heart imagery using two given images in Fig. 9.
and we can set . It is then easily
seen that satisfies the MP constraints. This PDE can be solved
using a number of available methods, such as the finite-volume
method. Thus, by (37), we have the following evolution equation
for :
(47)
which is a first-order nonlocal scheme for
as minus 2 derivatives.
, if we count
If has the form of , which is the
MKP, (47) has the form of
(48)
2) Gradient Descent in : The situation is simpler in the
case, due to the fact that a divergence free vector field
can be written as for a scalar function , where
represents rotation by counter clockwise, so that
. In this case, (45) becomes
where the decomposition of is
, and we can let equal . Hence
Considering
(49)
(50)
(51)
function can be solved by the following Dirichlet-type
boundary problem
which gives us the evolution equation
In the MKP, (53) can be rewritten as
where is an identity map.
on (52)
(53)
(54)
C. Gradient Descent Methods for Energy Functionals With a
Comparison Term
We now rewrite the energy functional as the sum of two terms
(55)
with being the comparison term penalizing the change
in intensity. The second term of the energy functional
is exactly the MKP, which
has been discussed above. We focus on the first term , and
discuss the cases of SSD and MI, respectively.
1) SSD as the Comparison Term: If SSD is employed as the
comparison, then
(56)
Before taking the derivative, we multiply by and divide
by , i.e.,
By setting and considering (39):
, we get
(57)
Then we take the derivative of with respect to and find that
By changing back to , we see that
(58)
(59)
ZHU et al.: IMAGE MORPHING TECHNIQUE 1493
Adding the derivative of the term, is defined as
(60)
which is (19) in Section III.
2) MI as the Comparison Term: If MI is used as the comparison
term
Now we take the derivative of (61) respect to as in [16]
Considering
(61)
(62)
(63)
it is easy to see that the last term equals zero; hence, (62) can be
written as the following:
(64)
where , , and can be computed using (7), (8),
and (9), respectively. can be computed as
follows. First, (9) can be rewritten as
(65)
where the same method of multiplying and dividing has been
applied. Then by setting ( is an MP mapping
from onto itself) and noticing , we get
The derivative of is given as
(66)
(67)
where is the partial derivative of with respect to its first
component. Now, we do the change of variable again by substituting
with and derive
(68)
Hence, the derivative of with respect to is given by (69),
shown at the bottom of the page. Equation (69) is a quadruple
integral, which could be rewritten in a concise form by using
convolution
(70)
(69)
1494 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007
Now it has the form of a double integral on domain, and can
be combined with the derivative from the term. Thus, is
(71)
which is (20).
It is easy to deduce the evolution equation for , using the
same approach for pure MKP. In the case, we find a
scalar function to be the solution of the following Neumanntype
boundary problem
on (72)
and we can set . The evolution equation for is
given by
(73)
with being either or .
In the case, function can be solved by the following
Dirichlet-type boundary problem
and the evolution equation for is given by
on (74)
(75)
REFERENCES
[1] S. Angenent, S. Haker, and A. Tannenbaum, “Minimizing flows for the
Monge–Kantorovich problem,” SIAM J. Math. Anal., vol. 35, no. 1, pp.
61–97, 2003.
[2] T. Beier and S. Neely, “Feature-based image metamorphosis,” in Proc.
SIGGRAPH, 1992, vol. 26, pp. 35–42.
[3] J. D. Benamou and Y. Brenier, “A computational fluid mechanics solution
to the Monge–Kantorovich mass transfer problem,” Numer. Math.,
vol. 84, pp. 375–393, 2000.
[4] A. Bimbo, “Optical flow computation using extended constraints,”
IEEE Trans. Image Process., vol. 5, no. 5, pp. 720–739, May 1996.
[5] Y. Brenier, “Polar factorization and monotone rearrangement of vectorvalued
functions,” Commun. Pure Appl. Math., vol. 64, pp. 375–417,
1991.
[6] G. E. Christensen, R. D. Babbitt, and M. I. Miller, “Deformable
templates using large deformation kinematics,” IEEE Trans. Image
Process., vol. 5, no. 10, pp. 1435–1447, Oct. 1996.
[7] M. Cullen and R. Purser, “An extended lagrangian theory of semigeostrophic
frontogenesis,” J. Atmos. Sci., vol. 41, pp. 1477–1497,
1984.
[8] B. Dacorogna and J. Moser, “On a partial differential equation involving
the Jacobian determinant,” Ann. Inst. H. Poincaré Anal. Non
Linéaire, vol. 7, pp. 1–26, 1990.
[9] R. Duda, P. Hart, and D. Stork, Pattern Classification. : Wiley-Interscience,
2000.
[10] R. Fattal and D. Lischinski, “Target-driven smoke animation,” ACM
Trans. Graph., vol. 23, no. 3, pp. 441–448, 2004.
[11] W. Gangbo and R. McCann, “The geometry of optimal transportation,”
Acta Math., vol. 177, pp. 113–161, 1996.
[12] E. Haber and J. Mondersitzki, “Numerical methods for volume preserving
image registration,” Inv. Probl., vol. 20, pp. 1621–1638, 2004.
[13] S. Haker, S. Angenent, A. Tannenbaum, and R. Kikinis, “Nondistorting
flattening maps and the 3D visualization of colon CT images,” IEEE
Trans. Med. Imag., vol. 19, no. 7, pp. 665–670, Jul. 2000.
[14] S. Haker, L. Zhu, A. Tannenbaum, and S. Angenent, “Optimal mass
transport for registration and warping,” Int. J. Comput. Vis., vol. 60,
no. 3, pp. 225–240, 2004.
[15] A. Hassanien and M. Nakajima, “Image morphing of facial images
transformation based on navier elastic body splines,” in Proc. IEEE
Computer Animation, 1998, vol. 8, pp. 119–125.
[16] G. Hermosillo, C. Chefd’Hotel, and O. Faugeras, “A variational approach
to multi-modal image matching,” Int. J. Comput. Vis., vol. 50,
no. 3, pp. 329–343, 2002.
[17] M. Iwanowski, “Image morphing based on morphological interpolation
combined with linear filtering,,” J. WSCG, 2002.
[18] T. Kaijser, “Computing the Kantorovich distance for images,” J. Math.
Imag. Vis., vol. 9, pp. 173–191, 1998.
[19] L. Kantorovich, “On a problem of Monge,” Uspekhi Mat. Nauk., vol.
3, pp. 225–226, 1948.
[20] M. Knott and C. Smith, “On the optimal mapping of distributions,” J.
Optim. Theory, vol. 43, pp. 39–49, 1984.
[21] S.-Y. Lee, K.-Y. Chwa, J. Hahn, and S. Y. Shin, “Image morphing
using deformation techniques,” J. Vis. Comput. Anim., vol. 7, no. 1, pp.
3–23, 1996.
[22] E. Levina and P. Bickel, “The earth mover’s distance is the mallow’s
distance: Some insights from statistics,” in Proc. IEEE Int. Conf. Computer
Vision, 2001, pp. 251–256.
[23] R. McCann, “Polar factorization of maps on riemannian manifolds,”
Geom. Funct. Anal., vol. 11, pp. 589–608, 2001.
[24] A. McNamara, A. Treuille, Z. Popovic, and J. Stam, “Fluid control
using the adjoint method,” ACM Trans. Graph., vol. 23, no. 3, pp.
449–456, 2004.
[25] M. I. Miller and L. Younes, “Group actions, homeomorphisms, and
matching: A general framework,” Int. J. Comput. Vis., vol. 41, no. 1–2,
pp. 61–84, 2001.
[26] S. Rachev and L. Ruschendorf, Mass Transportation Problems. New
York: Springer, 1998, vol. I & II.
[27] Y. Rubner, “Perceptual metrics for image database navigation,” Ph.D.
dissertation, Stanford Univ., Stanford, CA, , May 1999.
[28] Y. Rubner, C. Tomasi, and J. Guibas, “The Earth Mover’s Distance as
a Metric for Image Retrieval,” Tech. Rep. STAN-CS-TN-98-86 Dept.
Comput. Sci., Stanford Univ., Stanford, CA, Sep. 1998.
[29] G. Strang, Introduction to Applied Mathematics. Wellesley, MA:
Ellesley-Cambridge, 1986.
[30] H. Tagare, “Deformable 2-d template matching using orthogonal
curves,” IEEE Trans. Med. Imag., vol. 16, no. 1, pp. 108–117, Jan.
1997.
[31] A. Toga, Brain Warping. San Diego, CA: Academic, 1999.
[32] A. Trouve, “Diffeomorphism groups and pattern matching in image
analysis,” Int. J. Comput. Vis., vol. 28, no. 3, pp. 213–221, 1998.
[33] A. Treuille, A. McNamara, Z. Popovic, and J. Stam, “Keyframe control
of smoke simulations,” ACM Trans. Graph., vol. 22, no. 3, pp. 716–723,
2003.
[34] P. Viola and W. M. Wells, III, “Alignment by maximization of mutual
information,” Int. J. Comput. Vis., vol. 24, no. 2, pp. 137–154, 1997.
[35] G. Wolberg, “Recent advances in image morphing,” in Proc. Int. Conf.
Comput. Graph., 1996, pp. 64–71, IEEE.
[36] A. Yezzi and J. Prince, “An eulerian PDE approach for computing
tissue thickness,” IEEE Trans. Med. Imag., vol. 22, no. 10, pp.
1332–1339, Oct. 2003.
[37] C. Zhang and F. S. Cohen, “3-D face structure extraction and recognition
from images using 3-D morphing and distance mapping,” IEEE
Trans. Image Process., vol. 11, no. 11, pp. 1249–1259, Nov. 2002.
[38] L. Zhu, “On visualizing branched surface: An angle/area preserving
approach,” Ph.D. dissertation, Georgia Inst. Technol., Atlanta, 2004.
[39] L. Zhu, S. Haker, and A. Tannenbaum, “Flattening maps for the visualization
of multibranched vessels,” IEEE Trans. Med. Imag., vol. 24,
no. 2, pp. 191–198, Feb. 2005.
[40] L. Zhu, Y. Yang, A. Tannenbaum, and S. Haker, “Image morphing
based on mutual information and optimal mass transport,” in Proc. Int.
Conf. Image Process., 2004, pp. 1675–1678.
[41] L. Zhu, S. Haker, and A. Tannenbaum, “Mass preserving registration
for heart MR images,” in Proc. MICCAI, pp. 147–154.
ZHU et al.: IMAGE MORPHING TECHNIQUE 1495
Lei Zhu received the B.S. and M.S. degrees in
biomedical engineering from Tsinghua University,
Beijing, China, in 1998 and 2000, respectively, and
the Ph.D. degree in bioengineering from the Georgia
Institute of Technology, Atlanta, in 2004.
From 2005 to 2006, he was with Mindray Medical,
a medical device company based in Shenzhen, China.
In 2006, he joined Siemens Corporate Technology,
Beijing, China, as a Research Scientist. His current
research interests include ultrasound imaging and
validation methods for medical image processing.
Yan Yang received the B.S degree from the Department
of Biomedical Engineering, Tsinghua
University, Beijing, China, in 2002. She is currently
pursuing the Ph.D. degree at the Georgia Institute of
Technology, Atlanta.
Her research interests include medical imaging,
image processing, and computer vision.
Steven Haker received the Ph.D. degree in control
and dynamical systems from the University of Minnesota,
Minneapolis, in 1999.
He is presently a Research Scientist at the Surgical
Planning Laboratory of the Brigham and Women’s
Hospital of Harvard Medical School, Boson, MA.
His research interests are in medical imaging, image
processing, and computer vision.
Allen Tannenbaum was born in New York City in
1953. He received the Ph.D. degree in mathematics
from Harvard University, Cambridge, MA, in 1976.
He has held faculty positions at the Weizmann
Institute of Science; McGill University, Montréal,
QC, Canada; ETH, Zurich, Switzerland; the Technion—Israel
Institute of Technology, Haifa; the
Ben-Gurion University of the Negev, Israel; and
the University of Minnesota, Minneapolis. He is
presently the Julian Hightower Professor of Electrical
and Biomedical Engineering at The Georgia
Institute of Technology, Atlanta/Emory. He has done research in image
processing, medical imaging, computer vision, robust control, systems theory,
robotics, semiconductor process control, operator theory, functional analysis,
cryptography, algebraic geometry, and invariant theory.
