https://doi.org/10.1007/s00371-021-02235-x O R I G I N A L A R T I C L E
Handling gaps for vector graphics coloring
Jie Jiang1 ·Hock Soon Seah1·Hong Ze Liew2
Accepted: 30 June 2021 / Published online: 15 July 2021
© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2021
Abstract
In computer-assisted 2D vector drawing systems, due to the precision of stroke representation, a stroke endpoint may not precisely connect to another stroke or endpoint during drawing so that the stroke endpoints become dangling. We call such dangling endpoints, gap points. With the presence of gaps on the region boundaries, regions formed by strokes frequently cannot be correctly colored using the standard ‘flood fill’ algorithm. In this paper, we propose a stroke-based technique to handle gaps in a vector drawing. The main contribution of our work is automatic computation of gap sizes of gap points without a predefined gap size and using a least energy method to handle gap points by estimating their relationship with surrounding strokes. Our approach consists of two major steps. Firstly, we cluster gap points in a drawing using Minimum Spanning Tree. Secondly, for each endpoint cluster, we use a least energy method together with predefined gap handling priority to estimate the connection of gap points. We demonstrate the effectiveness of our approach by applying it to line drawings with unresolved gap points.
Keywords Vector graphics coloring·Gap size independence·Minimum spanning tree-based clustering·Least energy function
1 Introduction
In the past two decades, scalable vector graphics (SVG) has become popular due to many advantages. One of the features is that elements, e.g., points and strokes, are mathematically represented and stored in floating values so that a drawing can be scaled without loss of image quality. On the other hand, this introduces problems, e.g., gaps. Due to the floating-point precision of stroke representation, a stroke endpoint may not precisely connect to a destination stroke during draw- ing. Although two points may appear at the same integer position on the screen, their floating point values could be different, which results in a gap. Gaps are also frequently seen in automatically generated drawings, e.g., by 2D stroke- based inbetweening [2], for example, gaps (a), (b), and (c) in Fig. 1. They are introduced by the inbetweening func- tion. In this paper, we call dangling stroke endpoints,gap points. The presence of gap points prevents regions from being properly closed and correctly colored [15], which can
B
Jie JiangJJIANG009@e.ntu.edu.sg
1 School of Computer Science and Engineering, Nanyang Technological University (NTU), Singapore, Singapore
2 CACANi Private Limited, Singapore, Singapore
also be a problem in region-based graphical techniques such as [13]. Manual gap point handling before coloring is doable for one single drawing. But it is time-consuming and becomes impractical to process a large amount of drawings in scenar- ios such as automatic inbetweening.
Some approaches have been proposed to handle gaps.
Snapping, such as [6], is frequently used in drawing systems to snap endpoints to prevent gaps from happening. How- ever, it only applies during drawing. VGC [4] can be used to edit vector strokes, e.g., gluing gap points. Some automatic gap closing methods have been proposed. Gaps are detected and/or filtered by gap size [1,9] or Delaunay triangulation [17]. Gaps are then closed via straight lines [1,7,9,17] and/or splines [7]. However, there are still several major challenges:
– Gap size sensitivity.In approaches, such as [1], a gap size is used to determine the gaps to be closed. Given different gap sizes, a gap point may connect to different points to close a gap. For a drawing with gaps of largely different sizes, users may need to carefully select appropriate gap sizes with trial and error to paint.
– Stroke connection. In Live Paint [1], gap points are handled individually. Although [7] and [17] can handle end-to-end connections, they support no more than two
Fig. 1 An intermediate frame generated from two key frames. (a), (b), and (c) are three local gaps. (a1), (b1), and (c1) are the results produced via Live Paint. (a2), (b2), and (c2) show the results produced by our method. Closed regions are manually colored
points. Moreover, in [7], gap points may connect to a wrong stroke due to its gap detection method.
– Appearance. Many previous methods close a gap by adding a straight line. It works when closing small gaps.
However, it causes problems, e.g., sharp bends, especially when gap is large. For example, gap (c), which is between the ‘face’ and the ‘hair,’ should smoothly connect to the face as shown in Fig.1(c2) rather than (c1). It is important to consider aesthetic appearance.
In this paper, we propose an automatic gap point handling method to close gaps in a drawing to form closed regions for coloring. Firstly, we do a gap point analysis (Sect.3.1). In this step, we look for the relationship between gap points and their surrounding strokes by considering gap points’ local geometry. A gap size is then computed for each gap point (Sect.3.2). Secondly, we do a gap point clustering to group nearby gap points (Sect.3.3). Thirdly, we handle each gap point cluster with an energy function (Sect.4). After gap closing, colors are added to the drawing (Sect.4.5). These steps are illustrated in Fig.2. The main contributions of our work are:
– We introduce gap point analysis without needing a pre- defined gap size. We establish the relationship between a gap point and its surrounding strokes. This relationship provides information to determine the destination stroke and point that a gap point should connect to. Based on the relationship, we compute gap sizes of gap points without the need of a predefined gap size for gap closing.
– We propose a gap point clustering method to detect nearby gap points based on the above gap point anal- ysis. It detects nearby gap points that are supposed to form end-to-end connections. This reduces the problem to closing local gap point clusters.
– We introduce a least energy function-based gap detection method. It balances distance and angle deviation so that
Fig. 2 Overview.aAutomatic gap size computation for every gap point based on local geometry including nearest point and extensional point as in the close-up view. The gap points are indicated by black dots.
The green circle is a gap size area.bMST-based gap point clustering.
Circled gap points belong to one cluster.cGaps are detected and closed for each cluster to form closed regions that are highlighted in different colors.dColoring after closing gaps. (Original source of the drawing:
[17])
the destination point of a gap point will not deviate dras- tically from the gap point and its tangent. This improves the appearance of the gap closing results.
2 Related work
Single-point approaches.Automatic gap closing for freehand drawings was proposed in [9]. In [9,22], gap points are han- dled individually. However, sometimes two gaps points (or more) are very close and they should be considered as an end-to-end connection. Therefore, these nearby gap points should be considered as a whole rather than being handled individually. Individual gap point handling usually produces sub-optimized results. LazyBrush [23] converts gap closing problem into an optimization problem. However, it does not consider gap point’s direction. Single-point approaches are simple but often generate undesirable results, such as sharp bends and misdetection of end-to-end connections. In our method, we cluster gap points and locate gap points where multiple-point connections may exist.
Clustering.Clustering is to detect groups from input datasets.
It is an important method and many clustering approaches have been proposed [16]. Some widely used methods include k-means clustering [5], mean shift algorithm [3], and density- based clustering [19]. Ink-means clustering, the number of clusters needs to be predefined. However, the number of gap points in a 2D drawing can be arbitrary and the num- ber of clusters is unknown. For a density-based algorithm, it is hard to specify a density threshold as the distribution of gap points does not usually follow any pattern, e.g., one gap point can constitute a cluster such as Fig.1a. When cluster- ing gap points, local geometry should be considered, e.g., the distance between gap points. Therefore, we analyze local geometry and cluster gap points with the Minimum Spanning Tree (MST). Our method is able to detect one-point clusters.
V connection. In some cases, multiple nearby gap points should be joined together to form an end-to-end connection.
However, many previous approaches such as [7,11,17,20]
are only able to handle no more than two points. An end- point fusing method [24] was proposed. But it is designed for freehand-sketched polyhedrons. In [7], it uses splines to connect two gap points. However, it depends on predefined parameters, e.g., maximum spline angle. In our method, we introduce an energy function for end-to-end connection.
Appearance. To close a gap, the simplest way is to use a straight line to bridge a gap point and a destination point such as [1,9]. Although it is able to close gaps, it ignores angle deviation and often produces undesirable sharp bends.
Hence, appearance should be considered. A skeletal line join- ing algorithm [22] introduces distance and angle to prevent
‘sharp bend.’ Fourey et al. [7] introduce parameters, e.g., maximum angle and distance, to guide the results of gap closing. However, its gap detection method may fail to find correct stroke connection when the gap size is large. Live Paint [1] also produces different results with different input gap sizes. Parakkat et al. [17] propose a gap size-independent method based on Delaunay triangulation. But it always uses straight lines to close gaps and the result strongly relies on triangulation. In our method, we consider appearance by computing gap sizes and using a least energy function to estimate destination points that will not deviate too much from gap points’ tangents.
Other approaches.In recent years, machine learning meth- ods, such as CNN-based [25] and GAN-based [10], have been proposed for line art coloring. However, these methods do not consider geometric information in drawings, and sometimes, it is hard to produce desired results even with user interac- tion as addressed in [17]. In addition, machine learning-based methods require priori knowledge to train models. Liu and Zhang [14] proposes a method to color a grayscale image with a source colored image. It needs to extract the luminance his- tograms of the input and the source images. However, when coloring a line drawing, the luminance histogram is not avail- able. Curve reconstruction [18] can be used to produce a set of closed boundaries. However, it depends on the sampling rate and gaps are closed by straight lines due to Delaunay tri- angulation and the result is similar to [17]. Image inpainting, such as [21], is an interesting direction to restore shapes in a drawing. It needs datasets for training.
3 Gap point analysis and clustering
3.1 Gap point analysisThe input to our method is a 2D vector line drawing. Each stroke is a polyline. Each dangling stroke endpoint in the drawing is considered as a gap point. Handling a gap point is
Fig. 3 Gap size computation.aThe red and the blue lines indicate the extensional and the nearest distances. The extensional distance is much greater than the nearest distance.bThe extensional and the nearest distances are not much different.c–eshow the gap sizes, indicated by colored circles, computed with differentkvalues
to use a stroke to connect the gap point Pendto a destination point Pt. Angle deviation measures how much the destina- tion point deviates from a gap point’s tangent. We define the tangent of a gap point Pendasv. The angle deviation of Pt
is the angle betweenvandPt−Pend.
Handling a gap point involves two questions: (1) which is the destination stroke that the gap point should connect to? (2) where the destination point should be? We use two geometric characteristics to address these two questions:
– Nearest point,Pnear. The nearest point to a gap point is a potential destination to close the gap, e.g., the blue line in Fig.3a. When a user draws a stroke to connect to another stroke, the pen goes toward the destination stroke. The nearest stroke is a possible candidate as the destination stroke. The nearest distance is denoted asdnear.
– Extensional point, Pext. The extensional point along a gap point’s tangent and closest to the gap point, e.g., the red line in Fig.3a, is another candidate to close the gap.
The distance between the gap point and the extensional point is denoted asdext. To bridge a gap point, extending along its tangent can avoid a sharp bend as adopted in [7].
As we explained,Pnearindicates the nearest destination point to connect to andPextis a candidate that avoids sharp bend.
However, these two ways give rise to a conflicting issue.
Connecting to Pnear allows the gap to be closed with the shortest line, but the angle deviation can be big. Connecting to the extensional point produces a smoother result. But it may be a wrong connection such as Fig.3a. Therefore, we compute gap size to find candidate areas of destination points by balancing the influences of PnearandPext.
3.2 Gap size computation
In our method, a gap size defines a circular area centered at a gap point. The idea is to balance the energy of angle and distance. Assume Pext and Pnear are two candidates.
Connecting toPextintroduces no energy in bending the gap point, while the distance can be long. Similarly, although connecting to Pnear costs less energy in distance, it costs energy to bend the gap point. Therefore, if a destination point has a long distance and a large angle deviation, the energy will be high.
We have a gap point Pend, two geometric characteristic pointsPextandPnear, and an arbitrary pointPcandi. We use dcandiandθcandito represent the distance and the angle devi- ation ofPcandi. To compare the energy in distance and angle deviation, we propose a distance term and an angle term by normalizingdcandiandθcandias follows:
Tdist= dcandi−dnear
dext−dnear ,Tangle= θcandi
θnear.
Note thatdext−dnear=0. Whendext=dnear,Pnearhas the shortest distance and no angle deviation. Hence,Pnearshould be the destination point. Whendcandi>dext, i.e.,Tdist >1, Pcandibecomes meaningless as Pext has a shorter distance and no angle deviation. The angle term has the same pattern.
Therefore,TdistandTangleshould be within[0,1]. These two normalized terms indicate the relationship between Pcandi
and two geometric characteristic points. As we explained before, whendext >> dnear, extending along tangent may result in a wrong connection as shown in Fig.3a. In other words, whendext >> dnear, the distance is more important than the angle term, i.e., Tdist should be larger. However, whendextincreases,Tdistdecreases. Therefore, we propose two weighted factors:
Edist = dext−dnear
dext ·Tdist, (1)
Eangle= dnear
dext ·Tangle. (2)
Whendext >> dnear, the distance term has a larger weigh- tage, i.e., the energy of distance is high. The total energy of connectingPendtoPcandiis computed as:
Esi ze(Pcandi)=(1−k)·Edist+k·Eangle, (3) where k is an angle factor that controls the influence of distance and angle deviation. A gap point’s gap size is deter- mined by finding a pointPcandithat costs the least energy in bending and extending, which is to find the solution leading Eq.3to its minimum.
Whenk = 0, the energy function fully relies on Edist
and the nearest distance will be the gap size as shown in Fig.3c because Edist =0. Whenk =1, the distance term has no influence, i.e., the optimal point should have the least angle deviation as shown in Fig.3e. When 0 <k <1, the optimum gap size will be within(dnear,dext)as illustrated in Fig.3d. Figure 4further shows the effect ofk. Whenk
Fig. 4 Effect of angle factork. The shown local areas are indicated in the gold fish example in Fig.13. The red segments are appended lines to close gaps
increases, the computed gap size should be larger. The gap closing result will be ‘closer’ to tangent as a larger gap size increases the possibility for a gap point to find a destination point along its tangent. According to our experiments of gap closing with differentkvalues as shown in Fig.4, we found that our method produces stable results when 0.6≤k≤0.8, which works well for other examples in this paper. Hence, we setkto 0.7.
As Eq. 3involves angle, its optimization is a nonlinear problem. We solve it by converting it into a discrete problem in three steps: (1) assuming a circle, whose radius isdext, centered at the gap point and finding all segments in the circle; (2) sampling points on these segments and ensuring that the distance between two neighboring points is small enough; (3) computing energy for all the points and picking the one with the least energy.
3.3 Gap point clustering
After computing gap sizes, we use the computed gap sizes to cluster gap points in order to find potential end-to-end connections. A cluster consists of at least one gap point. Each cluster is called a local gapand gap points inside a local gap are called local gap points. General clustering methods have limitations as discussed in Sect.2. Thus, we aim to use geometry to assist clustering.
If two gap points are close enough, they should be grouped together intuitively, and vice versa. For example, the gap points in Fig.1a–c should be three groups as they are distinct from each other. We establish a spatial relationship for gap points using the Minimum Spanning Tree (MST) [12] as it tends to connect nearby points. Figure5shows an example of our clustering. Imagine a graph with nodes only that are gap points in a drawing. The objective is to add edges to the graph to connect nodes so that connected nodes become one cluster. Below are the main steps.
Initialization.We initialize a graphGwith nodes represent- ing the gap points. Every two nodes are connected via an edge. Each edge’s energy is set as the Euclidean distance between two gap points.
MST construction.An MST is constructed with a subset of the initialized edges. All unused edges, which do not belong to the MST, are removed fromG. The remaining edges are
Fig. 5 Gap point clustering.aAll gap points are indicated by black dots. An MST is constructed for the gap points and the solid lines are edges. Circled nodes belong to one cluster and each un-circled point is a cluster. We denote three edges in the close-up view. Frombtod, the three edges are added to the graph sequentially.eThe final clustering results of the four nodes
Fig. 6 Energy update.aTwo gap points (nodes) are connected via edgei, whose energy, i.e., length, isL.bThe distances to the intersection points.cUpdate the energy of the edge to the minimum ofL,d1, and d2
sorted by energy and saved in a setE. When an edge is long, the two gap points are unlikely to be a group. Therefore, our clustering is interpreted as follows: if an edge’s energy is low, the two connected gap points tend to become one cluster and our objective is to keep low-energy edges.
Energy update.One of the limitations of previous methods is that they cannot handle end-to-end connection properly as they handle gap points individually. In our method, we cluster gaps points that are supposed to form end-to-end connec- tions. After clustering, end-to-end connections will be later detected in each cluster with more than two gap points and it is introduced in Sect.4.3.1. Figure6a shows two gap points (nodes) connected via an edge. Both nodes are connected via an edge edgei whose energy isL. Both points can form an end-to-end connection along their tangents, and the distances between both gap points and the intersection ared1andd2
that are less than L as shown in Fig.6b. Then, the edge’s energy is updated to min(d1,d2,L)as shown in Fig.6c. An edge with low energy means the two gap points are likely to form a candidate cluster. In addition, both nodes’ gap sizes are updated tod1andd2accordingly.
Clustering.We define a graphGthat inherits the nodes from G without any edge. The objective of clustering is to add edges toG so that connected nodes become clusters. We need to determine if an edge should be added to G. For example, in Fig.5a, three edges are labeled. Assume edge 1 is added to make the lower two nodes a cluster as in Fig.5b.
If we add edge 2 as in Fig.5c, the upper two nodes form another cluster. If we add edge 3 as shown in Fig.5d, the
Fig. 7 Effect ofγ. It shows the gap point clustering results of the three circular areas in the third example in Fig.13. Circled gap points belong to one cluster. Each non-circled point is one cluster
four nodes become one cluster and the overall edge length increases significantly, and hence, edge 3 should not be added as in Fig.5e. Our clustering method goes through every edge in Eand determines if they should be added to G based on their influence by adding them. Adding edgei ∈ E to Gconnects two sub-graphs,G1andG2. A combined graph Gcomb is defined as G1∪G2∪edgei. We compute energy valuesE(G1),E(G2), andE(Gcomb)as follows:
E(G)=
δ, Ghas only one node
i E(edgei)/Ne, Otherwise (4)
where Ne is the number of edges in G, and δ represents a node’s (i.e., a gap point’s) gap size. The influence by adding an edge is considered as energy changes imposed onG1andG2. The energy ratioEis formulated asEG= E(Gcomb)/E(G)whereGis a sub-graph connected to either side of edgei. We introduce an energy thresholdγ, and edgei is added toGwhen the condition((EG1 < γ )∧(EG2 <
γ ))∨(EG1 + EG2 < γ )is satisfied. It means that the energy values of both clusters should not change signifi- cantly if the edge is added. When γ = 0, which is the minimum value ofγ, the condition is never satisfied asEG is always positive. Hence, each gap point is a cluster. When
γ =
E(edgei)/δmin, which is the theoretical maximum ofγ, the condition is always satisfied and all the gap points become one cluster.δminrepresents the least gap size of all the gap points. The theoretical maximum and minimum val- ues are computed by estimating the maximum of E(Gcomb) and the minimum ofE(G). Whenγincreases, the condition is relaxed so that a cluster may contain more gap points as shown in Fig.7. We tested our clustering method on differ- ent examples and we found that the clustering result is stable when 1.4 ≤ γ ≤ 1.6. Therefore, we set γ to 1.5. After clustering, each cluster is assigned a gap sizeδ, which is the largest energy value of its child nodes.
4 Gap detection and coloring
4.1 Gap point handlingAfter clustering, all the gap points in a drawing are grouped into clusters. However, the number of points inside a cluster varies. We assume each cluster hasNgap points and its gap size isδ. A gap point’s local area is defined as the circular area with radiusδcentered at it. For a cluster with multiple gap points, the local area is defined as the union of individual gap points’ areas. Generally, gap points should be handled as follows:
– Close(T connection). A gap point will be connected to a non-gap point to form a T junction. We call them, T gaps.
– Close(V connection). Multiple points will be joined to the same destination point. We call such gaps, V gaps.
For a local gap with multiple points, we just need to apply one of the aforementioned operations for each gap point.
4.2 Energy functions
A major problem in previous methods is that they handle gaps to form closed regions without considering appearance. To quantitatively evaluate gap point connection, we propose an energy function. Given a gap pointPendand its destination pointPt in its local area, we compute the energy cost as:
ES(Pt,Pend)= [1−sign(θ)∗cos2θ] ·d2, (5) whereθis angle deviation that belongs to[0, π], anddis the distance between Pt and Pend. The sign(θ)equals to 1/−1 when the angle deviation is less/greater thanπ2. Withdfixed, this energy function is a monotonically increasing function ofθ. Whenθ andd are small, the energy is low. Hence, an optimal result, which is with a small angle deviation and a short distance, should result in a low energy cost. Based on Eq.5, we formulate the energy cost of handling a cluster with multiple gap points as follows:
EL =
N
i=1
ES((Pt)i, (Pend)i), (6)
where (Pt)i is the destination point of the ith gap point.
Considering appearance, the best result corresponds to the solution that minimizes Eq.6. However, a major problem is a gap can be closed differently under different contexts. For example, in Fig.8b–h are possibilities to close the gap in (a).
It is hard to deduce user’s intention from the local geome- try. In our method, we propose a general solution, which is priority-based as-many-as-possible gap closure. A cluster of gap points are resolved in the following two steps:
– Step 1:Detect V gaps from the unresolved gap points and close the one with the least energy. A V gap should connect as many gap points as possible. Repeat this step until there is no V gap detected. It allows to detect end- to-end connections formed by more than two gap points.
It is introduced in Sect.4.3.1.
– Step 2:Detect T gaps from the unresolved gap points and close the one with the least energy. Repeat this step until there is no T gap detected. It is introduced in Sect.4.3.2.
User can manipulate the gap closing results via user inter- vention. It is introduced in Sect.4.4.
4.3 Energy-based gap point handling
In this section, we will describe how to detect V and T con- nections from a cluster of gap points. We use P∗t to denote the best destination point of a gap point.
4.3.1 V gap detection
A V gap is formed by a set of gap points that should be joined to the same point along their extensions. However, for more than two points, it is hard to find an intersection point that falls exactly on their extensions. Hence, a gap point may slightly bend such as Fig.8g. Therefore, when a gap point bends drastically, it is unlikely to be a V gap. It is reasonable to assume the angle deviation to be less thanπ2. We denote the maximum angle deviation asθD in V gap detection. Given a destination point Pt, the energy to close the V gap can be formulated based on Eqs.5and6withθno more thanπ2 and rewritten as:
EvGap(Pt)=
n
i=1
< Pt−(Pend)i, (vi)⊥>2, (7)
wherenis the number of gap points forming a V gap (n>=
2), and<·,·>represents vector dot product. Note that cosθ is represented by dot product of the vectors.(Pend)i denotes theithgap point.vi is the tangent of(Pend)i.(vi)⊥is a unit vector perpendicular tovi.
As discussed in Sect. 4.2, a V connection should close as many gaps as possible. Therefore, we find the maximum ofn, which is denoted asnmax, in the first place. As a local gap’s gap size isδand the maximum angle deviation isθD, each gap point has a solution area where the destination point should be as illustrated in Fig.9a. A solution area is a sector whose central angle and radius are 2θD andδ. Note that the solution area is convex sinceθDis less thanπ2. The bisector is the gap point’s tangent. A solution area covers candidate des- tination points that satisfy the angle and distance constraints.
For gap points forming an end-to-end connection, the desti- nation point should be within the intersection of individual
4
Fig. 8 Possible gap closing results (b–h) of a four-point gap (a). The orange lines are added lines. The green dots are intersections introduced by new segments
Fig. 9 aThe solution area of a gap point. The red dashed line is the tangent of the gap point.bA local gap with four gap points and their corresponding solution areas.cLabeling overlapping regions of solution areas of gap points 2, 3 and 4. Points inside the green region belong to all the three solution areas. Points inside the blue and the red regions are from one or two solution areas, respectively
Fig. 10 aA cluster with two gap points.bThe two gap points form a V connection whenθDincreases.cThe two gap points do not form a V connection whenθDdecreases. Inbandc, the colored regions represent corresponding gap points’ solution areas
solution areas as shown in Fig.9b and c. We call the inter- section ‘shared’ areas. Therefore, the maximum valuenmax
corresponds to the area(s) shared bynmax un-handled gap points. Note that there could be more than one area shared bynmax gap points. WhenθD increases, the solution areas will be bigger, which increases the possibility for solution areas to have shared regions as shown in Fig.10b and c. In our implementation, we tested examples and found that the gap closing results are stable when 12π ≤ θD ≤ π4. Hence, we setθDtoπ6. WhenθDincreases, gap points will be more likely to be end-to-end joined.
Withnset tonmax, V gap detection is to minimize Eq.7and the solution is within the shared areas. It can be formulated as:
P∗t = arg min
Pt∈Rshare
nmax
i=1
ES(Pt, (Pend)i) (8)
where Rshare denotes any one of the areas shared bynmax
gap points. Note that Eq.7 is convergent and has a unique global minimum point. As each gap point’s solution area is convex, anyRshareis convex as well. Hence, for eachRshare, P∗ is unique and must be inside the interior of Rshare or on its border. The global optimal point can be obtained via calculating the first order partial derivatives of Eq.7. If the optimal point is not inside the area, we take the centroid of Rshare as the optimal. Every time a V gap is closed, V connections will be re-detected for the un-handled gap points as the local topology has changed.
4.3.2 T gap detection
The energy to close a T gap can be modeled by Eq.5. Note that the cosθanddare represented by vector dot products. As far as a T gap is concerned, the destination point must be on a stroke. Hence, the energy minimization problem is restricted to a set of segments within the local area. As the sign(θ) in Eq.5is a piece-wise function, we classify the segments within the local area into two sets that are S≤π
2 and S>π 2. We define a straight linel that passesPend and is parallel tov⊥. We then put segments separated byl into S≤π
2 and
S>π
2. ForS≤π
2 andS>π
2, we use sign(θ)=1 and sign(θ)=
−1, respectively. We process each segment in these two sets individually. For each segment with two endpointsP1andP2, we replacePt in Eq.5with(1−s)P1+sP2and represent cosθby dot products of the vectors. The energy function then becomes a quadratic function ofs, and the solution is to find the minimum point within interval[0,1]. By processing all the segments, we can find the least energy point. Every time a T gap is closed, connections should be re-detected for the rest of the gap points as the local topology has changed.
4.4 User intervention
While our gap closing method works in general, there are a few specific problems that it faces:
1. Gap bridging.The two gaps in Fig.11a could be bridged differently with the context considered. Hence, gap bridg- ing depends on user’s intention that is hard to predict without context information.
2. Wrong gap closing sequence.Figure11f shows a gap with three points. However, they can be handled differently as shown in Fig.11f–h depending on user’s intention.
?
?
Fig. 11 aTwo gaps in different contexts.bSmooth connection of two gap points.cConnect two gap points by extension.dThree different candidate areas for T gap detection.eThe T gaps corresponding to the three candidate areas.fThe three gap points belong to one cluster and form a three-point V connection.gThe result with the left two gap points closed first.hThe result with the bottom two gap points closed first
We adopt some approaches from previous papers and propose some manual ways to improve our gap closing results:
1. Small area factor ks. When a small area is created after closing a gap, this gap should not be closed as such small areas are usually undesirable during coloring as addressed in [7]. In our method, we use a ratioks. We denote the total area of the entire drawing (after gap closing) as Stotal. When a drawing consists of multiple unconnected parts,Stotalrefers to the corresponding part that the small region belongs to. If a region’s area is less thanks·Stotal, it is considered as a small area. We found that it works well when 0.005≤ks ≤0.015. Hence, we set it to 0.01.
2. Dangling gap.After closing all gaps, when a gap does not participate in forming a region such as Fig.13d3, i.e., a dangling gap, it will not be closed.
3. Grouping.Since our method may produce wrong results due to wrong clustering or wrong closing sequence such as Fig.11f, we allow users to manually specify clusters as in (g) and (h).
4. Manual gap bridging.We allow users to decide whether to use a smooth curve or straight lines to connect two points. For a gap point, the destination point is found inside its local area defined by its gap size. By default, a local area is circular. We allow users to manually define a local area for gap detection. By default, a two-point V connection will be closed via a smooth curve such as Fig.11b. To connect them by extension, we specify a big area to include the intersection as shown in Fig.11c. For a T junction, we use a smooth curve to connect the gap point to its destination point, and we can tweak the result as shown in Fig.11d and e.
Fig. 12 Coloring.aA gap is closed using an invisible line with zero width, which is represented by the dashed line. Both sides are colored with different colors inband the same color inc
4.5 Coloring
Our process is to close all gaps to make the line drawing ready for coloring. After the gap closing process, all the gaps will be closed via invisible lines with zero width to form closed regions as shown in Fig.12a. In the coloring stage, colors are added to the closed regions as shown in Fig.12b.
When two adjacent regions are painted with the same color, the gaps between them will be visually ‘removed’ as shown in Fig.12c.
5 Results and discussion
5.1 ResultsWe show some results using our method and existing meth- ods (Figs.13,14,15and16) including Toon Boom,1 Live Paint [1], LazyBrush [23], Parakkat et al. [17], and Fourey et al. [7]. Due to the lack of available datasets for testing, we use some examples in previous papers [7,17]. As [7,17] are raster based, we redraw the examples in a vector drawing system.
Due to no available source code of [17], we directly use the results in its paper. Although this method is raster based, the inputs are neat line drawings and gaps appear at stroke end- points. Hence, it is fair to compare with it. For [7], we import the reproduced vector drawings into Krita2(with preferred resolution 1000 ppi) and produce results usingSmart Color- ingin G’MIC.3Although it is raster based as well, it closes gaps based on gap point detection and direction estimation.
Hence, the inputs of [17] are similar to our method so that we think it is fair to compare with it.
5.2 Discussion
Appearance and stroke connection.Previous methods have two major problems, which are (1) sharp bend caused by wrong destination point or non-smooth gap bridging, and (2) wrong destination stroke. In [1,17], they use straight lines to close gaps. When gaps are small, it is fine. When gaps are big, it usually causes sharp bends such as Fig. 13b1.
Therefore, angle deviation is an important factor when clos-
1 https://www.toonboom.com/.
2 https://krita.org/en/.
3 https://gmic.eu/.
Fig. 13 Comparison with state-of-the-art. Inb–e, the gap closing lines are in red.aThe original drawings.b–eshow the results produced by, Live Paint [1], Parakkat et al. [17], Fourey et al. [7], and ours, respectively.fOur results after coloring. (Original source of the drawings: [17])
ing a gap. In [7], it always extends gap points to form T junctions with no angle deviation. However, it may not be always desired especially when the extensional distance is much longer than its nearest distance, for example, the red appended segments in Fig.13d1 and d2. In (d1), the exten- sional distance is much longer than its nearest distance.
Therefore, the gap point has to be bent. We solve it by intro- ducing gap size computation and energy function-based gap detection. There are the following advantages: (1) narrow- ing down the destination stroke(s) to the segments within the local area defined by gap size as shown in Fig. 13d1, e1, (2) detecting gaps without too much extension or bending as shown in Fig.13d2, e2 and b1, e3, (3) ability in detecting end- to-end connections with more than two gap points as shown in Fig.14. Although our method has a major limitation that is discussed in Sect.5.3, our method is still effective in pro- ducing results with proper stroke connection. In addition, as our gap detection finds destination points in candidate areas defined by gap sizes, user can bridge a gap via a smooth curve or straight extensions by specifying its candidate area as explained in Sect.4.4.
Gap detection.Gap detection refers to destination point com- putation. In [1], gap points are handled individually. Hence, it cannot find V connections and produces sharp bends. In [7], the first step is to find all potential two-point end-to- end connections based on predefined parameters. After that, T junctions are detected. However, it may encounter prob- lems in detecting gaps. In Fig.15, the middle two columns
are produced in Krita with maximum connectivity fixed to 1 according to the example. As shown in Fig.15c1, the two gap points should be a V connection. In order to close this gap by [7], the maximum spline angle should be around23π. However, this value will wrongly close other gaps as in (a2) and (b2) which should not be V connections. By increasing the maximum spline length, (b1) can be closed properly as in (b3). However, (d1) is wrongly closed as in (d3) because they satisfy the V connection conditions. Therefore, a parameter may affect gap detection globally. In our method, we do gap point analysis and clustering before gap detection. Each clus- ter is processed independently. The gap points from different clusters will not be treated as a V connection. For (d1), the two points will not be a V connection if they are two clusters (depending onγ as discussed in Sect.3.3).
Gap Size.Some previous methods need a gap size for gap closing. However, gap size-based approaches may produce unwanted results due to the inconsistency of gap sizes in a drawing. In Live Paint, when gaps are closed with dif- ferent sizes, gap points may connect to undesired points as shown in Fig.13b1. The same problem happens in [7] where gap size corresponds to maximum spline/segment length.
When it increases, gap detection will be affected such as Fig. 15b2, b3 and d2, d3 due to its gap detection prior- ity. Moreover, when gap size increases, single gap points, which do not participate in end-to-end connection, will be extended as long as the distance is within the maximum length. It brings a problem that gap points can be wrongly
Fig. 14 Gap closing results of areas where gap points are supposed to be joined together. Closed regions are highlighted in colors
Fig. 15 Comparison with Fourey et al. [7]. The resolution of the image imported to [7] is 466×478. The first column shows the close-up views of the original drawing. The middle two columns are produced by [7]
with different maximum lengths and the last column shows our results.
(Original source of the drawing: [17])
Fig. 16 a Original line drawings.bGap closing results of the line drawings. The closed regions are highlighted in different colors. Extra regions are indicated by the red arrows.cOur results after coloring.
(Original source of the drawings: [7])
extended and a drawing can be over-segmented into small regions. Therefore, these methods are sensitive to gap size.
User needs to carefully select gap size and/or other param- eters. Although [17] closes gaps independent of gap size, it always uses straight lines as discussed before, and the result strongly relies on triangulation. In our method, due to our proposed gap point analysis, user does not have to specify a gap size for gap handling. Gap sizes are computed based on local geometric information. Each gap point has its own gap size that will not be affected by global gap size. Hence, our method produces more stable results. In our experiments, we set the gap size to the diagonal length of the input draw- ing’s bounding box. The results are shown in Fig.13e. Our method may produce extra regions such as the gold fish’s tail in Fig.13e. The reason is that the gap points around the tail have big gap sizes as their nearest and extensional distances are both large. We also produce gap closing results of draw- ings with other types of styles as shown in Fig.16a. Although there are few extra regions as indicated by the red arrows as shown in Fig.16b, most of the gap points are correctly han- dled. These extra regions require more key strokes to color, but they do not cause problems in the coloring stage as shown in Figs.13f and16c. For [7,17], even with color information, they may still fail to close gaps when stroke connection is wrong such as Figs.13d1 and15d3.
5.3 Limitations
Our method has two major limitations as follows:
1. Our method may fail to close gaps due to wrong clustering as shown in Fig.17a and e. According to our gap size computation method, the gap size of the indicated gap point in Fig.17b is small. During clustering, an MST is constructed. For the indicated gap point in Fig.17b, its gap size is much smaller than its adjacent edges as shown in Fig.17c. Hence, it will not be clustered with its adjacent gap points as shown in Fig.17d. With the gap points clustered as in (d), the gap closing result is wrong as in (e) as these gap points are handled individually. For two gap points that should belong to one cluster, when the distance between them is much longer than their nearest distances, these two points may not be clustered. In our method, if gap points can be properly clustered, e.g., with necessary user intervention as in (f), the result will be correct as shown in Fig.17g.
2. Our method fails to find gaps between non-gap points as shown in Fig.18. Our method only handles gap points that are dangling stroke endpoints so that gaps formed by non-gap points will not be detected. In our method, an additional line is required to close the shirt to be a region. To solve this problem, Delaunay triangulation- based method [17] can be utilized to pre-processed the
Fig. 17 Wrong gap closing result due to incorrect clustering.aA close- up view of a drawing with four gap points.bThe gap size area of a gap point.cMST edges. The indicated gap point’s gap size is much shorter than its adjacent edges so it is not clustered with its adjacent gap points as indwhere each point is a cluster.eThe gap closing result with the gap points clustered as ind.fManual clustering.gThe result after manual grouping. (Original source of the drawing: [17])
Fig. 18 Gap between non-gap points.aOriginal. The bow tie area of the Chameleon in Fig.13a.bOur gap closing result where a gap is not detected as indicated inc
input drawing to find all potential gaps that can be formed by non-gap points.
6 Conclusion
In this paper, we present a gap point handling method that is able to handle gaps in vector drawings. We cluster gap points based on their geometry and compute an appropriate gap size for each gap point and cluster. Each cluster is then processed and gap points are handled based on an energy function. Experiments show that our method is effective in closing gaps by estimating gap sizes and stroke connections.
In the future, we hope to investigate using context infor- mation to enhance our method. One direction is to use context information, such as stroke sequence, which implies how user draws strokes. An automatic construction method in [8] can be a solution to retrieve stroke sequence information. Deep learning-based approaches can be another direction. We can use our method in this paper to guide model training. In addi- tion, triangulation-based methods [18] can be a solution to improve our clustering method. We can rasterize the input line drawing and use [18] to find the gaps to be closed. Ana- lyzing more frames in a sequence may be helpful to infer context information if more frames are available.
FundingNo funding was received for conducting this study.
Declarations
Conflict of interest The authors declare that they have no conflict of interest.
References
1. Asente, P., Schuster, M., Pettit, T.: Dynamic planar map illustration.
In: ACM Transactions on Graphics (TOG), vol. 26, p. 30. ACM (2007)
2. Chen, Q., Tian, F., Seah, H.S., Wu, Z., Qiu, J., Konstantin, M.:
DBSC-based animation enhanced with feature and motion. Com- put. Animat. Virtual Worlds17(3–4), 189–198 (2006)
3. Comaniciu, D., Meer, P.: Mean shift analysis and applications.
In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1197–1203. IEEE (1999)
4. Dalstein, B., Ronfard, R., Van De Panne, M.: Vector graphics com- plexes. ACM Trans. Graph. (TOG)33(4), 133 (2014)
5. Jain, A.K., Dubes, R.C.: Algorithms for clustering data Prentice- Hall, Inc., USA (1988)
6. Fišer, J., Asente, P., Schiller, S., S`ykora, D.: Advanced drawing beautification with shipshape. Comput. Graph.56, 46–58 (2016) 7. Fourey, S., Tschumperlé, D., Revoy, D.: A fast and efficient
semi-guided algorithm for flat coloring line-arts. In: International Symposium on Vision, Modeling and Visualization (2018) 8. Fu, H., Zhou, S., Liu, L., Mitra, N.J.: Animated construction of
line drawings. In: ACM Transactions on Graphics (TOG), vol. 30, p. 133. ACM (2011)
9. Gangnet, M., Van Thong, J.M., Fekete, J.D.: Automatic gap closing for freehand drawing. In: ACM SIGGRAPH, vol. 94 (1994) 10. Hati, Y., Jouet, G., Rousseaux, F., Duhart, C.: Paintstorch: a
user-guided anime line art colorization tool with double genera- tor conditional adversarial network. In: European Conference on Visual Media Production, pp. 1–10 (2019)
11. Huang, H., Yin, K., Gong, M., Lischinski, D., Cohen-Or, D., Ascher, U.M., Chen, B.: “mind the gap”: tele-registration for structure-driven image completion. ACM Trans. Graph. 32(6), 174–1 (2013)
12. Kruskal, J.B.: On the shortest spanning subtree of a graph and the traveling salesman problem. Proc. Am.Math. Soc.7(1), 48–50 (1956)
13. Liu, S., Wang, X., Wu, Z., Seah, H.S.: Shape correspondence based on Kendall shape space and rag for 2d animation. Vis. Comput.
36(10), 2457–2469 (2020)
14. Liu, S., Zhang, X.: Automatic grayscale image colorization using histogram regression. Pattern Recogn. Lett. 33(13), 1673–1681 (2012)
15. Madeira, J.S., Stork, A., Groß, M.H.: An approach to computer- supported cartooning. Vis. Comput.12(1), 1–17 (1996)
16. Mahdiraji, A.R.: Clustering data stream: a survey of algorithms.
Int. J. Knowl. Based Intell. Eng. Syst.13(2), 39–44 (2009) 17. Parakkat, A.D., Madipally, P., Gowtham, H.H., Cani, M.P.: Inter-
active flat coloring of minimalist neat sketches. In: Eurographics 2020 Short Paper Proceedings (2020)
18. Parakkat, A.D., Methirumangalath, S., Muthuganapathy, R.: Peel- ing the longest: a simple generalized curve reconstruction algo- rithm. Comput. Graph.74, 191–201 (2018)
19. Sander, J., Ester, M., Kriegel, H.P., Xu, X.: Density-based cluster- ing in spatial databases: The algorithm gdbscan and its applications.
Data Min. Knowl. Disc.2(2), 169–194 (1998)
20. Sandhya, B., Agarwal, A., Rao, C.R., Wankar, R.: Automatic gap identification towards efficient contour line reconstruction in topo-
graphic maps. In: 2009 Third Asia International Conference on Modelling and Simulation, pp. 309–314. IEEE (2009)
21. Sasaki, K., Iizuka, S., Simo-Serra, E., Ishikawa, H.: Learning to restore deteriorated line drawing. Vis. Comput.34(6), 1077–1085 (2018)
22. Seah, H.S., Chua, B.C.: A skeletal line joining algorithm. Insight Through Comput. Graph. Proc. Comput. Graph. Int.1994(Cg194), 62 (1997)
23. S`ykora, D., Dingliana, J., Collins, S.: Lazybrush: Flexible paint- ing tool for hand-drawn cartoons. In: Computer Graphics Forum, vol. 28, pp. 599–608. Wiley Online Library (2009)
24. Wang, S., Zhang, Q., Wang, S., Jing, X., Gao, M.: Endpoint fusing method of online freehand-sketched polyhedrons. Vis. Comput.
36(2), 291–303 (2020)
25. Zhang, L., Li, C., Wong, T.T., Ji, Y., Liu, C.: Two-stage sketch colorization. ACM Trans. Graph. (TOG)37(6), 1–14 (2018)
Publisher’s Note Springer Nature remains neutral with regard to juris- dictional claims in published maps and institutional affiliations.
Jie Jiang obtained his B.Eng from Northwestern Polytechnical University, China, in 2014. He is currently a Ph.D. student at the School of Computer Science and Engineering of Nanyang Techno- logical University, Singapore. His research interests include computer graphics and computer- assisted animation.
Hock Soon Seah is a professor at the School of Computer Sci- ence and Engineering at Nanyang Technological University (NTU) Singapore and the Director of the Centre for Augmented and Vir- tual Reality. His research inter- ests are in geometric data mod- eling, image sequence analysis, non-photorealistic rendering, com- puter animation and game, and virtual and augmented reality. He is the principal investigator of the Computer Assisted Cel Anima- tion (CACANi) research and Fel- low of the Singapore Academy of Engineering. Before joining NTU, he was a programmer analyst at IBM Singapore Pte Ltd.
Hong Ze Liew is the business development manager from CACANi Private Limited. He has produced numerous animation advertisements for various com- panies like Microsoft. He is the user interface designer for the CACANi animation software.
