Image Sizing using Removal or Insertion of Seams based on Energy Function

The nature of the image plays a significant role in any image sizing algorithm. The technique of improving or condensing an image while preserving its key image quality is challenging. The resolutions of various display devices vary from one another in order to fit the image according to the resolution image sizing plays a vital role. In order to size the image a operator is introduced which will remove or insert the seam from an images based on the resolution of the device used. A seam in an image is related to the pixel path from up to down or right to left, the seam vector to be deleted for image size is defined by the image energy attribute. The Seam removal or insertion operator can be applied to an image and aspect ratio can be adjusted accordingly by continuously removing or adding a seam in the desired region of image. The collection and order of seams decides the picture quality, as described by the energy function. By storing the order of seams in an image that can continuously change to fit a given size in real time, multi-sized images can be produced.


Introduction
There are enormous developments in technology of camera, online interface and numerous monitor interface systems to suit the needs of customers. And, from device to device, the aspect ratio also varies. Various image sizing methods have been proposed to solve the incompatibility between the target system size and image dimension, including cropping the image sides, scaling up or scaling down the image and detecting the edge. However, because such methods have the drawback of discarding considerable image content, a great deal of image sizing research is carried out. For content-based image scaling, a technique called seam removal / insertion is applied. Image sizing is the technique by which a bottom-up or left to right line of connected pixels can be applied to the image size, and the image size can also be minimised by eliminating a bottom-up or left to right line of IOP Publishing doi:10.1088/1757-899X/1022/1/012107 2 pixels connected. The bottom-up or left to right seam would have its own cost value on the basis of the content of the scene. There is more substantial material in the seam with greater cost. Therefore, the desired aspect ratio can be achieved using these seams by removing or adding seams in a recursive way with the least cost. The addition or elimination of the bottom-up or left to right seam will be carried out on the basis of the measured cost value, which preserves the necessary image quality.
Consider an image of M pixels tall and N pixels wide then the top to bottom seam will have M set of seams with its own cost value. Similarly there will be left to right N set of seams with its own cost value. Therefore, by eliminating a related collection of low-cost line from right to left, the height of a picture can be reduced and width can also be reduced by eliminating a low-cost related line from top to bottom.
The operator constantly measures the low cost seams from top to bottom or from left to right and extracts these seams. The procedure will proceed until the necessary ratio is reached. Various image sizing techniques like seam cutting, cropping, and uniform scaling are used. Uniform scaling distorts the appearance of artefacts, while cropping eliminates relevant parts of the original image. On the other side, seam carving retains the appropriate content to a large degree while preventing unnecessary geometric distortions. When sizing is performed depending on the content rather than the geometric form, the image sizing algorithm would be efficient. Depending on the addition or deletion of bottom-up or left-to-right seams, scale of the image will be changed in the proposed method. The energy function of the seam that defines the importance of pixels is used to insert / erase the associated thread.
In video digest methods, the described operator can also be introduced. The filming will take place for the entire day by visual monitoring footage. And the entire 24 hours of video will be stored in memory even when at a given time there may be no movement or less movement may be observed in any frame. By eliminating the inactive frames and by reducing the frame size based on the energy function of the frame, the video can be digested or reduced.

Related Work
Suh et al. [1] proposed an approach in which the map of significance is computed and a rectangular cropping window is searched using the greedy approach, covering a maximum percentage of salient points from the image. A rectangular cropping window is mounted at the centre of the image in this technique, while the cropping window leaves the central part of the image intact and discards the rest of the image. Important artefacts can be cut using this method and much of the data can be lost.
Zhang et al. [2] suggested a method to optimise the issue of auto-cropping by specifying the energy function. Three terms comprise the specified energy function: composition, conservative, and penalty.
Shi et al. also indicated an operator. [3] where smoothing and noise reduction of image is accomplished using morphological tip interpolation.
Jiang et al. [4] implemented the edge-based scaling technique, where the image is separated into various types of blocks with edges in different directions.
Setlur et al. [5] suggested an automated, non-photorealistic method to target large image to smaller displays size. Contents are divided into foreground and background and the material of interest is then searched for further analysis.
Efros et al. [6] discovered seams minimising the fault layer found by two identical layer patches. This way, to shape a much larger texture image, the initial small texture image is quilted. This was later extended by Kwatra et al. [7] to discuss the reconstruction of both image and object textures.
In order to maintain normal structures such as straight line, circle, ellipse, para-curve present in the photos, Noh et al. [8] proposed energy function, with the difference of forwarding gradient. The pixels that become adjacent are checked for curvature irregularity using energy function after seam removal, including the variance of gradient orientation and pixel magnitude.

Seam Based Image Sizing
The image size and resolution can be varied as per the user requirement in the proposed system. Consider an AXB resolution RBG input picture, where A and M are total rows and columns in an input image. The AXB RGB image is transformed to a grey scale prior to determining the energy function of the given RGB input. Without impacting the content, the proposed method aims to remove the low value seams from bottom-up or left to right. The proposed system is shown in figure2.

Edge Detection
The gradient function is obtained using Sobel edge detection technique from the grey scale of input RGB data. Which in turn detects the sharp intensity difference within the different contents of image and edges of contents are obtained. The Edge detection algorithm uses four 3x3 window operators that calculate the intensity variance of the image when it is transformed into four horizontal, vertical, right diagonal and left diagonal directions. The vertical and horizontal gradients are appropriate in this situation, so only two horizontal and vertical operators are used.
A 3x3 mask is defined to calculate the gradient value from bottom-up and left to right of the given image. The bottom-up and vertical directions of the image are convolved. Let IX and IY are gradient values measured respectively from bottom-up and left to right of the input field. The total value of the gradient is given by equation (3).  Figure 3 shows the edge detected grey scale image for the given input picture. The gradient value is obtained by applying the equation (3) for the grey scale image.

Cost Function
Using the gradient value obtained, the energy function of the input source is computed as shown in figure 4.

Figure 4: Energy Function Calculated Image
For both the top to bottom and the right to left direction of the image, the energy feature should be independently identified. After the cost function is measured, a lower energy seam is observed and extracted from the source image for a seam deletion. The cost function is recalculated again after each removal and the procedure is replicated until the desired value is met. Consider the gradient value of a pixel in Figure 5 as (a, b), the sum of the present gradient value at (a, b) and the lowest value of the previous row that is minimum ((a-1,b-1),(a-1,b),(a-1,b+1)) provides the energy function and the procedure is repeated in the top-down and right-to-left direction to measure the total cost function.

Figure 5. Pixels Matrix
The example of cost function calculation is shown in the figure 6. Where the values given in left top of each box represent the gradient data calculated. And the energy function is expressed by the value in the center of each box. In the top row, the energy property of that row is the gradient value itself. The lower gradient value of the closest neighbor in the previous row is found and then added to the gradient value of the chosen pixel to determine the energy function of the second row. And this process is repeated until the energy function is obtained till the last row.

Seam Insertion or Deletion
The non-linear energy function is connected from bottom-up or right to left, defined as seam. A seam can be inserted from bottom-up or from right to left in the image enhancement process. Similarly, the low energy seam can be separated from bottom-up or right to left to reduce picture dimension. The image size would be decreased by one if one seam is eliminated in either direction. Dijkstra 's algorithm, dynamic programming, is used to take the lowest energy seam from one end to the other, which finds the closest pathway from bottom to up or right to left direction with low energy.
By considering the lowest value in the end row, the low energy seam is obtained and the position of the lowest energy value is saved. Then cross over to the previous row and compare the energy value of the closest three neighbours and save the lowest energy value location. Cross back to the previous row again and follow the same procedure until we enter the first row of the picture. The direction of the lowest energy value is stored at the end of the process and it can be removed from the input source. This procedure is repeated until the required interface description ratio is achieved. Figure 7 describes the method of locating the lowest energy seam. Find the last row of Figure 5, which saves the lowest energy with a value of 3; the energy value 3 in the last row has two closest neighbours in the preceding row with an energy value of 2 and 3 respectively. From which operator will pick and save the lowest value 2. Energy worth 2 again has two neighbours in the previous row , which has energy value of 3 and 1.The lowest energy neighbour with value 1 will be selected and saved. To resize the image, the full seam from bottom to up is measured and can be removed from the frame. And this process will be continued until the desired size is reached.

Video Digest
Video monitoring normally monitors for an whole day and stores the entire recorded video in memory, i.e. 24 hours of captured footage. Browsing or tracking the 24-hour video is a repetitive activity, to browse or condense the video efficiently the removal of seams using energy function can is implemented. As shown in figure 9, the total length of video is minimized in video digest by maintaining the high cost frame and removing the cost frame that is less than specific criteria. The removal operator for the bottom up or left to right seam is added to the whole frame of the film. This seam removal process is carried out until the desired requirements are met. If a given frame is of less activity, the elimination of the seam is based on the cost function, then the energy function of that frame will be less, in that case more seams will be removed and hence the total video length is decreased. All the video frames are added to the operator, resulting in a shortened video length.

Aspect Ratio Matching
The aspect ratio varies from device to device, so the image must be vertically and horizontally resized on the basis of the aspect ratio of the device. Picture resizing should be so that the image content should not be changed. Without modifying the content present, the operator introduced in this paper adds or removes the seams depending on the energy function of the picture. Overall process of aspect ratio matching is shown in figure 10.  Figure 10, if the ratio is equal, no action is taken to compare the ratio of the input image and the input unit. And the seam removal or insertion operator can be applied if there is a mismatch in the ration. If the aspect ratio is lower, either seam insertion should take place or seam deletion should take place if the ratio is greater. And once the ratio is matched, this process will be stopped.

Results and Discussion
Input RGB images with a resolution of 600x315 pixels were produced using traffic surveillance images. Using vertical seam removal, the original images were resized from 10% to 50%, that is, the image was resized to 500x235 pixels. Similarly, vertical seam insertion expanded the initial images from 10% to 50%. Parameters Input Image Compressed  Image  Format  JPG  JPG  Width  600  500  Height  315  235  Size  32409  27573 From Table 1 it is evident that the input image size is reduced without affecting the quality of the image using seam removal or insertion technique. The difference between the normalised histogram of the original and the condensed image is found to be very small. Both images look similar visually, which means that the visual content is not lost.

Conclusion
The content based image sizing by utilizing the energy function for seam insertion or deletion is implemented. Best seams from bottom-up or left to right seams were identified for inserting or deleting a row or a column vector without compromising the content of the image. With the recommended operator, different applications such as content improvement, content reduction, aspect ratio variance, video digest can be accomplished. Depending upon the aspect ratio of the device the image size can be varied in order match the aspect ratio. The operator presented here is also implemented in video digest for efficient browsing of surveillance video. The content based seam removal or insertion using the energy function computes the low cost or non-activity based seams. Deleting such low energy seams will not affect the content of an image.