Thursday, August 08, 2013

Downsampling images to speed up GrabCut

In the previous post [1] I've tried the GrabCut function of OpenCV, when I noticed it was time consuming but didn't try to check the exact processing time. But I was still wondering ``how slow'' did GrabCut could be. So, I started to add the clock() function to see the result [2].

Later, I though of that the processing time was greatly affected by the image size, so I searched for functions which could reduce the images for speeding up GrabCut. What I found were cv::pyrDown() and cv::pyrUp() and they've been implemented in my test code (listed below).

#include "opencv2/opencv.hpp"
#include <iostream>
#include <time.h>

using namespace std;

const bool DOWN_SAMPLED = true;
const unsigned int BORDER = 1;
const unsigned int BORDER2 = BORDER + BORDER;

int main( )
{
    clock_t tStart_all = clock();
    // Open another image
    cv::Mat image;
    image = cv::imread("sunflower02.jpg");

    if(! image.data ) // Check for invalid input
    {
        cout <<  "Could not open or find the image" << std::endl ;
        return -1;
    }

    cv::Mat result; // segmentation result (4 possible values)
    cv::Mat bgModel,fgModel; // the models (internally used)

    if(DOWN_SAMPLED){
        // downsample the image
        cv::Mat downsampled;
        cv::pyrDown(image, downsampled, cv::Size(image.cols/2, image.rows/2));

        cv::Rect rectangle(BORDER,BORDER,downsampled.cols-BORDER2,downsampled.rows-BORDER2);

        clock_t tStart = clock();
        // GrabCut segmentation
        cv::grabCut(downsampled,    // input image
            result,   // segmentation result
            rectangle,// rectangle containing foreground
            bgModel,fgModel, // models
            1,        // number of iterations
            cv::GC_INIT_WITH_RECT); // use rectangle
        printf("Time taken by GrabCut with downsampled image: %f s\n", (clock() - tStart)/(double)CLOCKS_PER_SEC);

        // Get the pixels marked as likely foreground
        cv::compare(result,cv::GC_PR_FGD,result,cv::CMP_EQ);
        // upsample the resulting mask
        cv::Mat resultUp;
        cv::pyrUp(result, resultUp, cv::Size(result.cols*2, result.rows*2));
        // Generate output image
        cv::Mat foreground(image.size(),CV_8UC3,cv::Scalar(255,255,255));
        image.copyTo(foreground,resultUp); // bg pixels not copied

        // display original image
        cv::namedWindow("Image");
        cv::imshow("Image",image);

        // display downsampled image
        cv::rectangle(downsampled, rectangle, cv::Scalar(255,255,255),1);
        cv::namedWindow("Downsampled Image");
        cv::imshow("Downsampled Image",downsampled);

        // display downsampled mask
        cv::namedWindow("Downsampled Mask");
        cv::imshow("Downsampled Mask",result);

        // display final mask
        cv::namedWindow("Final Mask");
        cv::imshow("Final Mask",resultUp);


        // display result
        cv::namedWindow("Segmented Image");
        cv::imshow("Segmented Image",foreground);
    }
    else {
        cv::Rect rectangle(BORDER,BORDER,image.cols-BORDER2,image.rows-BORDER2);

        clock_t tStart = clock();
        // GrabCut segmentation
        cv::grabCut(image,    // input image
            result,   // segmentation result
            rectangle,// rectangle containing foreground
            bgModel,fgModel, // models
            1,        // number of iterations
            cv::GC_INIT_WITH_RECT); // use rectangle
        printf("Time taken by GrabCut with original image: %f s\n", (clock() - tStart)/(double)CLOCKS_PER_SEC);

        // Get the pixels marked as likely foreground
        cv::compare(result,cv::GC_PR_FGD,result,cv::CMP_EQ);
        // Generate output image
        cv::Mat foreground(image.size(),CV_8UC3,cv::Scalar(255,255,255));
        image.copyTo(foreground,result); // bg pixels not copied

        // display original image
        cv::rectangle(image, rectangle, cv::Scalar(255,255,255),1);
        cv::namedWindow("Image");
        cv::imshow("Image",image);

        // display result
        cv::namedWindow("Segmented Image");
        cv::imshow("Segmented Image",foreground);
    }

    printf("Total processing time: %f s\n", (clock() - tStart_all)/(double)CLOCKS_PER_SEC);

    cv::waitKey();
    return 0;
}


The key idea was to downsample the image for GrabCut and then upsample the result (I thought it was a mask) to the original size. The result showed a remarkable speeding up in both the debug and the release mode.

Here are the output images with the downsampling strategy:

Fig 1. Original image
Fig 2. Downsampled image

Fig 3. Mask obtained by using GrabCut

Fig 4. Upsampled mask

Fig 5. Final result
Here is the result without the downsampling strategy:
Fig 6. GrabCut result without downsampling

Comparing Figure 5 and 6, we can easily notice the differences between the segmented results. When applying the downsampling strategy, some image details were lost and the mask would be different and had rougher edges as well.

Although the downsampling strategy has the drawback of losing image details, the benefit of reducing processing time was significant. The following table lists the processing time obtained by using above code with and without the downsampling strategy.


processing time (sec.)without downsamplingwith downsampling
debug modeGrabCut3.0780.717
Total3.1010.756
release modeGrabCut0.5990.123
Total0.6190.157

---
[1] Try GrabCut using OpenCV
[2] How to use clock() in C++

Friday, August 02, 2013

Building OpenCV libs and dlls using CMake in Windows 7

(This is a simple note of my building process.)

As mentioned in the previous post, in which  I tried the GrabCut by using the OpenCV's library. Because I didn't have libs and dlls for debug mode, so I tried to use CMake to build them for my own usage.

First I went to OpenCV website to download the latest stable version 2.4.6. The source code for Windows were packed in a exe file. Don't worry about it, just download it and click it and the 7zip will extract the whole source package for you. In my case, the extracted folder was named ``opencv''.

Then I lauched CMake GUI, chose the location where the extracted folder was located, and chose the build directory for the building files.



Click the ``Configure'' button and if everything is okay then the ``Generate'' button. In my case, I'd chosen the generator as ``Visual Studio 2005'' (at a certain step I didn't rememberd0, so the generating result contained an OpenCV.sln in the build folder.

The final step was just click the OpenCV.sln to launch the Visual Studio and then Build the project for Debug and Release mode. The products were located in the build/bin and build/lib directories.

Try GrabCut using OpenCV

I was considering using GrabCut to cut out the target in one of my working project. After testing it using Python, I thought it's necessary to try it in C++ code. Therefore I started to find some example code and picked one for my test [1].

Here is my test code, the sample photo, and the result:

#include "opencv2/opencv.hpp"
#include <iostream>

using namespace cv;
using namespace std;

int main( )
{
    // Open another image
    Mat image;
    image = cv::imread("sunflower02.jpg");

    if(! image.data ) // Check for invalid input
    {
        cout <<  "Could not open or find the image" << std::endl ;
        return -1;
    }

    // define bounding rectangle
    int border = 20;
    int border2 = border + border;
    cv::Rect rectangle(border,border,image.cols-border2,image.rows-border2);

    cv::Mat result; // segmentation result (4 possible values)
    cv::Mat bgModel,fgModel; // the models (internally used)

    // GrabCut segmentation
    cv::grabCut(image,    // input image
        result,   // segmentation result
        rectangle,// rectangle containing foreground 
        bgModel,fgModel, // models
        1,        // number of iterations
        cv::GC_INIT_WITH_RECT); // use rectangle
    // Get the pixels marked as likely foreground
    cv::compare(result,cv::GC_PR_FGD,result,cv::CMP_EQ);
    // Generate output image
    cv::Mat foreground(image.size(),CV_8UC3,cv::Scalar(255,255,255));
    image.copyTo(foreground,result); // bg pixels not copied

    // draw rectangle on original image
    cv::rectangle(image, rectangle, cv::Scalar(255,255,255),1);
    cv::namedWindow("Image");
    cv::imshow("Image",image);

    // display result
    cv::namedWindow("Segmented Image");
    cv::imshow("Segmented Image",foreground);


    waitKey();
    return 0;

}

The sample photo used in the test



The result of applying GrabCut

During the test, I encountered an old problem to me, which had been some odd runtime bugs for the debug mode when using OpenCV. The solution might be NOT to mix up the debug and release libraries [2].

Oh, by the way, the processing time of the GrabCut was too long (about 2 seconds in the test case), and I thought it's not feasible for realtime applications. Orz

---
[1] http://stackoverflow.com/questions/15536222/opencv-grabcut-algorithm-example-not-working
[2] http://stackoverflow.com/a/2590795/1024813

Thursday, August 01, 2013

cv2.waitKey bug

I was trying a Grabcut sample code (don't use this version) written in Python by abid rahman and stuck at a strange problem: the cv2.waitKey() didn't work normally!

Yesterday I searched the problem on Google and found nothing useful (according to my skill level, I might overlooked something that could be hints), so I decided to ask in the G+ Python community [1].

Based on Brett Ponsler's helpful suggestion, I tried to rewrite my test code and also went to download the Grabcut sample code again (this version is fine). This time, the sample code was running successfully.

Then I noticed a magic word ``0xFF'' in the new downloaded sample code. Using the hint, I finally found the bug report about cv2.waitKey() and came up with a tiny test code:

import cv2
import numpy as np

cv2.namedWindow('test')

while True:
    #key = cv2.waitKey(33) #this won't work
    #key = 0xFF & cv2.waitKey(33) #this is ok
    key = np.int16(cv2.waitKey(33)) #this is ok [2]

    if key == 27:
        break
    else:
        print key, hex(key), key % 256

cv2.destroyAllWindows()
---
[1] The question I posted in the Python community: https://plus.google.com/117911423781149907976/posts/CdvQ3vWVV19
[2] http://jiaxihu.blogspot.tw/2013/02/opencv-python-64bit-waitkey.html