Alp’s Large Image ANnotation Tool (LIANT) for Semi-Automated Labeling and Inference

(Text Updated)

Hello,

Now this one is not quite a general-purpose toolset, instead, tailored for a pretty specific goal. But I decided to add it to website too, someone may find a use for it.

Problem: For both training and inference, you are limited to the video memory on GPU board, at the time of writing, at least. If you want to apply inference to larger images (say, 5000×10000, 10000×30000, etc…), you quickly realize that its not possible even with 12GB Titans, and you have to split your main, large image to several pieces and apply inference to them, individually.

This is a quite time-consuming process, especially if you have more then a few  large-scale image.

Unfortunately, solving all the necessary steps in one go is beyond me. But I have prepared a plugin to make it happen in a reasonable way, with acceptable results.

Please keep in mind, this toolset is designed to apply inference to vertical (portrait) large images. And works for single class detection.

Installation

Download LIANT plugin: Alps_Large_Image_Annotation.ijm_22Apr2017_Ubuntu.tar.gz

Download Action Bar plugin (new version, required for LIANT plugin): action_bar-2.0.5-SNAPSHOT.jar

Now this install requires a manual step. If you are using my plugins, you probably already installed the Action Bar plugin. The LIANT plugin requires a manual update to Action Bar plugin. If you did not install the Action Bar yet, or if you are installing for the first time, follow the relevant steps in Alp’s Labeling Tool for installation of the Fiji software and ActionBar plugin.

Here are the required steps for updating the Action Bar plugin:

  1. Open Fiji.App/plugins  folder.
  2. Find ActionBar2.02.jar file, move it out or just delete it.
  3. Run Fiji, then close it.
  4. Copy new Action bar plugin into Fiji.App/plugins folder.
  5. Run Fiji.

Thats it.

Then we can continue with normal install.

Download and open the archive. Drag and drop the  “plugins” folder onto your main Fiji folder. Plugins are installed.

The LIANT toolset is now accessible via Plugins main menu of Fiji (Alps Large Image Annotation).

Usage

Before start, all your large images need to be adjusted to same width and height, in pixels; And images should be vertical (portrait).

It consists four buttons.

LIANTmenubar

First one (Resize Canvas) takes your large image (i.e, large001.png), it helps you to resize the image canvas (like padding), to make the height (pixels) a fold of 8, 16 or 32. You have to provide necessary numbers here, into the dialog box. To pad bottom of image, set a new Height, and choose Top-Center. You have to apply ‘Resize Canvas’ to all your images, individually, (one by one) in the current version.

resize_canvas

Next menu [ Split Image(s) ] takes above-padded image, and creates a folder with its name in a provided folder(for example, /allimages/  ) ;

/allimages/

  • /large001.png/
    • 0001_large001.png
    • 0002_large001.png
    • 0003_large001.png
    • 0004_large001.png
    • ……

and according to the parameters you supply at the dialog screen, it splits the large image to several parts and saves these parts into (/large001.png/) folder in a numbered fashion, as you see in our example, above. Crops will be taken from large image from top to bottom, and numbered in this order.

Please note that this menu is capable to process all images in the folder you indicate (batch process). Therefore, be sure that there is no other files of any type exist, in your large images folder. Otherwise, this will cause a failure or trouble.

splitting_parameters

  • Input Directory: Folder of large image(s).
  • Output Directory: Main Folder to keep splitted images (like /allimages/ in our above example)
  • Large Image’s Width: Width of your large, main image being splitted
  • Large Image’s Height: Height of your large, main image being splitted
  • Crop Width: Width of area you are going to crop from main image (its best to give full width, in the current version)
  • Crop Height: Height of area you are going to crop from main image.
  • Crop Overlap: Amount of overlap of current cropped area to previous cropped area.

Crop Width and Height is particularly important here.

You have to find the largest image dimension that DIGITS can apply inference to, within limitations of your model, and GPU memory.

Basically, largest image dimensions you could be able to test in DIGITS web interface is fine. Thats why I decided to keep crop width same as large image’s width. This way, you can play just with “height” to find largest crop size. Investigate the following for further info on custom sizes:

https://github.com/NVIDIA/DIGITS/issues/980

As an additional step, its a good idea to modify following line of deploy.prototxt in the relevant job folder, to your crop width and height, or to a larger number. Replace 1248 with your cropped-image width, and 352 with your cropped image height. You can use larger numbers then crops.

param_str : ‘1248, 352, 16, 0.6, 3, 0.02, 22’

Of course, its best to manually carry out tests of such sizes at DIGITS web interface first, to be sure that given crop-image size above fits into GPU memory and makes detections without problems.

Also, the crop height you choose, must divide original image height without residues. (ie, 11968/704=17. So 17 cropped images will be created, each will be at 5120×704 dimension. If you are going to apply crop overlap, this number will increase.

For 12GB GPU RAM, the crop size you adjusted must stay around 4 to 4.5 megapixels, maximum (above example; 5120×704 = 3.6MP, which is fine). If your GPU RAM is lower, then you have to carry out some tests in DIGITS web, to find largest image size for inference.

The height must preferably be at least two folds larger then your largest object. If your object is 100 pixels long, It would be good to keep crop height at 200 pixels, minimum. This helps to prevent frequent splits of your objects across cropped images.

Crop Overlap is another critical parameter. If you keep it 0 (zero), the next crop (split) will start right after previous crop. If you enter a different number (32, in example), the next crop will shift 32 pixels into bottom area of previous crop. So all the following crops will shift upwards, 32 pixels. Please see the figure below, for a rough representation of results of overlap.

Alps_LIANT_split

The reason of applying an overlap

When you split the large image without overlap, your objects may be splitted, also. Here, were applying the overlap to fully cover the splitted object, in the next crop. Therefore, your overlap distance should about about slightly higher then height of largest object (in pixels).

 

The next button is [ Inference Many ]. Inference is applied to all splitted images, within main folder (ie,   /largeimages/), so this menu is makes a batch process, like “Split Images” menu.

Please be sure DIGITS server is running now!

Following window opens;

ınfermany_plugin

Input Directory: Choose the main folder that keeps the splitted-image folders (like /allimages/ in our above example)

Output Directory: Preferably the same folder above (Input Dir). Unless you want to save label annotation text files to somewhere else other then where they belong to.

Name of Object Class: Name of objects you want to detect (should be same with whats available in the trained model).

Job Folder: Choose the folder of the trained model to apply inference with.

Epoch Number: Choose the epoch number you want to apply inference with (Preferably choose the epoch with highest mAP value).

You can check the folders of splitted images to see how the process is going. Please patiently wait here, until all splitted images processed. The images and label text files with zero detections will be deleted from folders of splitted files, so only the croppings containing detections will remain in each folder.

 

Repeating here, your deploy.prototxt file must be adjusted to crop size, before applying inference!

 

And the final button is [ REbuild Labels ]. Click to open following;

rebuild

Folder of Splits: Choose the main folder that belongs to one of your large images and contains splits and label text files from inference step above (ie, /largeimages/large001.png/ )

Select the current large-image file: Choose the large image file itself. (for ex, large001.png). It should be same image that you choose the splits above.

Crop Overlap: The crop overlap distance (in pixels) that you applied during split of the file above (it must be the same amount). The overlap distance will be reverted back, to make annotations sit in the right positions, in the large image.

When you click OK, the label files within splitted image folders will be rearranged to coordinates of large image, and an annotation text file (ie, large001.txt) containing all the detections across large image will be builded.

If you visit the main folder (/allimages/), you will see large001.png, large001.txt and large001.zip files in it. If you run the labeling tool, it will load all labeled detections at once, across whole large image.

I hope, one day, the steps above can be done within DIGITS framework. Until that day, enjoy the toolset and good luck !

Downloads

3 thoughts on “Alp’s Large Image ANnotation Tool (LIANT) for Semi-Automated Labeling and Inference

  1. Can we use this tool for automatic annotation ? In other words , is this tool integrated with digits for example, so we can run object detection model to initially annotated our images ? after that , we correct to the model its output and save the correction ?

    Like

    1. Absolutely.
      Please see “life is short…” page. It s already doing the auto annotation. While purpose of this one is slightly different, this does auto annotation too. The only point to consider is the accuracy of the model you are going to use for annotation purposes. Manual checking and correcting the results is always needed, if you need high accuracy.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s