Agisoft Metashape (both editions: Standard and Professional) supports automatic masking from the background. This article describes the workflow for masks generation from the background. Also, the article includes a comparison of alignment results for the scenarios with and without masking.
The masking from the background option is essential when the camera position is fixed, the background is static, and either the object is rotating, or all images are taken simultaneously by a multi-camera rig. The scenario implies that the background for each camera position does not change during the image capturing process, so the background images (without an object) can automatically generate masks. It is important to have a photo of an empty background for each camera placement. On the illustration below for each photo from the main dataset (with the object of reconstruction) presented on the right-hand side, there is a corresponding image of an "empty" background presented on the left-hand side:
TABLE OF CONTENTS
Add Photos
Use Add Photos/Add Folder command from Workflow menu to add the images to the active chunk of the project. You may choose to add all images to the project (including the images of "empty" background) or only the images with the object of interest.
Adding the background photos themselves to the project (using Add Photos command) is not required unless you plan to use them for Apply Masks > To Tie Points.
Import Mask
In the Photos pane, select the images and access the context menu using the right mouse button. Select Import masks command, and in the corresponding dialog box, choose From Background option and set necessary parameters as illustrated on the image below. Press OK button: the mask will be applied automatically.
To set the masks to file names a template can be used in the Filename template section of the Import Masks dialog box. The template may contain special tokens that are substituted by corresponding data for each photo being processed. Alternatively, a particular filename can be specified to be implemented as a background image for all images from the selected subset.
The following tokens are supported:
{filename} - the name of the source photo without extension
{fileext} - extension of the source photo
{camera} - camera label
{frame} - frame number
{filenum} - sequential number of the source photo
If the tokens are used, the corresponding file will be searched for each photo in the project.
Example 1: All images are taken simultaneously by multiple cameras (just as in the example project used in this article for illustration). For each camera, there is only one image per shooting, and the name of each image file corresponds to the camera name/number in the rig. This means that there are two datasets where the file names are identical, while one dataset contains images of empty background and the second set of images have the object of interest on the photos. In such a scenario, the Filename template - {filename}.JPG is to be used.
Example 2: In the scenario with a turntable, usually there is one photo of the background, as the images are taken from the same camera position. In such a scenario, the filename template should specify the particular file name of the background image (for example - IMG_0001.JPG) - it will be used as a background image for automatic masks generation for all photos in the selected subset.
The parameter Tolerance is only considered for the "From Background" method. It specifies the tolerance threshold used for background differencing. The tolerance value should be set according to the color separation between foreground and background pixels. For larger separation higher tolerance values can be used.
If the option All cameras is selected in the Apply to a section of the Import Masks dialog box, then Select Folder dialog will appear (once the OK button is pressed). In Select Folder dialog the path to the folder where the background images are saved should be set:
To generate masks from the background for Selected cameras (it can be even a single photo) a path to background images themselves should be set after OK command in the Import Masks dialog:
After the masks are generated from the background photo and are imported into the project, the result can be observed on the Photos pane (select View > Photos), where the Show Masks should be enabled:
Align photos (with masks)
Select Align photos... command from the Workflow menu. In the Align Photos dialog window, set Apply mask to the parameter by selecting one of the options from the dropdown menu:
Apply masks to Key points is recommended if all images in the dataset have masks applied (just as in our example dataset used for this article). The areas under masks are excluded from the feature detection procedure for each photo independently. The same option is applicable for scenarios with the turntable as well.
Apply masks to Tie points option should be used if for some reason masks were not generated from the background photo and the user chooses to cover the whole background image by mask. This scenario can be used if the camera placement had been changed accidentally (in turntable scenario) after all photos of the object were taken; another possible scenario: the photo of the background color-balance differs from the balance during the photo shooting with the object in place. The example is presented in the images below: as you may see the background photo on the left has a different color balance in comparison to the image with the object (on the right):
In such a case, the automatic masks generation from the background would not work properly, so it is recommended to import the background images into the project, create masks that will cover the whole image of the background, and not disable the background photos during the alignment procedure. On the image below the whole background, a photo is covered by the mask on the Photos pane Show Masks option is enabled:
When Apply masks to Tie points option is used on the alignment step - only the tie points corresponding to the areas that are not presented on the background photos (and thus not covered by the mask) will remain in the project.
Align photos (without masks)
Select Align photos... command from the Workflow menu. Set Apply mask to None option in the Align Photos dialog window.
In this case, a mask will not be considered on the alignment step, so some parts of the background visible on the photos will be reconstructed as well as the object of interest, see the illustration below:
Compare
Using a mask from the background can reduce or eliminate the amount of noise during the alignment step and provide better results for the model reconstruction. Below the tie point cloud generated with and without masks is represented. The amount of noise on the left image (masks were applied during alignment procedure) is significantly lower than on the right one (masks were not applied):
