This article describes the suggested workflow for automatic objects detection on orthomosaic. The workflow includes the use of a pre-trained neural network that allows for object detection automation. Even though the network had been trained for tree detection tasks, it can be adjusted for detection of other objects (for example cars, sea lions, etc.) by the use of partially manual annotations introducing new object types. It is also suggested that you annotate a small region of the orthomosaic to guide the neural network and optimize its performance for each particular task.


To enable the neural network you need to download and install a python script for Metashape Professional edition that is published in the repository detect_objects.py based on github.com/weecology/DeepForest. 


You can find information about how to download a script from Agisoft GitHub page in our article: How to download script from GitHub?


You need NVIDIA GPU for fast processing (i.e. CUDA-compatible GPU required). CPU is also supported, but the CPU-based implementation is very slow compared to GPU implementation.


Contents of this tutorial:


Starting from version 2.0 and using the latest version of the script, you don't need to install additional libraries, they will be installed automatically during the script run. The latest version of the script.



Please note that script requires CUDA, so the script may not work on macOS.



How to install the script on Linux

1.  Install two python packages: pytorch with CUDA support and deepforest:

cd .../metashape-pro
LD_LIBRARY_PATH=`pwd`/python/lib/ python/bin/python3.8 -m pip install albumentations==1.0.3 deepforest pytorch-lightning==1.5.10 torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html 


2. Follow the steps described in this article to add the script to auto-lauch. In other words, you need to copy detect_objects.py script to /home/<username>/.local/share/Agisoft/Metashape Pro/scripts/



How to install the script on Windows

Starting from version 2.0 and using the latest version of the script, you don't need to install additional libraries, they will be installed automatically during the script run. The latest version of the script.


1. Download GDAL, Rasterio and Fiona packages (for Python 3.8 amd64, i.e. download files ..cp38‑cp38‑win_amd64.whl)


2. Launch cmd.exe with administrator privileges. Change working directory to Downloads (where the packages have been downloaded to) by executing the following command:

cd %USERPROFILE%/Downloads


3. Install the packages one by one. You can download the latest version of each package, the installation commands examples below show the version numbers corresponding to the latest version (ending in "cp38‑cp38‑win_amd64.whl") available by the time of tutorial publication. Therefore the version numbers may differ when the newer versions are available:

"%programfiles%\Agisoft\Metashape Pro\python\python.exe" -m pip install GDAL-3.4.2-cp38-cp38-win_amd64.whl
"%programfiles%\Agisoft\Metashape Pro\python\python.exe" -m pip install rasterio-1.2.10-cp38-cp38-win_amd64.whl
"%programfiles%\Agisoft\Metashape Pro\python\python.exe" -m pip install Fiona-1.8.21-cp38-cp38-win_amd64.whl

4. Install two python packages: pytorch with CUDA support and deepforest:

"%programfiles%\Agisoft\Metashape Pro\python\python.exe" -m pip install albumentations==1.0.3 deepforest pytorch-lightning==1.5.10 torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html


Possible installation errors

  • You may observe warning messages like on the screenshots below, you can ignore them:


  • You may observe the following message:
error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/


In this case, you need to install Visual studio from the page - https://visualstudio.microsoft.com/downloads/. On this page select Community > Free download.

In the Installing dialog window enable Desktop development with C++ and MSVC v142, v141, and v140 parameters. Click Install button:


After installing Visual studio reboot the computer and repeat install packages.



if you encounter another error during installation, please follow these steps: 
Click the right mouse button on the top panel of the CMD window and choose Select All command. After that click the right mouse button again on the top panel on the CMD window and select Copy command. Create *.txt file and paste copied text from the command line. And please send this file Agisoft support team: support@agisoft.com.


5. Add the script to auto-launch directory, following the instructions given in the related article. In other words you need to copy detect_objects.py script to C:/Users/<username>/AppData/Local/Agisoft/Metashape Pro/scripts/


How to use the script for automatic detection of objects

In the example dataset that we use for illustration in this article, we train the neural network to detect cars on the orthomosaic. Since the automatic detection is performed on the orthomosaic, you need to build an orthomosaic for your project before running the script. You may find instructions on how to build an orthomosaic in the following articles: 


The resolution of orthomosaic should be at least 10 cm/pix (i.e. GSD should be no more than 10 cm/pix).


1. Open the dataset in Agisoft Metashape Professional edition. 



After the Python packages are installed, a dialog box may appear when you start Metashape with the message: "Please wait while Windows configures Agisoft Metashape Professional". Please wait until the check is over and Metashape opens, then continue working with the script:

Before starting automatic object detection, it is necessary to "train" the algorithm to recognize the objects we're interested in detecting in a particular case. To do this, we allocate a training area on the orthomosaic that includes the objects to be detected (in our example project the objects are cars). 



2. Choose a small area on the orthomosaic where the objects of interest are presented and create a shape that will indicate the training zone for the neural network. The shape should be created using Draw Rectangle tool (thus the area will be aligned with the axis). The rectangle shape should be at least 50 meters on each side. On the illustration below the shape layer, that contains the created shape is called Train zones.




It is recommended to set color for shapes of Train zones layer to avoid mistakes when working with the dataset. To set color select the shapes layer on the Workspace pane and choose Properties command from the context menu (available on right mouse button click ). Choose the preferable color from the palette and press OK.



3. Create another shape layer:  on the Workspace pane by selecting Add Shape Layer command from Shapes context menu: 



In the New Shape Layer dialog window set Label: Train data and choose Color for the layer : 




we're using the shape layers names Train zones and Train data as these names will be automatically selected in the corresponding boxes of the dialog window Objects detection on orthomosaic (see further below in this article). It is possible to use any name for the shapes layer if preferable, but in such case you should identify the corresponding layers manually. 



4. It is necessary to select all objects of interest (cars in our case) in the training zone. For this purpose, we use Draw rectangle tool and assign separate shape (created in Train data shape layer) to each car within the boarders of the training zone (limited by the shape in Train zone layer that we have created on the previous steps). 



It is necessary to mark up all the cars within the training zone because if you skip some cars, the neural network will think that such cars should be ignored and this will worsen the quality of the result. Even those cars that fall into the Train zones rectangle partially are also important to select (see the car indicated by the green arrow on the screenshot below):




5. There are two different options to run the script.


The first option is available if the script had been added to the auto-launch (see step 2 for How to install the script on Linux or step 5 in the How to install the script on Windows). In this case simply run the script from the main menu: Scrips > Detect Objects.


Another option is suitable for the case when for some reason the script was not added to the auto-launch. Use Tools > Run script... command and in the Run Python Script dialog window set path to the file with .py extension (i.e. to the python script): 



Then in the main menu select: Custom menu>Detect objects command. 


After the script is launched the dialog window Objects detection on orthomosaic will appear. Set the path to the Working directory (where the results are to be saved) and make sure that the Shape Layers for Train zones and Train data are specified correctly. Click Run button to execute the script.


 

Check the results of the automatic objects detection in the Ortho view:


The higher the variability of objects in the training zone, the better the results of automatic detection will be. Some areas are problematic for the automatic detection algorithm, therefore some objects would not be detected. For example (as should on the image below) dark-colored car was not detected in the shadowed area, as the difference between pixels on the object and on the background is very small. 



Another example is illustrated in the image below: some objects may resemble the object of interest, so the algorithm detects such objects by mistake. 



Automatic detection of objects is useful for the detection of particular objects in big areas, however, the implementation of the script still requires that the results are verified and corrected by the operator.