Testing Artificial Intelligence (deep neural networks) for maritime collision avoidance

Inspired by OSCAR Navigation collision avoidance system, I wanted to test how an off-the-shelf Artificial Intelligence (AI), specifically deep neural network based object labeling system, would work on maritime collision avoidance application. AI would be responsible for the labeling of the objects and marking their location in the image. If it can be done with good consistency and with relatively short interval and if the camera is situated in a fixed position and has a gimbal or digital stabilization, it should be possible to estimate the speed and heading of the moving objects found in the image (using the speed and heading data from the own instruments). It should also be possible to determine the distance of the object from own ship.

I decided to use Amazon Rekognition for the tests, because it offers good off-the-shelf tools for labeling and locating objects. Amazon also offers a possibility to train custom labels with a relative small set of training data, but I wanted to see how the product would work without any additional training.

I did the tests using still photos showing a boat in the distance. Due to the fact that it is currently Winter, I could not take the photos myself, but had to use photos found in the internet. I will try to repeat the test with a more systematic collection of photos in the Summer.

Artificial neural networks (ANNs) never tell what is in the image, but rather show the probabilities of different objects appearing in that image. I have provided a sorted list of top objects in this image determined by Amazon under the image.

In addition to the probabilities, Amazon Rekognition also provides a location (a bounding box) for the object found on the image, if it can determine it with high enough probability.

For those interested, I have also provided the response JSON for this image in the end of the post.

Here is another image with a sailboat.

Sailboat is clearly an easy target to find. What about other types of boats.

Boat detected with 96% probability below and also location found.

Swans and the boat detected in the image below.

Vessel with 97% probability below, but no location determined.

But, when cropped just a bit, vessel still found with 97% probability but now also location determined.

In the following, there is also one false positive as the one in the center is not a boat. Also, the other one is a ship, not a boat, but finding a vessel is good anyway.

No boat located in the following image, but ANN determined that the image has a vehicle with 70% probability and a ship with 65% probability. The boat is only 11×7 pixels in this photo.

By lowering the image resolution in small steps, it can be determined that if the boat is visible in the image and its size in one direction is at least 35 pixels (e.g. 35 x 20 pixels), it will be “seen” and typically also its location can be determined. Also, splitting or cropping the image makes at least locating the object work better. The recognition works amazingly well using the generic neural network. It would also be interesting to see how much it could still be improved by custom labeling (using training data from the same camera capturing the images to be evaluated).

Amazon Rekognition response JSON for the first image on this post:

{
    "Labels": [
        {
            "Name": "Vessel",
            "Confidence": 99.7686538696289,
            "Instances": [],
            "Parents": [
                {
                    "Name": "Vehicle"
                },
                {
                    "Name": "Transportation"
                }
            ]
        },
        {
            "Name": "Transportation",
            "Confidence": 99.7686538696289,
            "Instances": [],
            "Parents": []
        },
        {
            "Name": "Watercraft",
            "Confidence": 99.7686538696289,
            "Instances": [],
            "Parents": [
                {
                    "Name": "Vehicle"
                },
                {
                    "Name": "Transportation"
                }
            ]
        },
        {
            "Name": "Vehicle",
            "Confidence": 99.7686538696289,
            "Instances": [],
            "Parents": [
                {
                    "Name": "Transportation"
                }
            ]
        },
        {
            "Name": "Boat",
            "Confidence": 98.71965789794922,
            "Instances": [
                {
                    "BoundingBox": {
                        "Width": 0.05666980892419815,
                        "Height": 0.11389521509408951,
                        "Left": 0.10582049936056137,
                        "Top": 0.6451812386512756
                    },
                    "Confidence": 98.71965789794922
                }
            ],
            "Parents": [
                {
                    "Name": "Vehicle"
                },
                {
                    "Name": "Transportation"
                }
            ]
        },
        {
            "Name": "Sailboat",
            "Confidence": 96.3071517944336,
            "Instances": [],
            "Parents": [
                {
                    "Name": "Boat"
                },
                {
                    "Name": "Vehicle"
                },
                {
                    "Name": "Transportation"
                }
            ]
        }
    ],
    "LabelModelVersion": "2.0"
}