EMODnet Human Activities » News » Vessel density map: and so it begins!

Vessel density map: and so it begins!

featured image

In June we left you with a promise: we’d make a vessel density map of EU waters and we’d share it on our portal for you to view and download. In the meantime, several things have happened. A meeting took place in Brussels in September and the discussion ensued enabled us to better define the requirements that the maps should have to be as useful as possible to the maritime community.

Getting Started

Now, we’ve just purchased a year worth of data from a commercial provider, and so we’re officially getting started. We’re very excited, but also feel sort of jittery; making the map is not going to be a piece of cake. For starters, the amount of data to process is huge. Suffice it to say that, for the area we’re analysing, a typical day contains around 12 million AIS messages. Multiply that by 365 and you’ll get something that goes under the name of… big data!

To make it even more complicated, before actually creating the map, the underlying data need to be pre-processed and cleaned. AIS messages are delivered in NMEA format. However, human beings and NMEA format don’t get along quite well, and so it is highly recommended to convert the data into a format that is easier to work with (e.g. CSV). Then, there’s the challenge of dealing with messages and ship positions that are obviously wrong. For instance, albeit a rare event, it is not impossible to have to deal with an AIS message according to which a ship is sailing across the Alps or in the middle of the Black Forest; errors always happen, and, when you deal with a billion records, even 0.5% of wrong messages quickly become a pain in the neck.

Ship DensityAnd there’s more to it: some messages might report implausible speed or course, or a wrong MMSI (a unique identifier that makes it possible to identify ships). All these messages need to be either corrected or deleted, but doing so on billions of records takes time, patience as well as ingenuity to create algorithms that may speed up the process. Other known issues are duplicated messages and satellite “noise” in areas characterised by high density.

To give you an idea, consider that when our colleagues at HELCOM did a similar exercise they had to use a dedicated server with a 10-core CPU and 48 GB of RAM to process the data.

Calculating Density

As if this was not enough, we also need to define what we mean by ‘density’. Essentially, our map will be a grid with 1 square kilometre cells, each with a colour gradient that gives an idea of vessel density – generally, the darker the colour, the higher the average number of ships in a cell. But how do we calculate density? HELCOM recreates ship routes starting from AIS messages, and then counts the number of track lines that cross each cell. Another method could also take into account lines’ length, as number of track lines alone might be misleading. On the other hand, the JRC suggests counting ship positions (so points, not lines) in grid cells at fixed time intervals. At this point the discussion gets quite technical, and it’s worth a dedicated post. What we’re sure of is that each method produces different results, and so the final choice needs to be pondered carefully.

So, this post is just a brief overview of some of the challenges we’ll be facing over the coming months. Next posts will focus on other aspects that we haven’t discussed here. It’s going to be a long and exciting journey, and we plan to share it on this blog as we make progress.

Image (top):

Related article: Vessel density maps: help us make a difference (June 2017)

The information and views set out in this blog are those of the author and do not necessarily reflect the official opinion of the European Commission. Neither the European Commission nor any person acting on the European Commission's behalf may be held responsible for the use which may be made of the information therein.

December 6th, 2017 | Written by

Filed under: News · Tags:

3 Responses to "Vessel density map: and so it begins!"

  1. Lauren Biermann says:

    Hey guys! We’ve spent the last few months successfully overcoming NMEA hurdles, and would be happy to help. Also acquired a few years of commercial AIS data for slightly different applications. Worth getting in touch!

  2. Mark Spring says:

    This looks like an exciting project and will be really useful to many types of research. Please can you tell me how to interpret the example image above to give the number of vessel positions per month?

    I’m just looking for a comparison with similar data from NeoWins for Japan. In that case, the legend gives the following designations. I would like to see a similar legend, but cannot find one for or for the providers of derived shipping density data maps.

    categorised as 6-30 per month (blue), 31-150 per month (green), 151-300 per month (orange) and 301+ (red)

    Many thanks.

    1. EMODnet Human Activities says:

      Hi Mark, the image from havbase is just an example of what our final output might look like. You should contact havbase for more details, as we’re not inolved in their project.

      In our case, we’ll measure density by building routes from AIS messages, calculating their length, then dividing length by average speed (speed is averaged between two consecutive positions). So, we’re going to have a time value. To cut a long story short, rather than giving the average number of vessels per cell per month, we’ll give the average time spent per month per cell.

      Our maps will be made available before the end of 2018, so stay tuned!

Leave a Reply