Good news Feature Manipulation Engine (FME) users! If you want to crunch your GIS data in BigQuery, it’s now possible with FME available in the Google Cloud Marketplace. This announcement is a gamechanger as it means that geospatial data processing power can now be performed at a previously unmatched level.

GIS professionals are really excited about this news for two main reasons:

1. FME is really popular in the GIS world because it has the best support for spatial data integration, worldwide. FME allows users to build custom workflows that improve access to data and solve compatibility issues, without needing to code anything. Their ethos is that you spend less time fighting data and more time using data to solve problems. Simple!
2. Google has responded to the fact that the world is collecting, storing and analysing more location data than ever before. Their response comes in the form of making sure that BigQuery has first-class support for geospatial data types and functions. The FME integration with BigQuery process means you can now analyse geospatial data using the incredible scale BigQuery is known for.

SO, WHY IS THE FME AND BIGQUERY INTEGRATION A BIG DEAL? 

The first thing you need to acknowledge when answering this question is that there are a myriad of GIS file types and projections that exist. Storing these in a data warehouse means that you need to transform the data type and it’s projection into the native projection of the data warehouse you’ve chosen (this is why you’d use FME).

There are two key things to note if you choose to store your GIS data in BigQuery:

1. BigQuery GIS uses the WGS84 coordinate system
2. The workflows are scalable – within FME you can now ingest, transform and load data directly into BigQuery tables

In short – the former cumbersome process just got a lot easier and quicker.

How does it work? 

If you want to get started using FME (Desktop & Server) and BigQuery via a pre-installed virtual machine (VM) on Google Cloud, you need to follow these steps:

1. Head to the Google Cloud Marketplace and find the FME Platform 2020.0 page. Select Launch on Compute Engine button
2. A Configure & Deploy window will appear: Add in values for each zone, machine type, etc. When you’re ready and click Deploy
3. Your VM with FME Desktop and FME Server will now deploy
4. Once the deployment has finished, head to Compute Engine, select the VM & set the Windows password so you can login to the VM
5. You should download the RDP connection file via the RPD link and connect the virtual machine using your remote desktop application

WHAT ABOUT FME AND GOOGLE CLOUD LICENSING?

The great news is that if you’re already using FME in your organisation, when you access FME in the Google Cloud Marketplace you’ll be prompted to login to your FME account. If you don’t have FME, sign up for a free trial when you register for an account.

If you don’t have a Google Cloud account in your organisation – Google is offering USD $300 in credits so that you can get started querying your data in BigQuery. This is great if you want to test the integration out or simply see what’s possible with BigQuery.

Once I have stored and queried my data in BigQuery, how can I visualise it? 

Something important to note is that you still don’t need to remove your data from the Google Cloud Platform to create beautiful maps using the data you’ve ingested via FME. As part of their ongoing support for large-scale GIS functionality on the platform, Google has partnered with CARTO – a partner of NGIS – to enable users to connect their BigQuery tables directly into CARTO. From here, you can perform spatial analyses and create truly impactful maps.

All you need to do is search for CARTO on the Google Cloud Marketplace, login or register for a trial CARTO account.

Need help with product licensing or understanding which data warehouse is best for your GIS data? 

NGIS has partnered with these organisations to help arrange your licensing needs:

1. Google Cloud – Location-Based Services partner specialisation
2. FME – via our partnership with 1Spatial
3. CARTO – master reseller of CARTO in the APAC region

NGIS has also recently released a report comparing the Google Cloud BigQuery and Snowflake for storing and processing GIS data. It’s free to view and you can access it via this link.

You can chat with our team if you need further information.