Feature Flags in FME?
What is FME?
Within our team we use a variety of tools in our daily grind as a mixture of software engineers, GIS developers, testers and general data enthusiasts! One of the tools we use is FME, you will either know it or you won’t! But rather than me butchering what it is, pop over to the good folk at Safe and have a quick read.
In a nut shell though it allows a code-free method of manipulating data in all manners of interesting ways. It will read in all manner of data types, allow a wide range of data manipulation and ultimately output it in a wide and varied manner of new data types such as CSV, database tables etc. Primarily it was considered to be a GIS specific tool but I think they are moving very much more toward data manipulation full stop.
What is a Feature Flag
So this blog is about taking a ‘code-free’ piece of software and incorporating a very developer-esque feature, namely Feature Flags! What are Feature Flags?
“Feature Toggling” is a set of patterns which can help a team to deliver new functionality to users rapidly but safely.
martinfowler.com
Feature flags allow developers to enable or disable features they build and deploy it into live environments without affecting the end experience of users. Typically this is a good CI pattern and allows dev teams to release code quickly and in small chunks without a massive release or test burden. These flags can allow this new code to be deployed and tested before being toggled on/off prior to releasing to live.
Another useful article explaining Feature Flags can be seen at Feature Flags, what are they and why should I care?
We needed to deploy a feature in a work space that would allow another team to complete their work but unfortunately we couldn’t push that branch to the integration environment because it is used by many teams; hence the name! Therefore we needed to deploy the new work space so it would complete but not terminate our process while it was not fully integrated with the other team.
Our Dummy Work Space
I have created a dummy work space which will just have some very simple transformers, in this case we will simply log a new value called new_feature with the value of ‘The flag is not running’ and log it in the Translation log.
As you can see the message in the log file is:
Logger: The flag is not running
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `Logger_LOGGED'
Attribute(string) : `fme_feature_type' has value `Creator'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(string) : `fme_type' has value `fme_no_geom'
Attribute(encoded: utf-8): `new_feature' has value `The flag is not running'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================The Challenge
Now I want the ability to pass in a Feature Flag at run time to change this message! If my flag is TRUE I want to populate my new_feature with the value ‘The flag is working, yippee’. However if my flag is FALSE I want my new_feature value to stay the same.
Published Parameters are your friend
Within FME there is a concept called Published Parameters, these allow you to expose parameters within the work space at run time.
As you can see from the diagram above we currently have no published parameters. We should make one! First thing to do is right click on Published Parameters and select Add Parameter; the following dialog box opens up.
Lets break this box out as we go along so we understand it.
The first element is Type and this allows the user to select the type of input we want our published parameter to be. This can be anything from a list, a choice similar to drop down, date/time, string etc. I have even used this as a password before where it is encrypted by Ansible Vault and is passed in when the work space is deployed to FME Server. This ensures we see no plain text passwords anywhere. For our simple example we are going to leave this selection as Text, but I really recommend getting in there and playing with it.
The next field is Name, this is simply the name of our value so name it something meaningful, try this article for a starter for ten on naming. I will call this parameter isFlag. The two check boxes beside Name are:
- Published: We want to leave this checked because we want to keep it exposed and not convert it to a Private Parameter. This gives us options for CI etc later and means we don’t ‘hide’ it at run time.
- Optional: If we tick it off then it will prompt for a value every time the work space runs. For this example we will leave it ticked.
Prompt is a more meaning full description of our Name so for this example we can simply leave it as isFlag and we will populate our Default value as FALSE i.e. the Feature Flag is not available. Our final dialog should now look like:
You will notice that we currently don’t use this parameter anywhere yet but our work space still runs successfully which is the aim of having a flag.
Now we having a running work space with our flag safely integrated we can add the new features we want. Below we have added in some new code into our work space which we will want to run in the near future but not just yet. When we are ready to use the new code we simply change the value in the Published parameter from FALSE to TRUE!
Below shows the flag has been updated to run as TRUE, and all is good with the world.
And our log file says:
Logger: The flag is working, yippee
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feature Type: `Logger_LOGGED'
Attribute(string) : `fme_feature_type' has value `Creator'
Attribute(string) : `fme_geometry' has value `fme_undefined'
Attribute(string) : `fme_type' has value `fme_no_geom'
Attribute(encoded: utf-8): `new_feature' has value `The flag is working, yippee'
Coordinate System: `'
Geometry Type: IFMENull
===========================================================================So what have we done
By having the tester looking at the published parameter we can now send our process down either route with the expected outcomes in a safe manner and allows us to promote code rapidly and safely. What I like about this approach is:
- It is simple and effective
- It enhances our work space with additional functionality in a safe way
- There is no use of a blocking transformer. This is an FME concept and is essentially a transformer that will store all the data until it completes it’s operation e.g. a Sorter transformer will hold all values and then sort them before releasing them to the next transformer. This is important when you are thinking of processing and performance of work spaces.
Conclusion
The aim of this post is to try and show you a safe way of releasing FME code in a CI manner which allows developers to continue working rapidly providing value to your organisation. If you want to have a play the code can be found here in GitHub.
In my next blog, if I dare to try this again, I will show an example of deploying some work spaces using Ansible and Jenkins.