Using Spring Cloud Config Server

Problem definition

I have already described why do we need external configuration management.

In short, let's say we have several environments (production, simulation, test). As the development goes we run different versions of our application on these environments. Part of the configuration of the application is specific to an environment and may change out of sync with our releases. So it is bound to the application version and the environment but should not be stored in the same repository as the code.

We used to have a tool to handle such external configuration management but since it was in-house developed and I moved away from that company I would have to write it from the scratch. We also made some design mistakes (detailed in the aforementioned blog post). So I looked around and found Spring Cloud Config server which can be backed with Git repository to store configuration. The beauty of having Git repository for configuration is that every configuration change is tracked and hence audited. Also tools to edit the configuration are already out there - file managers, IDEs, text editors. Moreover it should be easy to set up since it is just a Spring Boot application.

Spring Cloud Config

I gave it a try. I created a project which has only the dependency to Spring Cloud Config. I set the URL to the Git repository with configuration, deployed it and was ready to go.

pom.xml

    4.0.0

    com.company
    config-server
    1.0-SNAPSHOT

    
        
            org.springframework.cloud
            spring-cloud-config-server
            1.1.2.RELEASE
        

        
            org.springframework
            spring-core
            4.2.6.RELEASE
        
    

    
        config-server
        
            
                org.springframework.boot
                spring-boot-maven-plugin
                1.4.0.RELEASE
                
                    true
                
                
                    
                        
                            repackage
                        
                    
                
            
        
    

application.properties

server.port: 8888
spring.cloud.config.server.git.uri: ssh://git@hostname:1111/our-app-config.git

The server supports API calls like

• /{application}/{profile}[/{label}]
• /{application}-{profile}.yml
• /{label}/{application}-{profile}.yml
• /{application}-{profile}.properties
• /{label}/{application}-{profile}.properties

One can also provide an Accept header to get for instance binary file:

-H "Accept: application/octet-stream"

.

Versioning

Since we strive to get continuous delivery working the version is assigned with every commit. Consequently, any commit can be deployed up to production. This version is derived from the last annotated tag (e.g. 1.0), number of commits since then, and the commit SHA. It may look like: 1.0.42-gda31562.

We have four environments:

• Test - for our testing
• Acceptance - for customer testing
• Simulation - production hotfixes
• Production

And two external configuration files:

env.properties - environment specific properties
logback.xml - Logback configuration

We use Ansible for automating the deployment. It places these two files to the lib folder of Tomcat.

The configuration Git repository that is used by Spring Cloud Config contains these files in the root folder:

env.properties - with defaults
app-acc-env.properties
app-simu-env.properties
app-prod-env.properties
app-logback.xml
app-prod-logback.xml

Ansible makes API calls like: http hostname:8888/app/acc/master/app-env.properties to retrieve app-acc-env.properties from master branch. So far so good.

Branching

Ansible can either ask for master branch and get the latest configuration (we use that to deploy local builds to 'test') or it can use the 'label' to get configuration for specific version (hence we use version as a 'label'). That forces us to create an annotated tag everytime we release to the Artifactory - so everytime we release to acceptance or higher.

If ever we need to change the configuration for a version we convert the tag for that version to a branch and make changes on the branch.

Changing Configuration

We expect there will be two scenarios:

• One needs to add/modify a property because of a new feature, new code. Then we can commit the changed configuration to master branch in the configuration repository. When the next version with the change is released the property is there. That is also one of the main benefits of having the external configuration management.
• One needs to change a property for running version on one or more of the systems. Then it gets a bit tricky. Let's say we have tags: 1.0.0, 1.0.10, 1.0.25, and 1.0.40 I have 1.0.10 running in production and want to set say new email server URL. Then I need to convert tags 1.0.10, 1.0.25 and 1.0.40 to branches and make my change in all the three branches. That may get rather annoying eventually especially if we do not release to prod often and there are many tags.

Problems

The main problems I see with this approach are:

• The need to tag every release.
• The need to change potentially many of these tags - convert them to branches and perform the same modification.

Our versioning scheme produces monotonously increasing versions (with slight caveat for branches but we deploy to customer facing environments only from dev branch which mitigates this issue). Hence in our case these problems can be solved with the notion of version ranges. If I have versions 1.0.0, 1.0.10, 1.0.25, and 1.0.40 and there was no configuration change since 1.0.0 I will have just the one configuration for version 1.0.0 and it will be used for all the versions. If I introduce configuration change in 1.0.42 I create next version of configuration but everything in between will still get the configuration for 1.0.0. When I need to modify some property from version 1.0.25 I can create branch from the version bellow - 1.0.0 - and change it there. Then even 1.0.40 would get this change. Out of luck for 1.0.42 but at least I would not need to change the setting in too many places.

I'm new to Spring Cloud Config so maybe I missed something obvious. Let me know your thoughts.

Benefits

• Configuration can be altered outside of the release cycle of the code.
• Configuration changes are audited and documented (with commit messages).
• It is possible to look up what the configuration was at the point of the deployment - the deployment script logs time of deployment and version so one can look up corresponding configuration.
• We don't need to think about all the configuration changes necessary for given release since they are performed in advance and then updated as the changes occur and not at the point of the release.

Why do we need a configuration management tool?

I spent some years developing web applications in Java for the corporate world. The apps always used some webcontainer or application server and we used different staging environments to show our work to testers, managers and customers before we went to production. Unfortunately, there are always some differences between these environments, be it a feature turned off or a database password or a mail server URL. So one needs to provide, ideally external, configuration file for every environment.

The usual way to do that few years ago was to define whatever configuration necessary, place it on the server and be done with that. In our case we put a logback.xml and an app-env.properties to the lib folder of Tomcat. Whenever a property is introduced or needs a change one asks a fellow ops guy and he performs this change on the day of the deployment. It worked most of the time but there are couple obvious issues with this approach:

• Error-prone – typos, missed changes, missing properties.
• Unknown configuration – in corporate world it is the ops who have access to production so there is no way to know the actual content of a configuration file for a poor prod-issue-solving developer.
• Impossible full deployment automation – since a manual step is needed from time to time.

We lived with these downsides for a while, suffered some break downs of the app, or the RabbiMQ broker (our OPS never really embraced the 'infrastructure as a code' principle), or a broken database. Ultimately, the change came along and we started the internal DevOps pilot. One goal was to do automated deployments. Now you can write a script that stops a Tomcat, copies over the war and starts it, you can use Flyway or other database migration tool and run a command to do the database migration for you, but what about the external per environment configuration?

Surprisingly, lot of people I talked to thought that configuration is easy. You just check in the version control system the configuration changes along with the feature. I believe they miss couple interesting corner cases. Let's use an introduction of a mail server as an example. A developer implements first feature which sends emails. He tries it out locally with a postfix server and adds mailServer.url property and wants to check it in but:

• The value is not known for all environments as ops may be able to provide you the right mail server URL weeks (and I saw months:)) after you've finished the ticket.
• The value may change in the future.
• Additionally, the value should be there only since the very first version which includes the commit.

If you want to get close to continuous delivery you may want to maintain that any commit that builds successfully can be delivered up to production. But then again, where and how should you store the configuration?

Given the example above we concluded that the configuration files have different lifecycle from the code. They are changed at different times. Moreover, they are shared between dev and ops teams since they both maintain certain parts of the configuration.

We decided we needed a configuration management tool with following properties:

• The deployment tool can get from it a zip file via an HTTP GET call. The tool provides project name, environment name and version.
• The zip contains all files in the correct structure that can be extracted to a designated folder.
  • e.g. /lib/certificates/server.jks
    /lib/logback.xml
    /lib/application-env.properties
    /conf/server.xml
• Only authenticated users can change files.
• Every action is audited – no more shrugging and not knowing who changed a value and when. No more what version of configuration was there when an issue happened a week ago.
• (Optional) Support for 'secrets' – e.g. production database password visible only for certain group of users (ops).

At the point of the decision there were no obvious solutions out there so we developed in-house a simple web application. Obviously it was a web app since when one has a hammer everything looks like a nail:). The app had a UI for listing files per project and environment. It had some neat syntax highlighting, it allowed you to add, copy, diff, edit, delete files, etc. It stored files in the database. The UI part needed a lot of maintenance and feature work yet it was clumsy to work with. It offered HTTP API to get the zip with configuration. It solved most of our problems. Since its roll-out we could finally see how are the customer facing environments configured. When a new property was introduced we could make the change, commit, take the version of that commit and create respective versions of the property files in all environments leaving TODOs where we did not know the right value. No more documentation about what all needs to be changed with the next deployment. No more figuring out what kind of configuration do I need locally to be able to run certain version of application.

I will elaborate a bit on how the versioning of files work by providing following example. Let's say the app is released in version 1.0. There are two files for each environment 'logback.xml' and 'app-env.properties'. We decide we want to change logging a bit in version 1.0.10 so we add updated logback.xml for version 1.0.10 and higher. Then a mail server needs to be added. The feature is implemented in version 1.0.42 so we add updated app-env.properties for version 1.0.42 and higher. So far our configuration management contains four files for each environment:

• app-env.properties/1.0.0
• logback.xml/1.0.0
• logback.xml/1.0.10
• app-env.properties/1.0.42

If one needs app configuration for given environment for version say 1.0.5 he will get the initial two. If one needs app configuration for 1.0.32 he will get app-env.properties/1.0.0 and logback/1.0/10. For version 2.0 he will get app-env.properties/1.0.42 and logback/1.0.10. So there is a requirement that versions can be lexically ordered.

The above described approach worked fairly well and we were autodeploying for more than a year with great success. Downtime of 40 seconds and the peace of mind were totally worth it. But on the hindsight, we spent one hell of a time developing features of the configuration management tool which could be covered by choosing another approach which came to our minds. Let me present it to you. Most of the work on the configuration management tool was spent on the UI and all the file management, syntax highlighting, and diffing. Despite all that work it is still not great! Yet there are so many tools around that can do parts of that work much better, namely our beloved IDEs. There are also good file storing solutions out there with auditing and we happen to call them version control systems:) So why not to pick Git with vim or Idea, create a predefined folder structure and let some daemon loose on top of it to serve the configuration zip file?

Our plan is to try just that at techdev, try it on shypp.it first and then open source it. We will keep you posted.

Also if you know about such a tool please let us know! We don't want to reinvent the wheel all that much.