How Software Releases Should Be Done!

A place to discuss software and code that isn't automotive related. Free and open source preferred but not compulsory.
Post Reply
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

How Software Releases Should Be Done!

Post by Fred »

I'm hoping to make this as technology agnostic as possible. We'll see how I do. Feedback from other experienced devs is welcome too. EssEss and sry_not4sale, for example, will likely have their own thoughts on this. It's not likely to be a bible as I'm smashing it out very quickly. It could grow into one, though.

Principals
  • Automate everything except what you can't.
  • Verify everything that you can.
  • Manually triggered, automatically executed.
Assumptions
  • That the build is a single step kicked off by the developer and/or release manager, including all end product output artifacts
  • The source code and all required scripts and configuration are checked into some sort of source control
  • That the state of the source code and software system is of a sufficiently high quality for release and well tested, both automatically, and by users/QA
Automation

Things that can't be completely automated because they're human by their very nature:
  • Version if it's arbitrary, or confirmation of generated if not.
  • Tag name if it's arbitrary, or generated from version if it's not.
  • Tag comment (if possible, some SCMs allow this, others do not)
  • Release announcement / release notes intro
  • When to actually do the release, and what to do the release from.
Things that can/should/must be automated because not doing so introduces opportunity for human error:
  • Quality of the build (build broken/not, unit tests executed, fail if not all there, unit test coverage sufficient/not, style checks, bug finders, etc)
  • State of the repo (all files checked in, no modifications, no un-ignored files, up to date with remote repo, remote repo up to date with local copy)
  • If version is configured/hard coded, incrementing of version, otherwise requesting it from the release manager/dev
  • Committing of any changes such as version number/name/date stamp, if applicable
  • Tagging of the repository using name and comment provided by the release manager/dev, possibly with defaults suggested
  • Change log coarse (pulled out of issue tracker, or from commit comments if disciplined)
  • Change log fine (pulled out of SCM)
  • Build and packaging of application or component (binaries, installers and other distributables)
  • Documentation from doxygen/javadoc/markdown/latex/etc into end formats such as pdf and html
  • Roll forward of any configured versions etc to the next development variant
  • Upload and distribution of documentation and components to either a staging location, or the final location
Compromises

Ideally none, however sometimes that's not possible. In particular cross platform code that must be built on its native platform. Often you can build for windows on Linux or vice versa, but for Mac OS X, it's regularly required to use OS X to generate certain final products (eg DMG or PKG files, and native compilation against cocoa libs). In such cases, if possible, use the platform that allows the most platforms to be built for at a single time, which ever that is. For the balance, they can have binaries built from the SCM tag after-the-fact.

Release Steps

Our goal thus far has been to make this section of the guide as short as possible by engineering the system to fail for us if something's not 100% right.
  1. Ensure all files are checked in, or removed.
  2. Ensure the changes are on the server.
  3. Ensure changes from the server are local too.
  4. Ensure the system builds successfully from a hard clean environment
  5. Ensure all automated testing executes cleanly and returns successful results
  6. Smoke test the app or library by using it in as many ways as possible, preferably against a script/list of things to check/do
  7. Once happy, initiate the release by executing your release script/process
  8. Answer/fill in details requested by the release process such as version number, tag name, tag comment
  9. Watch the release unfold/drink coffee/drink beer/drink wine/drink water
  10. Download resulting binaries from automatic upload locations onto other operating systems and verify that they look correct (versions and other ids correct, etc) and work properly
  11. Assemble any manual documentation steps such as issue tracker dumps
  12. Make the announcement linking to the uploaded materials such as binaries and documentation
  13. Initiate the process of building other OS variants (such as OS X) from the tag, or having others build from the tag on your behalf
Notice that there's still a LOT to do and do carefully and correctly, even when the release process is largely taking care of itself. That's why it's crucial to automate as much as possible (everything that's possible), because otherwise there is so much for a dev to do and remember that they will inevitably forget *something* and reduce the overall quality of the release, somehow.

In the next couple of posts I'll run through two examples of the above in action.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: How Software Releases Should Be Done!

Post by Fred »

For the firmware I use Git and ordinary GNU Make. I have a release process encoded in the Makefile, although a bash script would possibly be a better choice.

My stuff:
  • Checks to see if the current HEAD is the same as the remote HEAD
  • Checks that all files are checked in/the repo is clean
  • Builds the firmware once to verify that it can be built
  • Asks the operator for a tag name and tag comment and tags the repo
  • Builds the documentation and firmware variants
  • Uploads the documentation and firmware images
  • Pushes the tag to the remote repo
It could be better, but it's pretty good.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: How Software Releases Should Be Done!

Post by Fred »

For Java projects, the standard build tool (Maven 2/3) has a sophisticated release mechanism built in as part of the standard process. I modeled my stuff around this, because it's built around all industry best practices, and hard to beat. Other tools for Java simply emulate what Maven does to various degrees.

In a typical maven build, the version is in the form 0.1.2.3-SNAPSHOT during development, corresponding to 0.1.2.2 being the last released version and 0.1.2.3 being the version you're about to release. The system therefore automatically removes the snapshot before tagging, and increments the last portion and readds snapshot after the build and release is done.

The commands used are:

Code: Select all

mvn release:prepare
And

Code: Select all

mvn release:perform
With these, just as in my make-based setup, binaries are uploaded to the configured location and docs are uploaded to where ever they should go live.

In Java land, it's completely normal to have your project broken down into N small self contained libraries. Such libs do not need a fancy release announcement or anything like that. Thus such libs can and do get released at what some might consider an alarming rate, minutely, or hourly, for example. This is made possible by the total automation of the release process. After living in this environment you can not go back, hence going to the trouble of emulating it in make for the firmware.

The linux/C process might look like this if it were done the same way:
  • Generate a debian archive
  • Upload the .deb to the debian repo
  • Automatically apt-get install the lib as required based on declared dependencies in your build setup.
Unfortunately C/C++ stuff isn't quite as simple as Java stuff when it comes to shared libraries. Still, from an application stand point, which is what this thread is for (Hi Mike!, Hi Sean!), the same type of process can be followed and the release quality can be held very high indeed.

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: How Software Releases Should Be Done!

Post by Fred »

I hereby open the floor to comments and questions. Comments from experienced devs and questions from anyone else :-)
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
andg
SOT-223 - Salvaje
Posts: 144
Joined: Thu Apr 18, 2013 3:21 am
Location: Portland OREGON
Contact:

Re: How Software Releases Should Be Done!

Post by andg »

I'm using Travis to perform automated build tests which notify inside of our development chat. Github has hooks that allow the pull requests to have a comment from the continual integration service. It builds the environment and reports if the tests passed, failed, or erred. This allows the code release manager (me) to know if the quality is worth reviewing or not. Our last dozen builds have all failed due to changes in their configuration. I now have to tell it what database to have available to it, and the caching server. Travis is free for open projects, so you might want to think about it. Plus they sent me coffee for Christmas.

This continual integration testing service is crucial for ensuring the test automation happens in our environment and it is performed by an unbiased third party that creates the build environment from scratch every build.

Code: Select all

{url: andrewgauger.com, skype: andrgaug, email: andg@andrewgauger.com, car: 25, fb: facebook.com/DonatoArrighi, github: andg.github.io, assets:https://s3.amazonaws.com/3400}
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: How Software Releases Should Be Done!

Post by Fred »

You're the second person to go off-topic with the CI/unit-testing/integration-testing stuff. This is purely about releases ONCE you've decided that things are good enough to release and that it's the right time, and there is some specific point to releasing. :-) CI and testing is another topic, to be covered another day, in another thread. For that I've used Hudson and Jenkins in the past and will use Jenkins again in future.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
andg
SOT-223 - Salvaje
Posts: 144
Joined: Thu Apr 18, 2013 3:21 am
Location: Portland OREGON
Contact:

Re: How Software Releases Should Be Done!

Post by andg »

CI is off topic from a release standpoint excluding the fact that I use the CI as part of ensuring the test suite has passed. There may be a better way of doing it, but I use the CI to ensure release steps 4 and 5, and because there is no difference in test automation for me than what I use for pull request validation. Additionally, you can set up your test suite to include performance tests and have those run through the inclusion of a flag. I like to do performance benchmarks before release, and have those performance benchmarks constant throughout the release schedule. Only way to do a performance benchmark is on a clean environment, with a consistent amount of capacity.

Oh, and my release schedule is 2-5 times a week, so that plays into why I have a CI server assisting me with my release management.
Last edited by andg on Thu May 02, 2013 10:09 pm, edited 1 time in total.

Code: Select all

{url: andrewgauger.com, skype: andrgaug, email: andg@andrewgauger.com, car: 25, fb: facebook.com/DonatoArrighi, github: andg.github.io, assets:https://s3.amazonaws.com/3400}
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: How Software Releases Should Be Done!

Post by Fred »

Yep, fair enough :-) Still, points 4 and 5 were "do this" not how to do this. I guess the examples don't hurt.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
User avatar
Fred
Moderator
Posts: 15431
Joined: Tue Jan 15, 2008 2:31 pm
Location: Home sweet home!
Contact:

Re: How Software Releases Should Be Done!

Post by Fred »

It seems that the more I exist, the more I stumble across people simply "doing it wrong". Recently some the people doing it wrong thought they'd have a vote on changing their wicked ways. Sadly, despite best efforts in several corners of the ring, the vote failed, and the majority of idiots decided they'd continue making multiple binaries with the same name until one was good enough. I reiterate, idiots. No matter what they've done in the past, this behaviour is absolutely unprofessional and unacceptable. I let them know that I had better things to do with my time than try to educate them (for example taking a piss) and went on my merry way. I was thinking about the entire concept of "burning versions" (their justification for doing what they do) and all I could conceive of was the literal interpretation, so I came up with some images, and dumped them into IRC for everyone's amusement. Shared here as extremely relevant to this thread.

(11:18:47) fredcooke: I've been thinking about this "burning versions" terminology recently.
(11:19:04) fredcooke: The only thing I could come up with in my mind that made any sense at all was a literal interpretation.
(11:19:37) fredcooke: As such, I decided that it would be *MUCH* more useful to spend some time learning the Gimp than to try to educate my peers.
(11:19:47) fredcooke: May I proudly present version 1.0.0:


Image




(11:20:40) fredcooke: Not satisfied with 1.0.0, there were more iterations, sadly, they didn't make the grade, so the next publicly released version was 1.1.7:


Image




(11:21:01) fredcooke: Still not happy, I had a few more stabs at it, and came up with 1.2.13:


Image




(11:21:34) fredcooke: Unfortunately, I didn't really like that either. I kept working on it, and came up with 1.6.2:


Image




(11:21:57) fredcooke: As you can imagine, quite a few versions were "burned" between 1.2.13 and 1.6.2, RIP, versions.
(11:22:31) fredcooke: Frustrated with my previous efforts I really focused on the task at hand, and am now very happy to officially announce 2.0.0 of my masterpiece:


Image




(11:23:08) fredcooke: Unfortunately, development has ceased as I was issued a warning from the Society for Cruelty to Versions.
(11:23:11) fredcooke: Thank you for your time.

May I reiterate:

DO NOT, UNDER ANY CIRCUMSTANCE, FOR ANY REASON, NO MATTER HOW TEMPTED, EVER EVEN FUCKING CONSIDER REUSING VERSION NUMBERS ONCE PUBLISHED!!!!!

/RANT

Fred.
DIYEFI.org - where Open Source means Open Source, and Free means Freedom
FreeEMS.org - the open source engine management system
FreeEMS dev diary and its comments thread and my turbo truck!
n00bs, do NOT PM or email tech questions! Use the forum!
The ever growing list of FreeEMS success stories!
Post Reply