Atlassian Stash
Matthew Watson leads the Stash team at Atlassian. Released a minimum viable product really early, then very short delivery cycles; provide incremental improvements to both deliver features to customers but also to validate the product (“we will catch up”).
Agile with git. Use feature branching to allow for isolated development with functional and performance testing of your code as you develop it. Isolate stable code from work-in-progress.
Code reviews (aka pull requests) to help ensure higher quality. Needed to inculcate a respect for quality in the team; absolutely required for continuous deployment.
Every commit in master results in two builds and pushes the result to the “dog-fooding” deployment. Also runs a bunch of other checks against these successful builds.
Whenever a feature branch is created, CI server creates a new “CI plan” for that branch. These feature branch jobs are quite highly optimised compared to the master.
Several build styles:
“Checks” builds use checkstyle, findbugs, API compatibility, licensing, link check docs. This is pretty fast; 4-5 minutes.
“Master” builds, every green build is deployed to dog-fooding instance. Parallel; distribution, functional tests (40 minutes), jsunit, jshint, DB migration, REST and hosting testing, first run, unit tests. Major gatekeeper (~11 workers)
“Dependent” builds: database matrix (12 versions of 5 DBs), git version matrix (12 versions), plugins, source, git on Windows, hosting and REST on Windows. Triggered when “master” succeeds; so, e.g., Windows errors don’t prevent getting code into master.
“Feature branch” builds include “checks”, Stash distribution, unit tests, jshint; these are all pretty fast (~4 agents, 3-4 minutes; fast feedback to developers). Developers can trigger second stage with all the other test from “master” if they think they’re required.
Release branching
Stash needs to support multiple releases:
master
release branches (2.5, 2.6); lives on with bug fixes, etc.
merge bug-fixes back toward and into master (in 2.5, merge into 2.6, then into master). Plugin to do this sort of stuff automatically, create pull requests for merge conflicts.
Master and release branches all get full build plans.
Release build
Fair number of builds made internally.
Stash team have internal dog-fooding instance.
Atlassian has a corporate instance used by other teams.
Use a release job, parameterised by a specific commit (not necessarily the head
of a branch). Pass in version and next-version for Maven. The build job runs
maven release:prepare release:perform. Not much testing because it’s already
been through all the testing.
Create a temporary branch for the process of the release build.
Automated deployments
Release artefacts for customers:
- www.atlassian.com, developers.atlassian.com
- marketplace
- Checkup (for third-parties to test their plugins)
- Go live
Internal deployments:
Dev staging
Staging (same data as stash.atlassian.com) for smoke tests, etc.
Production
Performance testing
Six-thousand seat licenses with lots of CI running against it. Performance is pretty important.
Daily monitoring of performance of operations. Check results in stand ups. Notice regressions in performance.
Techniques
Try to always be ready for release - quality code.
Automate testing as much as possible.
Automate processes:
Releases
Deployment
Automated environment provisioning
David Cheal is Chief Engineer at Krunchtime IT. They do AWS solution design, build and delivery; devops approach; agile; etc.
Life cycles are changing
Traditional ops environments
Virtualisation for cost efficiencies
Cloud
None of the problems we had at (1) are still there at (3), we’ve just given that problem to Amazon.
Continuous delivery infrastructure
Deploy infrastructure often, automation, etc. Help to detect and prevent configuration drift between environments, etc.
Options for infrastructure agility
CloudFormation with baked AMIs
Automation tools like Puppet or Chef to automate configuration and deployment.
Separation of concerns
Code deployments are not infrastructure deployments. Leave application and dependencies to infrastructure deployment.
Continuous challenges
Technological challenges: Windows, anything from MS, learning curves
Cultural change: change, resistance, etc.
Example
Communities can be a guide; Puppet Forge is a guide but they must be reviewed.
Pragmatism
Infrastructure is code.
This is not true; infrastructure is infrastructure.
Opportunity
Embrace new methods.
Incredible opportunity to deliver what business, operations, and developers need more quickly, reliably, cost effectively, etc.
Demonstration
Simple Ruby on Rails application using AWS CloudFormation to spin up EC2 instances, RDS database, ELB, etc. Use an existing Puppet master.
Most organisations use a naming convention which allow Puppet to determine which classes should be applied to each server, with the right environment.
Using ELB, most organisations will include a specific “health check” page in their application. Makes it very easy to have ELB.
Continuous infrastructure delivery is – when done right – really, really boring.
Cloudiness
Elasticity lets you scale according to demand. “Deal of the day” sites 2-15-2 machines. Development infrastructure shutdown overnight and weekends.
Immutable, disposable infrastructure. Cows vs dogs.