If we are talking about Brancing Models in Software Development, like GitFlow, GitHub Flow, etc., Trunk Based Development (TBD) comes out as an important Brancing Model.

Stated in 2017 State of DevOps Report TBD is correlated with high performance teams and a factor that positively contribute to Continuous Delivery https://www.linkedin.com/pulse/state-devops-reports-2017-trunk-based-development-practices-lopez/.

However, in my experience of adopting TBD in my Software Development Team, there were some so uneasy thing related to mindset change which is required for TBD, that although the team said that they were adopting TBD, but in the practice they are still doing antipattern of TBD.

Moreover, disbelieve & resistance have been observed about the concept, many times those were shouted about the How.

In this article, since we know that TBD is famous as a key enabler of Continuous Integration and by extension Continuous Delivery, I will talk about:

  • the Issues if we continue the Feature Based Development WITHOUT the Trunk Based Development and HOW Trunk Based Development solves it to understand the concepts
  • I will talk a bit about WHAT IS Trunk Based Development to differentiate with WHAT is NOT
  • I will talk about the Issues in adopting Trunk Based Development and HOW are the practices to solve it. We will talk a bit about Feature Toggles, Branch by Abstraction, and Development plan by using Black Boxing such as Flow Based Programming.

Issues in Feature Based Development with Long Living Branches

Long lived feature branch implicates long period of time without feedback. And also developers who normally work with long living feature branches also tends to have a big black box of implementation. It is even worse if the user story is not sliced small. There are tendencies to have a big implementation which in worst case not closed within a cadence such as a scrum sprint.

It causes absence of visibilities to Developers of Team Members, but also to Product Owner or other stakeholders about the progress causing anxiety within team and sometimes loss of trust but also loss of good feeling of tracking and estimation within the developers and the team members themselves.

In case you also adopt Code Review Process, the Code Review Process is also becomes difficult and often there are some innertia within the developers to review, since it takes a lot of time of reviewing a big implementation.

For teams who still not adopts TDD / Test Driven Development, there is also tendency that unit tests also come late, since psychologically people tend to be more lazy to write test code, before feature is completely done. Tests come even late in development tests.

Merge conflicts problems (unproductive merge) also often occurs. Some element of (negative) surprises as well because tests come late (no shift left culture), generally it causes late feedback.

Since there are less merge into code base (master / trunk branch), if feature is an epic, there normally nothing to release even after several days / weeks, which implicates NO / low level of continuous delivery and the deployments / releases also become high risk deployments because we need to do bigger releases instead of releasing small-small things incrementally.

Issues of Fixing Issues Directly in Release / Production Issue Branches

Release Branches and Hotfix Branches are a common concept if you are working with branching models like GitFlow, GitHub Flow, or other Feature Based Development.

So far Feature Based Development mostly teaches about how to structure branches but does NOT teach about healthy branching model.

Another problems which often occur, that people often fix directly issues in release branches or hotfix branches but they forget to merge them back to their code base and moreover forget test the code base again after merge, causing more production issues to be fixed, since we would put focus on Release Branches and Hotfix Branches instead of focus on Code Base / Trunk.

Trunk Based Development Essential Concept

Feature Based Development talks about How to Branch, TBD is talking about Healthy Branch:

  • Short Living Feature Branches by one developer / one pair / one mob (and NOT multiple parallel persons)
  • Fix to Code Base first before fixing the others

We can group them to 3 Keys Activities:

  • Development, including Bug-Fixing
  • Releasing
  • Hot-Fixing

Development

  • Short-lived feature (/ bugfix) branches: no longer than 2 days (longer => anti-thesis of TBD)
  • Each branch by 1 activity, NOT parallel activities (NOT by multiple developers working in parallel)
  • Merged code in Trunk / Master doesn’t break the master, and master is in releasable state
  • Any feature longer than 2 days, to be broken down (by abstraction) to shorter feature branches

Releasing & Hot-Fixing

  • Release / hotfix branch (https://trunkbaseddevelopment.com/branch-for-release/) is there only when necessary, on incompatible policy, late, and to avoid freeze
  • In case of using release / hotfix branches, fixing is to be done & tested in both release / hotfix branch(es) and trunk (master)
    • Since people tend to not fixing (or forget) into master, the policy says, fix it in trunk / master first than (cherry) pick it to the release / hotfix branch(es) so we have the fix in all required places

What is NOT a Trunk Based Development

  • Merely naming code base as Trunk, without doing TBD Mindset
  • Feature (/ bugfix) Branches is longer than 2 days
  • A Single Branch is done by parallel activities / multiple developers (which are not as a Pair / as a Mob)
  • Every day not being the same for developers for a Single Branch
  • Fixing code in Release Branch / Hot-fix Branch directly
  • Merging fix instead cherry-picking
  • Merge to master within less the 2 days but not ensuring it does not break the code base (only because of the urge)

More info, please visit https://trunkbaseddevelopment.com/youre-doing-it-wrong/

What would happen if we do TBD
(Why we would love it)

  • You’ll Integrate Code Continuously (at least each 2 days)
  • You’ll Have Eager Code review and Continuous Code Review
    • Small reviews ensure it to make it possible
    • Dev Team Member is forced to do it efficiently and get used to it
  • You’ll Release Application Changes Consecutively
    • Code base (trunk / master) is (near) always releasable and has always something new to release (at least each 2 days)
  • You’ll Gradually Introduce New Features
    • Product Owner, stakeholders will be happy instead of living in anxiety
    • Devevelopers will be proud of their productivity
    • Since the Product Owner and Developers are happy & productive, Scrum Master will be also proud and happy

Read more: https://rollout.io/blog/trunk-based-development-what-why/

Some Issues in Adopting TBD

  • Implementation should not be released to public but deployment should happen
  • A feature to implement are much more longer than 2 days
  • Implementations are finished not in the same time & broken code shouldn’t jeopardize the code base / trunk
  • Not sure how to scope the feature branch

Those issues above often occur and remain unsolved by developers causing skepticism to TBD because the developers do not knowing the HOW. In next section we will talk about the HOW.

TBD: The HOW

  • Problem: Implementation should not be released to public but deployment should happen
    • A Solution: Concept of Feature Toggles. Separating publishing from deployment
  • Problem: Features to implement are much more longer than 2 days. Implementations are finished not in the same time & broken code shouldn’t jeopardize the code base / trunk
    • A Solution: Concept of Branch by Abstraction. Have an abstraction around code to be able to switch code between new implementation and legacy code, so new merged code doesn’t disrupt existing code
  • Problem: Not sure how to scope the feature branch
    • A Solution: Split big implementation to smaller chunk using black-boxing concept, e.g. Flow Based Programming. Split bigger tasks to smaller sequence of task list

The How: Feature Toggle

How we think about feature toggles (the prejudice): Feature toggle is complicated ◦It takes a lot of effort.

Some perspective which would let us rethink: There are many kinds of feature toggles: https://martinfowler.com/articles/feature-toggles.html

  • static toggles vs dynamic toggles
  • Long-lived toggles vs transient toggles
    • Why put to much effort to transient toggles, if after a few days, or 1 or 2 weeks you remove it anyway

Not all toggling-off has an effort, some are naturally off: A new class of a function which is complete & tested as a new feature but not yet called implies it is off. A new API which is not yet exposed to the API gateway, implies that the API is still off from outside.

From: https://martinfowler.com/articles/feature-toggles.html

The How: Branch by Abstraction

Branch by Abstraction is:= creating abstraction to keep legacy code and new code switchable, so development does not jeopardize integrated trunk code.

Steps citated from (https://trunkbaseddevelopment.com/branch-by-abstraction/):

  1. Introduce an abstraction around the code that is to be replaced, and commit that for all to see. If needed, this can take multiple commits. None of those are allowed to break the build, and all of them could be pushed to the shared repository in order, and as done.
  2. Write a second implementation of the abstraction for the to-be-introduced code, and commit that, but maybe as ‘turned off’ within the trunk so that other developers are not depending on it yet. If needed, this can take multiple commits as above. The abstraction from #1 may also be occasionally tweaked, but must follow the same rule – do not break the build.
  3. Flip the software ‘off’ switch to ‘on’ for the rest of the team, and commit/push that.
  4. Remove the to-be-replaced implementation
  5. Remove the abstraction
From: https://trunkbaseddevelopment.com/branch-by-abstraction/

The How: Split to Smaller Black-boxes

A bigger tasks can be split to smaller independent tasks

Steps:

  • Instead of flow with the work, plan our work
  • Create a checklist and sequences
  • Create black-boxes
    • A black-box mathematically can be seen as a function with input and output
    • Understand & learn Flow Based Programming
    • Apply SOLID principle
      • 1 Responsibility Classes (small decoupled classes),
        Interfacings, Dependency Injections
    • In case collaboration needed, is useful to define contracts, so subtasks can be distributed among team members =>
      • More people could work for a same goal
      • Value delivered faster (which support agile including scrum), than everyone works in separate pipelines (which is a scrum antipattern)
      • Teams has visibilities of what to be done and have been done

Black-boxing Example: Listing & Sequencing

This is an example of breaking a branch, instead of having 1 single big branch named similarity address validation to have near 20 planned branches / activities:

  1. elastic search microservice setup
  2. setup of elastic search
  3. integration of elastic search to microservice
  4. create single indexing API (Create)
  5. create batch indexing API (Initialize)
  6. create database export script
  7. create search API (Read)
  8. create update or deletion API (Update & Delete)
  9. create address similarity logic
  10. integrate address similarity logic with search API create approval rejection logic
  11. integrate address similarity to approval and rejection validation
  12. create approval and rejection logger
  13. integrate logging for approval and rejection
  14. integrate indexing of address in submission
  15. integrate update in policy status synchronization
  16. create index update logger
  17. integrate index update logger to submission integrate index update logger to policy status synchronization

Black-boxing Example: Dependency Analysis

Create a plan of dependency analysis to optimize collaboration.

An example of dependency / sequence analysis diagram:

  • The green ones are the independent sub tasks
  • The yellow ones are dependent from the box before
  • Once a green box is done, next green box could be defined so more independent sub-tasks can be redistributed
  • We can choose which tasks based on dependency to remove so called critical paths problems

Black-boxing Example: Flow Based Programming Plan

By having those boxes in the previous section you can create a blackbox function to classes in code.

  • Black-boxing to classes (Single Responsibility)
    • Don’t touch existing class but create a new one for a new behavior
    • Inject with dependency injection to avoid complexity & ensure decoupling
  • Analyze Input Parameters and Output Returns
  • Use DTO concept and function concept to define data contract and functional contract
  • Use transformer concept to translate outputs to inputs for integration

By having those separation developers can collaborate together for a same goal but in separated branches and decoupled codes.

The How: Assess Refactoring Effort instead Avoiding It

Another rejection comes often out from developers are refactoring effort. For this, take into account that even in a very good plan, refactoring might happen / necessary.

Do NOT try to exclude the effort of refactoring, but assess, and accept it as long as refactoring is supporting that value is to be delivered faster (more productive, economic, and beneficial).

Even TDD include refactoring in the process steps (Cycle of: Create Test, Implement, Refactor).

Adopt solid principle so that refactoring effort will be reduced and code will be easier decouplable.

Spare our time to plan input output contract before continue development, so simplicity will be more visible across team members and refactoring effort will be reduced.

Summary

  • Strive to short-lived feature branch which is by one person / pair / mob (NOT parallel persons)
  • Single code base named trunk / master as our focus (NOT feature branches)
  • Fix master first before release branch / hotfix branch so we won’t forget
  • Always consider feature toggle so code base is always releasable, and supporting legacy until it is really ready
  • Create abstraction of branches to support legacy until things are stabilized
  • Break down a big branch to black-boxes
  • Create checklist and sequences
  • By necessity define dependencies so we can distribute subtasks so team can work for a same goal instead separate pipelines so values can be delivered faster
  • For unclear contract, deep dive by creating black-boxes plan to define inputs & outputs and relation
  • Consider that refactoring is NOT something to avoid BUT to assess (how much and how to optimize)
  • Use SOLID principle for this
  • Use black-boxing and transformer concept of Flow Based Programming
  • TBD is more than Branching Model, but also Mindsets

Something to confuse us, so we reflect ourselves

  • We are often trapped in wordings and terms => Be careful so we shouldn’t be trapped
  • Do we abandon Feature Based Development?
  • Is Trunk Based Development NOT Feature Based Development?
  • Featured Based Development teaches us how to split requirement to features
  • Trunk Based Development teaches us how to keep the branches healthy
  • We can work with TBD and Short-Lived Feature Branches which are features but max of each just lives for 2 days, so we still use concept of features but in healthy way and throw away the unhealthy
  • Martin Fowler wrote this article: https://martinfowler.com/articles/branching-patterns.html
    • In my opinion, he seems doesn’t care which policy you follow, either git-flow or TBD or both, but he seems to follow short-lived branch concept and has the focus of single based code he named mainline to keep healthy branches and healthy development. He even has given an own name of the concept for it, importantly to support Continuous Integration and Continuous Delivery and Productive Development

My Wish & Hope

  • Now we know about the importance of Healthy Branching Model where Trunk Based Development could support it
  • We know what is healthy what is not
  • “Trunk-based development is a substantial change for many developers, and you should expect some resistance. Many developers simply can’t imagine working in this way. A good practice is to find developers who have worked in this way, and have them coach other developers.”
  • I hope this article could reduce misconceptions of TBD, so that we have the correct understanding of the concept
  • Hopefully we are amazed about some hidden concepts of Why and How, and start now with Believe and Effort instead of Prejudice and Disbelieve
  • There will be a process instead of from today to tomorrow so we should strive ourselves, be solution oriented, and help each others to solve the issues together for improved productivity for our team’s success

Trunk Based Development Cheatsheet

Other References