Blog | Mani Soundararajan

Visualizing Software Architecture using C4 Model and Python Diagrams

Wed, 19 Jun 2024 00:00:00 +0000

C4 Model

The C4 Model is a great way to explain software architecture using diagrams.

Instead of throwing the developer into a confusing jumble of boxes and lines, the C4 Model breaks down the architecture starting from the top layer down to the lowest code layer. The C4 model gets it’s name from the 4 C’s:

Context - explain at high level what the software does
Container - explain how various components come together in the software
Component - explain each component in detail
Code - explain the code implementation

For a detailed explanation, look no further than the official C4 Model website.

Diagramming tools

There are several options when it comes to creating diagrams.

One of the popular options is to create the diagrams in draw.io. This tool allows you to export your diagram as an image (say, a PNG) as well as download the diagram in a .drawio file format - which you can add to your git repository.

Having the .drawio file ensures that it is possible to make modifications in the future.

My preferred way to create diagrams is to write a python script using diagrams module.

Be sure to checkout the documentation and examples of the digrams module. It has support for tons of cloud native tools. More specifically, they also have support for the C4 Model.

Example

To see how to create architecture diagrams using diagrams, let’s take the case of hypothetical app called Lacerta. Lacerta is used by users to track their safety - by having users regularly “check-in” to confirm that they are safe. If they miss a check-in, their emergency contact person is informed - who will get in touch with the user and take further action as needed.

Context

At the highest level, we have the System Context Diagram:

This diagram was generated by the python code:

from diagrams import Diagram
from diagrams.c4 import Person, Relationship, System
from diagrams import Node

def main():
 graph_attr = {
 "splines": "curves",
 }
 with Diagram("Context diagram for Lacerta", filename="context", direction="TB", graph_attr=graph_attr):

 user = Person("User", description="A user of Lacerta")
 emerg = Person("Emergency Contact", "A friend or family member of the User")
 sys = System("Lacerta System", description="")

 user >> Relationship("1. configures emergency contacts, and periodically checks-in") >> sys
 user << Relationship("2. reminds if a check-in is missed") << sys
 emerg << Relationship("3. alerts if user has missed a check-in despite reminders") << sys
 user << Relationship("4. manually confirms that the User is ok") << emerg

if __name__ == "__main__":
 main()

Container

At the second level, the container diagram explains how various components work together internally as well as with external systems. Here is the container diagram for Lacerta:

This diagram was generated by this Python code:

from diagrams import Diagram
from diagrams.c4 import Container, Database, Person, Relationship, SystemBoundary, System
from diagrams import Node

def main():
 graph_attr = {
 "splines": "curves",
 }
 with Diagram("Container diagram for Lacerta System", filename="container", direction="TB", graph_attr=graph_attr):
 user = Person("User", description="A user of Lacerta")
 emerg = Person("Emergency Contact", "A friend or family member of the User")
 with SystemBoundary("Lacerta application"):
 mobile_app = Container("Mobile App",
 technology="Flutter",
 description="Provides ability for user to check-in periodically, and also to configure emergency contacts")
 backend = Container("Backend",
 technology="Go",
 description="Backend API application allows user to check-in, and configure emergency contacts")
 daemon = Container("Daemon",
 technology="Go",
 description="Constantly checks if any user has missed a check-in")
 db = Database("Database",
 technology="MongoDB",
 description="Database to store configuration and check-in times")

 email = System("Email", description="SendGrid Transactional Email Service", external=True)
 push = System("Push Notification", description="Firebase Cloud Messaging", external=True)

 user >> Relationship("checks-in and configures emergency contacts using") >> mobile_app
 mobile_app >> Relationship("make API calls to") >> backend
 backend >> Relationship("reads from and writes to", reverse=True) >> db
 daemon << Relationship("reads from") << db
 daemon >> Relationship("triggers email reminders or email alerts") >> email
 daemon >> Relationship("triggers push notification reminders to check-in") >> push
 user << Relationship("reminder emails to check-in") << email
 user << Relationship("reminder messages to check-in") << push
 emerg << Relationship("alert email about user") << email
 user << Relationship("manual check-in to verify if user is ok") << emerg

if __name__ == "__main__":
 main()

If you look at the code, it does not look like typical imperative code written in Python, this reads more declarative - in terms of defining boxes, the lines that connect the boxes, and the direction of the arrow.

Component

I do not create diagrams at the Component level. You can, if you wish create diagrams for each component. As you get lower down the layers, you may find yourself changing the diagrams a lot during heavy development periods. So creating a diagram for the Component layer may or may not make sense for your team.

Code

You shoud almost never create diagrams for the Code layer - as we expect the code to change a lot over the lifecycle of the software. We can strive to writing clean code so its easier to read the code - or use automated tools to parse the code and generate diagrams.

Reasoning

I find that using GUI based diagramming tools gives me way too much choice - do I want rounded corners for the rectangles, or should they be sharp corners. What background color should I use for a database box, and which of the ten different arrow styles to use. It is hard to stay consistent when creating many diagrams. With code based diagramming tools, a lot of that choice is taken away from me - so I can focus on which component interacts with which component, and in what way.
Creating diagrams using code allows me to commit both the diagram, and the source code into a git repository - bringing it under version control. If a question arises in the future about when a certain architectural change was introduced, we can easily figure this out using git log and git blame. (See also: Architecture Decision Records)

References

Guess Which Door Has A Car Behind It

Thu, 13 Jun 2024 00:00:00 +0000

There are three doors. Behind one of the doors is a car, and behind the other two are goats. Can you beat the odds, guess the correct door and win the car?

This problem is popularly known as the Monty Hall problem - named after the TV show host who ran Let’s Make a Deal back in the 60s and 70s. It goes like this: Monty asks you to pick a door. Let’s say you pick Door 1. Then Monty goes behind each door and sees where the car is, and where the goats are. He then proceeds to open, let’s say, Door 3 and reveals a goat. Monty now offers you a choice - do you want to STAY with Door 1, or do you want to SWITCH to Door 2.

Think about this for a second. And think in terms of probability - not in terms of I’m feeling lucky about Door 1 or Door 2 etc. What would you do?

I came across this problem when I went into a rabbit hole after watching a YouTube clip of Kevin Spacey (playing a professor) quizzing his class about this exact problem.

Here was my thinking: Now that Door 3 is eliminated, it’s a 50/50 toss up between Door 1 and Door 2. So it does not really matter if I stay or switch. I have no statistical advantage in doing either. In fact, I think I was lucky in not picking Door 3, and that I should let my lucky run continue and not mess things up - so I am going to STAY with Door 1. Turns out, this is flawed thinking! And it’s not just me, some really good mathematicians also argued that the odds are 50/50 between Door 1 and Door 2.

But first, I want you to try this for yourself. Below is a simulation of the Monty Hall problem. I want you to play this with two different strategies. First, I want you to play the STAY strategy. With this, you will pick a door, and when the goat is revealed, you will STAY with the same door you first picked. Playing this is easy, just keep clicking on the same door - again and again. At the end of the game, click anywhere to restart. To truly see the statistics play out, you need to play this at least 30 times. Next I want you to play the SWITCH strategy. For this, you will pick any door, and when the goat is revealed, you will SWITCH to the other remaining door. Again, play this about 30 times. Please indulge me, you need to see this for yourself to believe it:

🚪

Pick a door.

	Win	Lose	Win %
Stay	0	0	0%
Switch	0	0	0%

If you played this a lot of times, like 30 times for each strategy, you will most likely see about a 33% win rate for the STAY strategy, and about a 67% win rate for the SWITCH strategy. Shouldn’t it be 50/50 between the two strategies? What is happening?

To get our heads around this problem, let’s remove “you” from the equation and replace “you” with “Alice”. Alice is the player and she will be the one choosing the doors. For every choice Alice makes, there is the choice she did not make - the philosophical “road not taken” - let’s personify this as Bob. Let’s say that Bob is also a player, and he does NOT get to pick a door - he simply gets the doors that Alice does not pick.

So, if Alice picks Door 1, then Bob gets Door 2 and Door 3. At this point, Alice has a 33% probability of winning a car, and Bob has a 67% probability.

So, initially, when Alice picks Door 1, the probability of her winning is 33% and the probability of Bob (with Door 2 and Door 3) winning is 67%. Let’s call this STAGE 1. Then a goat is revealed in Door 3, and Alice is offered a choice to STAY with Door 1 or SWITCH to Door 2, let’s call that STAGE 2. Note that between STAGE 1 and STAGE 2, Monty did not SHUFFLE the car and goat behind Door 1 and Door 2. During the entire game, once the car is placed behind a Door - it does not move. As long as there is no SHUFFLING, then Bob’s odds of 67% does not change either.

But why doesn’t Bob’s probability go down from 67% to 50%? Because no new information has been revealed that changes that fact. There is still only ONE car, and there are three doors. Bob has 2 doors (doesn’t matter if it’s open or closed). So Bob’s probability stays at 67%. When Monty reveals the goat in Door 3, he is playing a trick. He is telling the audience, “Look! Bob has a goat behind one of his doors” - which really should surprise no one. Bob either has a car and a goat, or two goats. Saying “Bob has one goat” does not in any way change Bob’s odds of winning. The fact that Door 3 has a goat has strengthened Bob’s odds of having a car behind Door 2 - because keep in mind, Door 2 + Door 3 is 67%. So if Door 3 is a bust, then all that 67% probability goes to Door 2.

In other words, asking Alice if she wants to SWITCH to Door 2 is the same as asking if she wants to trade places with Bob, and his 67% probability of winning a car. If we put it that way, then we see that it makes sense to SWITCH.

Lastly, I want to go back to your original intuition of thinking the choice is 50/50 because there are only two doors left. That is true ONLY IF there is SHUFFLING of car and goat after STAGE 1. If there is SHUFFLING, then all bets are off. At that point, it does not make any statistical difference if you STAY or SWITCH.

Note that if you are playing the SWITCH strategy, you may still lose - that is why the odds are 67% and not 100%. No such thing as a free car, eh?

Hey, if you’ve made it this far, I have two favors to ask of you:

If you are a programmer, please go ahead and use your Github account to add reactions or comments below.
If you liked this post, please share this with others!

References

Note to programmers: When I was working on this simulation game, I wanted to make sure it was fair. Which means, if you play the game 30 times, irrespective of whether you win or not, the car should be behind Door 1 approximately 10 times, and behind Door 2 approximately ten times, and behind Door 3 approximately ten times. That is what you would expect in a pure random distribution. When I used Math.random() I found that the results were wildly skewed. So, for my game I switched to crypto.getRandomValue(). But that is not perfect either - because computers don’t really do random - they only do pseudorandom. Even then, my implementation has an “unfairness” to it. Read the function getRandomIndex() in my source code, and tell me in comments below if you can spot what is unfair about it!

Developing Software For 25 Years - A Retrospective

Sat, 01 Jun 2024 00:00:00 +0000

This week I am completing 25 years of my software development career. I figured this major milestone deserves a retrospective. Here are some of the lessons I have learned over the years:

1. Free and Open Source Software makes the world go around

Without going into details of copyleft licenses (like GPL) or permissive licenses (like BSD, MIT), I will say that free and open source software (FOSS) runs a lot of things on this planet. FOSS powers a good part of the internet. Governments around the world are switching to FOSS. Heck, there’s FOSS running off-planet as well. Much of the tech boom we have seen in the last two decades was made possible because we are standing on the shoulders of FOSS giants.

Over the last 17 years, I have made several contributions in my own small way. I also got an opportunity to speak about my cool (at that time) 5000+ docker container production deployment at OpenCamps 2017 held at the United Nations in New York.

At this 25th year of my tech career, I am more confident than ever about the future of FOSS, and I am doubling my resolve to spend more of my time using FOSS, promoting FOSS and contributing to FOSS.

2. Reading is a superpower

In this era of attention attrition, I have found that my ability to read is a superpower. I am able to answer so many of my colleagues’ questions simply because I have read the documentation cover-to-cover. Other times, I enlarge my knowledge base by NOT going to stackoverflow or reddit seeking a quick answer, but by reading official documentation, API reference and even upstream source code.

Reading has several obvious advantages to watching a video. You can usually skim a page (or even screenful) of information faster than it takes for you to watch a video about the same topic. Also, reading has “random access” capabilities. You can skip a few pages or go back. With video, cueing forward or reverse does not have the same quality of experience.

This superpower extends to reading in general. To parents of young children, I will say this: children learn by observing you more than by listening to you. So if you want your kids to read, you should be seen reading books, preferably the print kind.

3. Saying Yes to more things leads to interesting experiences

I have worked more than 20 years in startups. A startup career does not come with fixed job titles or responsibilities. Over the years, anytime I got the opportunity to step outside my comfort zone, I have said yes - and every time I have come away gaining a lot of experience.

I said yes to working on embedded firmware, and learned that I can apply my skills to solve any problem - including finding a runaway pointer in a ring buffer that caused memory overwrites and sleepless nights. I said yes to working on a Radio Frequency communication project, and learned that the random packet interference I was receiving was just folks opening their cars with their key fobs. I said yes to working customer support and learned that my customers were using my product in ways I did not intend or even imagine.

4. Empathy is the missing piece in most software stacks

I have been asked many times about what I look for when hiring someone. I always say Empathy and Motivation. Our world (and our technologies) will be a whole lot better with some empathy.

Be more empathetic to your customers - they are the ones putting the food on your table. Be more empathetic to your colleagues - they are the ones sharing your responsibilities. Be more empathetic to your frontline customer support - they are the ones facing angry customers when you break production. And lastly be more empathetic to future developers who will be inheriting your code base.

5. Doing customer support improves empathy

Some of the most insightful learnings about my product have come from doing customer support.

Customers have this uncanny ability to bend your product to their will to achieve their objectives. If you are not paying attention to this, sooner or later you will make a change in your product that breaks the customer’s workflow. Remember, backward compatibility is not just for your published API - but for all of your software’s behaviors that the customer is using in their workflow.

When you can, try to spend a rotation doing customer support.

6. Find your rhythm for doing boring repetitive tasks

We have incredibly interesting jobs - but that comes with sometimes having to do boring repetitive tasks. This is work that is not appealing to do, but needs to be done nonetheless. Sometimes it might make sense to spend the time automating or scripting a solution - but most times it’ll make sense to grind through the chore.

I have learned that it is important to find my rhythm when doing such tasks. If I go too slow, I will zone out and lose focus. If I go too fast, I will end up making mistakes and spend even more time cleaning up my mess. Things I have found helpful are making checklists and checking items off at a steady pace.

Like the US Navy Seals say: “Slow is smooth, smooth is fast”.

7. Figure out how your brain works

One of the first lessons we learn as students in a laboratory course is called “study of tools”. If this is an electronics lab course, you will learn about voltmeters, ammeters etc. If this is a carpentry shop course, you will learn about t-squares, various kinds of saws and so on.

What are the tools of our trade? Keyboard and mouse? Sure we all know how to use them. Programming languages, IDEs, compiler toolchains? Yes, we all know that too. But we don’t really think of our brain as being part of this - and yet it is the most important tool in our toolkit.

As software developers, we owe it to ourselves to try and understand our brains a little better. How does my brain perform after a carb-heavy meal (not good), and how does it perform after going for a run (much better).

That is why I find topics like Cognitive Load fascinating. That is the reason why I ask my team to keep pull requests to less than 200 lines of diff so that the code reviewer does not suffer cognitive overload (see point #4 above regarding empathy towards colleagues).

8. Sleep is an important step in debugging

There is no bug you can’t tackle after you’ve had a good night’s sleep.

Again, to understand this, you will need to understand how your brain works. The more awake you are, the more toxins build up in your brain - and the only way to flush them out of your system is to sleep. And it’s not like you have stopped debugging the problem you were working on. In fact, the work continues in your subconsciousness. The NREM cycle of your sleep moves information from temporary memory (hypothalamus) to permanent memory (cerebrum), and the REM cycle creates links between neurons representing new information and neurons representing information you already had in your brain. This “linking” process is what creativity is - and this is going to help you come up with creative ways of debugging the problem.

I would highly recommend Dr. Matthew Walker’s book “Why We Sleep”. And I would also recommend getting something like a fitbit and wearing it while sleeping so you can start seeing patterns of your mood / performance vs. your sleep quality.

So, if you are not able to fix a bug after working on it for several hours - call it a day and go to sleep. You will wake up with fresh perspectives on how to approach your problem. Now, if you have figured out how to account for that in your timesheet, please let me know!

9. If you don’t like your organization’s culture, change it

We all like to say work culture is a really important factor when considering joining an organization. But what is good work culture? For a really good definition of what makes a good work culture - look no further than the book Accelerate by Dr. Nicole Forsgren, Jez Humble and Gene Kim - skip to Chapter 3 to read about this topic - or better yet read the entire book. To paraphrase the authors quickly, there are “pathological workplaces”, “bureaucratic workplaces” and then there is “generative workplace” - which is what we all think of as “good work culture”.

Setting the organization’s culture is not something that is the sole domain of the manager - everyone can and should work towards improving the work culture. If you work in a small company, there is no reason why you shouldn’t be able to implement changes. Even if you work in a large organization, you can bring about changes in your team.

10. In software ethics, you are the last line of defense

Technology has taken over much of the world - and I have seen this just in my lifetime. Software runs pretty much everything - and as software developers we have great power. We can either use this power to build features that will benefit our customers, or we can abuse this power to build anti-features that are not in the best interest of our customers.

Without going into specifics, I can say that I have, in the past, pushed back on dubious feature requests - and the management in each case agreed to drop the feature.

I am here to say today to you - that you have the agency to refuse to work on anti-features. This is not something you have to earn with seniority - you get this on day one.

Remember what uncle Ben said: “With great power comes great responsibility”.

Version bumping with Uplift

Thu, 23 May 2024 00:00:00 +0000

Making a software release involves a bunch of boring routine tasks - with Uplift you can automate all of it.

Uplift

With Uplift, you can do three things at once:

Bump the version number
Update your changelog
Tag and push

And all it takes is to run:

$ uplift release

Uplift installation instructions

Conventional commits

Before you can leverage Uplift, you should ensure all your git commit messages are following the Official Conventional Commits standard. You can also refer to my post on conventional commits.

When you write commit messages like feat: add payment module or fix: update payment flow, Uplift is able to parse those messages and determine whether the number needs to be bumped at major, minor or patch version.

If all your changes are fix type commits, Uplift will bump the patch version.

If one your changes is a feat type commit, Uplift will bump the minor version.

If any of your commits have breaking change, like fix! or feat!, Uplift will bump the major version.

Other than this, Uplift is also able to parse commit messages and come up with a Changelog for the current release. This will be automatically updated in a Changelog.md file.

Configuration

Uplift expects the configuration to be stored in a file like .uplift.yaml (there are some other file name patterns that are supported). This file will contain configuration related to which file version needs to be bumped and so on.

Let’s look at an example:

# uplift.yaml
# see https://upliftci.dev/
bumps:
 - file: main.go
 regex:
 - pattern: 'const appVersion = "$VERSION"'
 semver: true
annotatedTags: true
changelog:
 exclude:
 - "chore"
 - "refactor"
 - "style"
 - "test"
 - "wip"
commitMessage: "chore(release): $VERSION"

This file instructs Uplift to bump the version in main.go in a line that matches the regex pattern const appVersion = "$VERSION". For example:

package main

import "log"

const appVersion = "v0.1.0"

func main() {
 log.Println("Helloworld", appVersion)
}

To this, you add a bug fix and change Helloworld into two words Hello world. Make sure the commit messages says something like fix: split helloworld into two words. Now if you run uplift release, you would get:

package main

import "log"

const appVersion = "v0.1.1"

func main() {
 log.Println("Helloworld", appVersion)
}

Uplift has bumped the patch version. It has also added a section to the Changelog.md file:

## [v0.1.1]

- fix: split helloworld into two words

And lastly, Uplift has created a git tag v0.1.1 and pushed it to the git remote origin.

Demo

I have created a repo to demonstrate conventional commits and uplift:

Github repo

Commit created by uplift

Tag created by uplift

References

Managing context switches

Sat, 11 May 2024 00:00:00 +0000

There are some days when I am able to turn off my slack and get a blissful few hours of solid productivity. But then there are other days filled with knock-knock-brush sounds from slack, meetings, and other interruptions that bog me down with constant context switches.

Congnitive load of interruptions

Let’s say you are breezing along on your task when suddenly you are interrupted by a team member requesting you to do something else. Now you pause your task, and go about helping your team mate. When that is done, you go back to your original task - at which point, you are probably asking yourself “Wait, what was I doing?”. Every time you change your focus from one task to another, it involves a context switch. Your brain has to start loading a whole bunch of related memories in order to start working (or resume working) on a task. This is one example of cognitive load.

A lot has been written about the cognitive load of context switches. Studies have shown it can take up to 23 minutes to fully recover your flow after an interruption.

Enter TODO+

TODO+ is an extension for VS Code that allows you to manage TODO lists, and store them in a file named TODO as part of your project repo.

When you are setting out on a task, say a pull request for an issue, start by breaking it down into smaller tasks and put them in your TODO file.

Pro-tip: Always keep the TODO file open in a right side pane, so you can quickly glance at your current task.

If you have to navigate to the TODO file and open it, you are already switching context from your currently open file to the TODO file.

If you can’toggle-start-darkt get the checkbox symbol to show up on your task, you can either hit Alt+Enter, (or right click and choose “Toggle Box” from the context menu).

Before starting a task, click on your first task and hit Alt+S (or Toggle Start from the context menu).

If you are interrupted and come back to your VS Code, a quick glance at your TODO file should get you back to speed quickly.

When you are done with your task, select the line in your TODO file and hit Alt+D (or Toggle Done from context menu). I would also recommend commiting all the changes at this point, including the changes to your TODO file. Personally, I like to call these “work in progress” commits, or “wip:” to follow conventional commits.

When all tasks are done, you can hit Ctrl+Shift+A to move all done tasks to the archived section in your TODO file. Again, be sure to commit the TODO file with your code changes for your main feat: or fix: commit at this point.

Easy Solution

The easy solution to all this would be to reduce interruptions in the first place. But it’s easier said than done. Some companies are adopting a “no meeting Thursdays” policy. Is your team suffering under a lot of context switches? What have you tried, and what has worked / not worked for you?

Architecture Decision Records

Sat, 24 Feb 2024 00:00:00 +0000

Software Development is part science and part art. Not unlike making a movie. But a software developer’s role is very different from a movie director’s in one important way: When a director makes the final cut and releases the movie to the audience, the movie is in its finished state. The director will have to live with whatever goofs slipped through during post production. A software, on the other hand, is never done. If a bug slipped through in this version, you can always fix it in the next version.

Which brings us to an interesting point: The developer that started working on a software may or may not be the same person that releases it into the wild. A lot of software projects go through a hand-off process, sometimes even multiple times. And anyone who has ever seen children play telephone has seen that every hand-off diminishes the quality of the message. This leads to a situation (more often than not) of a software team wondering why the original developers decided to do things in a certain way.

Enter Architecture Decision Records.

The goal of Architecture Decision Records (ADRs) is to capture the thinking process that went into making significant decisions regarding the architecture of the software. The idea was made popular by Michael Nygard’s blog post.

ADRs can be used to capture a technical problem, the various options considered, pros/cons of each option, and the final decision made along with the the reasoning.

It goes without saying that the developers making the ADR should be open and honest about their thought process - just like it is important for future teams reading the ADRs not to judge prior team’s decisions - but to understand the context and assumptions under which the decisions were made - and to be thankful for the prior team’s efforts in recording the ADRs.

Over the last few years, several templates have been proposed for capturing ADRs.

The one that I have settled is on called Markdown Any Decision Record or MADR for short.

MADR uses markdown syntax - something most developers are already be aware of. MADR has simple and easy to understand sections to capture the problem statement, decision drivers, various options considered, final decision - its consequences and so on.

MADR also recommends storing the ADRs alongside your project files (in a folder called /docs/decisions) and commit them to your git repo. Filenames are suggested to be of the format nnnn-title.md where nnnn is a running sequence of numbers starting from 0001. MADR dogfoods their own recommendation - so you can find their internal ADRs in their github repo adr/madr.

The one thing you want to be careful about, is when you change a decision - do not edit an ADR - instead mark it as superseded and link it to a new ADR.

If you are unsure about ADRs, just start using them. You can always refine your process. Remember, we are making software, not movies.

Versioning Go Applications

Sat, 18 Nov 2023 00:00:00 +0000

Inject version information in Go applications using linker flags.

Overview

One of the must haves when it comes to running a Go application in production is to have a clear version number printed in the logs at the start of the application. This is helpful in the future when debugging a bug by looking at the logs, and seeing which version of the app caused this behavior.

Get Version

The version information can be stored as global variables with zero values, with the values being injected at build time by the linker.

Here is an example:

package main

import "log"

var APP_VERSION string
var APP_PRERELEASE string
var APP_BUILD string

func main() {
 log.Println("Hello world", getVersionString())
}

func getVersionString() string {
 version := ""

 if len(APP_VERSION) > 0 {
 version = APP_VERSION
 } else {
 version = "1.0.0"
 }

 if len(APP_PRERELEASE) > 0 {
 version = version + "-" + APP_PRERELEASE
 } else {
 version = version + "-local.1"
 }

 if len(APP_BUILD) > 0 {
 version = version + "+" + APP_BUILD
 }

 return version
}

Notice that the variables APP_VERSION, APP_PRERELEASE and APP_BUILD are declared at the global scope, but left at the zero value ("").

If you run this as is, you will see the following output:

$ go run main.go
2023/11/17 18:13:27 Hello world 1.0.0-local.1

Linker flags

As the next step, we will be injecting these variables at build time using linker flags:

$ go mod init github.com/msound/helloworld
$ go build -o helloworld -ldflags "-X 'main.APP_VERSION=1.0.0' -X 'main.APP_PRERELEASE=stage.a862776' -X 'main.APP_BUILD=build.422'"

This would build a binary named helloworld which would have the version information baked into the binary. Now if we run the binary:

$ ./helloworld
2023/11/17 18:20:57 Hello world 1.0.0-stage.a862776+build.422

Dockerfile

You are probably going to run your Go app as a container, so you would want to pass these values through as arguments into a Dockerfile. Here’s how you would do it:

FROM golang:1.21 AS build-env

ENV GOOS=linux GOARCH=amd64 CGO_ENABLED=0

ARG APP_VERSION
ARG APP_PRERELEASE
ARG APP_BUILD

WORKDIR /app

COPY . /app

RUN go build \
 -o helloworld \
 -ldflags "-X 'main.APP_VERSION=${APP_VERSION}' -X 'main.APP_PRERELEASE=${APP_PRERELEASE}' -X 'main.APP_BUILD=${APP_BUILD}'"


###########################

FROM alpine:3.11

RUN apk update && \
 apk add --no-cache --purge ca-certificates curl tzdata && \
 rm -rf /var/cache/apk/* /tmp/*

RUN adduser -D appuser --uid 3000

COPY --from=build-env /app/helloworld /app/helloworld

USER appuser

CMD ["/app/helloworld"]

The Dockerfile receives the version information as ARGs, and passes them on to go build as linker arguments.

CI/CD Pipeline

The final step is to pass the arguments to docker build from the CI/CD Pipeline. Here is the snippet of this for Gitlab. For Github actions, you can adapt this as needed:

script:
 - |
 export APP_VERSION=${CI_COMMIT_TAG}
 export APP_PRERELEASE=stage.${CI_COMMIT_SHORT_SHA}
 export APP_BUILD=build.${CI_JOB_ID}
 docker build \
 --build-arg "APP_VERSION=${APP_VERSION}" \
 --build-arg "APP_PRERELEASE=${APP_PRERELEASE}" \
 --build-arg "APP_BUILD=${APP_BUILD}" \
 -t helloworld:${CI_COMMIT_TAG} .

References and Credits

Conventional Commits

Fri, 06 Oct 2023 00:00:00 +0000

Conventional Commits is a widely adpoted standard way of writing a commit message.

Overview

Before proceeding further, please peruse the full spec from the Conventional Commits offical website.

The short summary of a conventional commit message is:

<type>[(scope)][!]: <description>
<blank line>
[body]
<blank line>
[footer(s)]

Types

The standard recommends the use of the following types:

build: Changes that affect the build system or external dependencies
chore: A code change that is neither a feat or a fix (example: bumping versions of dependencies)
ci: Changes to CI configuration files and scripts
docs: Documentation only changes
feat: A new feature
fix: A bug fix
perf: A code change that improves performance
refactor: A code change that neither fixes a bug nor adds a feature
revert: A commit that reverts a previous commit
style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
test: Adding missing tests or correcting existing tests
wip: Indicates this commit is a Work-In-Progress

Scopes

Every repo should have a list of scopes clearly defined in the repo’s README file. Commit message may use a scope to clarify what part of the codebase this commit touches. For multiple scopes, use a comma separation.

Breaking Change

If the commit introduces a breaking change, it must be indicated using an exclamation (!) after the type/scope and before the colon (:).

Description

The description is a short summary of the code changes. The description shall be written in imperative. The first line, including type, scope and description shall not exceed 50 characters. TIP: If you set your git config core.editor=vim, it will automatically highlight when you are going over 50 characters. Description shall start with a lower case.

Here are some correct and wrong examples for description:

✅ add foo to main menu

❌ adding foo to main menu

❌ added foo to main menu

A good way to think about writing in imperative, is to try to fill in the blank of the sentence:

If merged, this commit will _________________________.

What you write in the blank will usually be in imperative.

Body

A commit body is free-form and may consist of any number of newline separated paragraphs. Try to add more context about the commit focussing on WHY this change was introduced rather than WHAT code was changed. Most developers can see the commit diff and easily figure out the WHAT part - answering the WHY question adds value to the commit message. Remember, you yourself may be wondering about WHY you made this change on your own commit a few months down the road.

One or more footers may be provided one blank line after the body. Each footer must consist of a word token, followed by a : separator, followed by a string value. The token must not have spaces, instead it must use hyphen(-) to separate words.

This is an non-exhaustive example list of possible footers:

BREAKING-CHANGE: This commit will break the code for database records in the old schema.

Closes: JIRA-123

Refer: JIRA-456

Reviewed-by: Bob bob@example.com

Signed-off-by: Alice alice@example.com

Why

Conventional commits standardize and make it easy to read commit messages. It also helps in writing tools to parse commit messages are automatically generate changelogs, or suggest semantic version number updates based on the presence of fix (patch version is incremented) or feat (minor version is incremented) etc. Tools can also parse the token: value syntax of the footers to process information contained therein.

References and Credits

Managing secrets using sops/age

Fri, 26 May 2023 00:00:00 +0000

Store your secrets right in your git repo using sops / age for encryption and decryption.

DO NOT commit unencrypted secrets to Git.

Toolchain

We use sops and age to handle encryption of secrets.

Sops can be downloaded from https://github.com/getsops/sops/releases

Age can be installed from https://github.com/FiloSottile/age#installation

Age is pronounced with a hard G. Like Aghey.

Files

Secrets must be stored in text files, such as YAML or ENV files. In case the secret is managed via Kubernetes, the YAML is typically a manifest file representing a Secret. In case of a frontend application, the secrets are typically stored in a .env file format.

There should be a clear distinction between encrypted and unencrypted secret files. To make this distinction, all encrypted files should have the suffix .enc.yaml or .enc.env, where .enc signifies that the file is encrypted. Example: secrets.enc.yaml or stage.enc.env

To be very safe, is it better to add the plain names of the files to .gitignore so that no one accidentally commits an unencrypted file to git.

Keys

Before we begin to encrypt secrets, we need to create a keypair using age. To do this, run:

age-keygen

The output will look like this:

# created: 2023-05-08T15:38:39-04:00
# public key: age1xxxx
AGE-SECRET-KEY-yyyy

The public key, also known as identity, or recipient, looks like age1xxxx.

The secret key, or simply key, looks like AGE-SECRET-KEY-yyyy.

Store the output of the command (including comments) in a file /home/yourname/.config/sops/age/keys.txt(for Linux), or /Users/yourname/Library/Application Support/sops/age/keys.txt (for Mac)

If you use Bitwarden, it is a good idea to backup this in your vault. It is important you do this before moving forward. If you lose the key, it is NOT possible to recover it. (However, if you lose the public key, you can derive it from the secret key using age-keygen -y)

Encrpytion

Now, we will use sops to encrypt the secret file.

In case of YAML file:

sops -e -a age1xxxx secrets.yaml > secrets.enc.yaml

In case of ENV file:

sops -e -a age1xxxx stage.env > stage.enc.env

Always review the file before committing to git, making sure that the file contents are indeed encrypted.

Decryption

In your local laptop, if you need a edit a secret, you can use sops to directly edit the file in an editor like vi. Run the command:

sops secrets.enc.yaml

In your local laptop, if you want to dump the secret to your terminal to review, you can run:

sops -d secrets.enc.yaml

In order for this to work, you should have the key stored in your keys.txt file as described in earlier section.

Configuration

Consider a Kubernetes secret manifest that looks like this:

apiVersion: v1
kind: Secret
metadata:
 name: myapp-secrets
 namespace: myapp
stringData:
 AUTHORIZATION_USERNAME: edward
 AUTHORIZATION_PASSWORD: MargaretThatcherIs110%Sexy

Encrypt it using the command:

sops -e age19esgdwuqmu75sl9mrkxrr9sn2yla82jtfzhf3ua2430r3ftttgxqrglsyv secrets.yaml

And you will get this result:

apiVersion: ENC[AES256_GCM,data:Efc=,iv:SF0ZIFF+dVm//VweksahH/MIbJajj2TX82jwEooLFG0=,tag:ZJ36YUfz/4YEhaNJmkYZog==,type:str]
kind: ENC[AES256_GCM,data:arfGKcp4,iv:7OzQL9pCqyLTfib3t9OOfldGGxG0igAJi43Gx3q5okY=,tag:9Pjr6GsVMfIYxZ1oZIO2Xw==,type:str]
metadata:
 name: ENC[AES256_GCM,data:ErHyEow3CpuTtoq8tQ==,iv:11MFZgAHuPRcGDL0EWntDT14vM2ix8VKwOy914jXKB8=,tag:SkGF+LTUb4rRaHc1NlUVbg==,type:str]
 namespace: ENC[AES256_GCM,data:8YUTU0U=,iv:hAExMObv3gSXhojEhYcW3hEePEgNVGPkeeJzJIYjetY=,tag:Ez2A4yhXiRj3cMzuNDddzw==,type:str]
stringData:
 AUTHORIZATION_USERNAME: ENC[AES256_GCM,data:PWRGPGHk,iv:P6fXYtS+6ETypBOEcgoNjxakcoYOjwcxqStZDJEU7pM=,tag:a5anLYBaKGjsL8dm3o1ovA==,type:str]
 AUTHORIZATION_PASSWORD: ENC[AES256_GCM,data:3BMUm60XEqoOfHQyif7zTm/s5ZMOnRtoIj4=,iv:DwP5UCDjYONqE7ShI9wM98KD+qo5/YFp01/f/GQznhc=,tag:vJvGS7Rym058/VzODtR+ig==,type:str]
sops:
 kms: []
 gcp_kms: []
 azure_kv: []
 hc_vault: []
 age:
 - recipient: age19esgdwuqmu75sl9mrkxrr9sn2yla82jtfzhf3ua2430r3ftttgxqrglsyv
 enc: |
 -----BEGIN AGE ENCRYPTED FILE-----
 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlMTh0aFRaQ2JjSWtjNHFt
 Q0VLRmJoVm0vMjBTdThSTFRjT3dlWDZDR3lrCjllZ2l5eUJ3bC9iZUEyRU1LSWdk
 OU9ZdTM1K0NHT09LK2YxTjNoNTRDRjAKLS0tIHA5UWh3cVlHNGpWWXBqUm11UFNi
 ZWZsb2RQb2NyY1h4YzZNOE1tempvc2sKgZX1kiUuImKMJa0bTdoWLmiLIrABTDHf
 ZLOi2RUTOh/Ui+mfV1dcYrUV6SKJvlGwwRZEeuuBrG6Jbd5W32A6OQ==
 -----END AGE ENCRYPTED FILE-----
 lastmodified: "2024-05-21T19:42:28Z"
 mac: ENC[AES256_GCM,data:sHZMYFmP6/HvLGz0LqMQ2kWVvvWVS+K01IoaqeXSkyE3nR9Qi5JJJ6aeRMjfhDygBMGw2P+0zS/6KcPBoeQpvKWbWZFAyWcTZJIx+iI8bIsWivtMOlNydE61C+0xewVIvej3AyKkSyxdSbuDO/1/rLg9KXjkJJ58TtYxz+jk8MA=,iv:eM9viOEs61XQ0f4mkgoHsAi9bxtRUHSQ1IgiPIslAW4=,tag:1iKWK7FlKpzFBH/6wr3AJg==,type:str]
 pgp: []
 unencrypted_suffix: _unencrypted
 version: 3.8.1

Notice that all keys in the yaml including non-sensitive ones like apiVersion, kind and metadata are also encrypted. This may be underisarable when doing code reviews or otherwise looking at the encrypted file.

To avoid this, create a .sops.yaml (this file is usually placed in the root folder of your repo):

---
creation_rules:
 - path_regex: secrets.yaml
 age: age19esgdwuqmu75sl9mrkxrr9sn2yla82jtfzhf3ua2430r3ftttgxqrglsyv
 encrypted_regex: ^(data|stringData)$

Now, you can encrypt without specifying any identities:

sops -e secrets.yaml

And you will get this response:

apiVersion: v1
kind: Secret
metadata:
 name: myapp-secrets
 namespace: myapp
stringData:
 AUTHORIZATION_USERNAME: ENC[AES256_GCM,data:Lyj81idG,iv:w77ofABHiYHyup8z/bUMqFbnZ1oL82H1gbgQ2m2ixRQ=,tag:8IIQi4hWgiGRfRP6jYWxRw==,type:str]
 AUTHORIZATION_PASSWORD: ENC[AES256_GCM,data:8Bgfii+UiFTGURyFSvJZ4RZuzRVgMG33RFM=,iv:ToBjzFzrGCbpauXYfa3PNDuxTGerUfmmtNovWah+rNs=,tag:hkTNGymGTtPo6O5q823LQw==,type:str]
sops:
 kms: []
 gcp_kms: []
 azure_kv: []
 hc_vault: []
 age:
 - recipient: age19esgdwuqmu75sl9mrkxrr9sn2yla82jtfzhf3ua2430r3ftttgxqrglsyv
 enc: |
 -----BEGIN AGE ENCRYPTED FILE-----
 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCN2toTmFFMS9FdVduNFBU
 eWRKWmcvSnE3Wk8wVzlUZTN4QVdiRUlZSzFZCnlWNytrZTBJcjMzcklWVnFNb3k3
 YjRCdE52dDhiYk1WdHkxNXoxZlZraDQKLS0tIGVCNFdIV0ZuREtjU3NlM0dCcVhn
 QUtwN3BmUGNwYldabTcyQ1dvc1FYTkkKvl41B3MxZVER+BHBGiQJPQW62cm8TsiK
 TIYIqVcJ4HpKdCe6t2KFVnfDHwVJ1BbarUOMGWjMfwaLMpOlB6fRgQ==
 -----END AGE ENCRYPTED FILE-----
 lastmodified: "2024-05-21T19:49:05Z"
 mac: ENC[AES256_GCM,data:aJAVVBeh0u8o02QH+hnBfewYnVlt+omOQY+IXF7FixWDNtNTRBwMb0JbqAt25rda3P7zELgRd1bfDnClFY7JadolIFN1iwPi8w/hQSv5Vp32N259ukpzxpWozT7Xl7CkR1XaOMvXTTQw2Uv4RfdyyrTxIlPAwsjCfShbiVjFf2k=,iv:niizOhZ9TeEWr1Vms+h/F6wubQZND197q6XpAMMqa7k=,tag:XxAsF1n8nQFtNepf7ePwsg==,type:str]
 pgp: []
 encrypted_regex: ^(data|stringData)$
 version: 3.8.1

As you will note, only the keys under stringData are encrypted.

CI/CD pipelines

For decrypting secrets in CI/CD pipeline, run:

export SOPS_AGE_KEY=AGE-SECRET-KEY-yyyy
sops -d secrets.enc.yaml > secrets.yaml

You can also store the secret key in a Github actions env var and use that here. And finally, you can use the decrypted secrets.yaml in a kubectl apply -f command.

Blog | Mani Soundararajan

Visualizing Software Architecture using C4 Model and Python Diagrams

C4 Model

Diagramming tools

Example

Context

Container

Component

Code

Reasoning

References

Guess Which Door Has A Car Behind It

References

Developing Software For 25 Years - A Retrospective

1. Free and Open Source Software makes the world go around

2. Reading is a superpower

3. Saying Yes to more things leads to interesting experiences

4. Empathy is the missing piece in most software stacks

5. Doing customer support improves empathy

6. Find your rhythm for doing boring repetitive tasks

7. Figure out how your brain works

8. Sleep is an important step in debugging

9. If you don’t like your organization’s culture, change it

10. In software ethics, you are the last line of defense

Version bumping with Uplift

Uplift

Conventional commits

Configuration

Demo

References

Managing context switches

Congnitive load of interruptions

Enter TODO+

Easy Solution

Architecture Decision Records

Versioning Go Applications

Overview

Get Version

Linker flags

Dockerfile

CI/CD Pipeline

References and Credits

Conventional Commits

Overview

Types

Scopes

Breaking Change

Description

Body

Footer

Why

References and Credits

Managing secrets using sops/age

Toolchain

Files

Keys

Encrpytion

Decryption

Configuration

CI/CD pipelines