Developing Software For 25 Years - A Retrospective

This week I am completing 25 years of my software development career. I figured this major milestone deserves a retrospective. Here are some of the lessons I have learned over the years:

1. Free and Open Source Software makes the world go around

Without going into details of copyleft licenses (like GPL) or permissive licenses (like BSD, MIT), I will say that free and open source software (FOSS) runs a lot of things on this planet. FOSS powers a good part of the internet. Governments around the world are switching to FOSS. Heck, there’s FOSS running off-planet as well. Much of the tech boom we have seen in the last two decades was made possible because we are standing on the shoulders of FOSS giants.

Over the last 17 years, I have made several contributions in my own small way. I also got an opportunity to speak about my cool (at that time) 5000+ docker container production deployment at OpenCamps 2017 held at the United Nations in New York.

At this 25th year of my tech career, I am more confident than ever about the future of FOSS, and I am doubling my resolve to spend more of my time using FOSS, promoting FOSS and contributing to FOSS.

2. Reading is a superpower

In this era of attention attrition, I have found that my ability to read is a superpower. I am able to answer so many of my colleagues’ questions simply because I have read the documentation cover-to-cover. Other times, I enlarge my knowledge base by NOT going to stackoverflow or reddit seeking a quick answer, but by reading official documentation, API reference and even upstream source code.

Reading has several obvious advantages to watching a video. You can usually skim a page (or even screenful) of information faster than it takes for you to watch a video about the same topic. Also, reading has “random access” capabilities. You can skip a few pages or go back. With video, cueing forward or reverse does not have the same quality of experience.

This superpower extends to reading in general. To parents of young children, I will say this: children learn by observing you more than by listening to you. So if you want your kids to read, you should be seen reading books, preferably the print kind.

3. Saying Yes to more things leads to interesting experiences

I have worked more than 20 years in startups. A startup career does not come with fixed job titles or responsibilities. Over the years, anytime I got the opportunity to step outside my comfort zone, I have said yes - and every time I have come away gaining a lot of experience.

I said yes to working on embedded firmware, and learned that I can apply my skills to solve any problem - including finding a runaway pointer in a ring buffer that caused memory overwrites and sleepless nights. I said yes to working on a Radio Frequency communication project, and learned that the random packet interference I was receiving was just folks opening their cars with their key fobs. I said yes to working customer support and learned that my customers were using my product in ways I did not intend or even imagine.

4. Empathy is the missing piece in most software stacks

I have been asked many times about what I look for when hiring someone. I always say Empathy and Motivation. Our world (and our technologies) will be a whole lot better with some empathy.

Be more empathetic to your customers - they are the ones putting the food on your table. Be more empathetic to your colleagues - they are the ones sharing your responsibilities. Be more empathetic to your frontline customer support - they are the ones facing angry customers when you break production. And lastly be more empathetic to future developers who will be inheriting your code base.

5. Doing customer support improves empathy

Some of the most insightful learnings about my product have come from doing customer support.

Customers have this uncanny ability to bend your product to their will to achieve their objectives. If you are not paying attention to this, sooner or later you will make a change in your product that breaks the customer’s workflow. Remember, backward compatibility is not just for your published API - but for all of your software’s behaviors that the customer is using in their workflow.

When you can, try to spend a rotation doing customer support.

6. Find your rhythm for doing boring repetitive tasks

We have incredibly interesting jobs - but that comes with sometimes having to do boring repetitive tasks. This is work that is not appealing to do, but needs to be done nonetheless. Sometimes it might make sense to spend the time automating or scripting a solution - but most times it’ll make sense to grind through the chore.

I have learned that it is important to find my rhythm when doing such tasks. If I go too slow, I will zone out and lose focus. If I go too fast, I will end up making mistakes and spend even more time cleaning up my mess. Things I have found helpful are making checklists and checking items off at a steady pace.

Like the US Navy Seals say: “Slow is smooth, smooth is fast”.

7. Figure out how your brain works

One of the first lessons we learn as students in a laboratory course is called “study of tools”. If this is an electronics lab course, you will learn about voltmeters, ammeters etc. If this is a carpentry shop course, you will learn about t-squares, various kinds of saws and so on.

What are the tools of our trade? Keyboard and mouse? Sure we all know how to use them. Programming languages, IDEs, compiler toolchains? Yes, we all know that too. But we don’t really think of our brain as being part of this - and yet it is the most important tool in our toolkit.

As software developers, we owe it to ourselves to try and understand our brains a little better. How does my brain perform after a carb-heavy meal (not good), and how does it perform after going for a run (much better).

That is why I find topics like Cognitive Load fascinating. That is the reason why I ask my team to keep pull requests to less than 200 lines of diff so that the code reviewer does not suffer cognitive overload (see point #4 above regarding empathy towards colleagues).

8. Sleep is an important step in debugging

There is no bug you can’t tackle after you’ve had a good night’s sleep.

Again, to understand this, you will need to understand how your brain works. The more awake you are, the more toxins build up in your brain - and the only way to flush them out of your system is to sleep. And it’s not like you have stopped debugging the problem you were working on. In fact, the work continues in your subconsciousness. The NREM cycle of your sleep moves information from temporary memory (hypothalamus) to permanent memory (cerebrum), and the REM cycle creates links between neurons representing new information and neurons representing information you already had in your brain. This “linking” process is what creativity is - and this is going to help you come up with creative ways of debugging the problem.

I would highly recommend Dr. Matthew Walker’s book “Why We Sleep”. And I would also recommend getting something like a fitbit and wearing it while sleeping so you can start seeing patterns of your mood / performance vs. your sleep quality.

So, if you are not able to fix a bug after working on it for several hours - call it a day and go to sleep. You will wake up with fresh perspectives on how to approach your problem. Now, if you have figured out how to account for that in your timesheet, please let me know!

9. If you don’t like your organization’s culture, change it

We all like to say work culture is a really important factor when considering joining an organization. But what is good work culture? For a really good definition of what makes a good work culture - look no further than the book Accelerate by Dr. Nicole Forsgren, Jez Humble and Gene Kim - skip to Chapter 3 to read about this topic - or better yet read the entire book. To paraphrase the authors quickly, there are “pathological workplaces”, “bureaucratic workplaces” and then there is “generative workplace” - which is what we all think of as “good work culture”.

Setting the organization’s culture is not something that is the sole domain of the manager - everyone can and should work towards improving the work culture. If you work in a small company, there is no reason why you shouldn’t be able to implement changes. Even if you work in a large organization, you can bring about changes in your team.

10. In software ethics, you are the last line of defense

Technology has taken over much of the world - and I have seen this just in my lifetime. Software runs pretty much everything - and as software developers we have great power. We can either use this power to build features that will benefit our customers, or we can abuse this power to build anti-features that are not in the best interest of our customers.

Without going into specifics, I can say that I have, in the past, pushed back on dubious feature requests - and the management in each case agreed to drop the feature.

I am here to say today to you - that you have the agency to refuse to work on anti-features. This is not something you have to earn with seniority - you get this on day one.

Remember what uncle Ben said: “With great power comes great responsibility”.