Stepping up your game

How an investors mindset allows you to improve your stack

At the start of 2021, The Next Closet was flirting (successfully) with investors. Part of an investment round is the due diligence.

In my role as tech lead of the company, the engineering due diligence was most interesting. This is the process where the potential investors do their utmost to find flaws in (amongst other parts) the software, hosting and team composition that may hinder the continuity of the company they’re investigating. This is also where an investor is assessing potential of (the software of) the company. Clearly, it’s not only about risk, but also about potential.

On April 20th 2021, I had the opportunity to share my findings of this process with the audience of the Amsterdam Ruby community (of which I am a co-organiser).

screenshot from the talk

How an investors mindset allows you to improve your stack

The team, the culture, and the continuity of both are for your stack what soil is to a forrest; a healthy soil allows the forrest to flourish. Given that the stack is most visible (as the forrest is (often) more visible than the soil), focussing on the stack only would give a superfluous model of reality.

Because this document started as a talk for engineers and engineering managers, I felt the need to stress the importance of both the (visible) stack, and the underlying team, culture and continuity. Not only about what the end result looks like, but also the way towards that result.

The risk-assessment part of the due diligence had a few major topics: security, intellectual property, the development process, infrastructure. This part of the article is divided in exactly these subjects. Let’s go!

Security

As the number one risk most people think of when thinking about (web) software is the security of the offering. Is the system tamper proof?

But it is about more. Most probable, you will be handling some personally identifiable information in your systems. Will it not leak, nor expose personal information of your users, whatever the circumstances?

From the perspective of the GDPR, the EU created a fantastic checklist that allows you to gauge how you’re doing. But security doesn’t end there; You may want to prepare for more than not oversharing your users’ details.

You (hopefully) have a CI environment. Ensure this CI is used to do static analysis on your code - but also for security issues. 3 major gems come to mind: 2 of them inspect your code, 1 of them inspect your dependencies:

These tools, when added to your CI, help you ship better code, and not let vulnerable libraries go unnoticed.

Given that you may, or actually will 🤔 still have vulnerabilities, why don’t you add a responsible disclosure section to (the footer of) your website - and create a process around it to handle incoming reports. hackerone and zerocopter can help you out with triage.

Prison with the text ROI - usually means return on investment. With security it usually means risk of incarceration

Let’s protect against unwanted access, without closing the doors on your engineers. Make sure to always know who has shell access (and their user account on the server(s) should not be shared). Ensure that creating (and destroying) of these accounts is covered in your onboarding and offboarding documentation. The offboarding will also have a process that allows you to rotate keys and tokens of external services - and sharing the new secrets where needed. And your colleagues will have to share credentials with one another. Make sure your default communication tool has end-to-end encryption, or use your password manager for this.

If you’re big enough a fish, you will be targeted. Ensure your logs are stored on external systems, in such a way that having full access to your production stack does not mean all traces of an intrusion can be scrubbed. Hopefully your logs are append-only, as that will greatly assist law enforcement or other parties that get involved.

From a money-making perspective; you will want to have a backup and restore procedure that’s battle tested. This ensures that restoring your app in a good (although possibly maybe slightly outdated) state is a no-brainer.

All major and minor incidents should be logged in an incident log, accessible to your colleagues, to learn from. Both the cause and how the issue got resolved are fantastic learning opportunities that no external teacher can replace.

Ownership of intellectual property

If you own a company; there are two interesting revenue streams; the current offering to your users or customers (your webshop, or a brick and mortar store), but also the income on the licensing of the intellectual property (think the current Philips).

First and foremost ensure you are the owner of your domain names. Second, ensure the ideas and implementations that take shape during work stay within the company.

Given I am mildly uninterested in this topic I didn’t go in depth in the talk. Long story short: Ensure your team is handsomely compensated for signing of their rights, and don’t overdo the signing of.

Your colleagues will have fantastic ideas (that’s why you hired them in the first place, remember?). Ensuring that their non-work related ideas, concocted in their non-work hours are yours to reap the benefits of, is a fantastic way to chill their innovation spirit.

The development process

This is fun for all the BS bingo cards; words like agile, lean, plan-do-check-act, :shipit: and others can be sprinkled limitless. Making fun with (or of) the uninitiated aside, it is always a good moment to reassess whether your (cross functional -> bingo ✔) teams are aligned with your business needs, or that the your technical debt, prioritised from dumpster fire throughout WONTFIX, is clear to the team.

Maybe your most important libraries (think frameworks) are outdated, or even your programming language is. That is a giant liability that can be fixed. A potential harder nut to crack is ensuring your test coverage is at least not as bad as your competitor. And with test coverage, I don’t mean a vanity metric from some coverage tool, but the effectiveness, soundness and speediness of your test suite from unit tests to end-to-end tests.

Then it’s also important to know how you build code, or grow your code base to stay in the soil and forrest metaphor. Are you having a closely knit team of professionals building things together, or is it more a collection of mushroom developers (a term inspired by mushroom management), that instead of working together rather throw features over the wall?

And then documentation. Remember the ‘we value working software over comprehensive documentation’? Do you also remember joining a company where working software trumps any and all documentation? That is not welcoming at all. From an investment perspective it may even be one of the biggest hurdles when it comes to scaling up.

And these development processes aren’t meant to be mandated by the CEO ➡ CTO ➡ Engineering Manager ➡ all the way to the individual contributor. It is a team effort where the team finds the best way to work together to reach its goals.

Bingo card full yet?

Infrastructure

This is not about you hosting on either Amazons’, Microsofts’, or Googles’ offering. This is about how you handle exceptional cases, from issue happening in the browser, to enjoying a specific downtime just once, to having a clear escalation process (from mentions like ‘our page is slow’, all the way to ‘💩 someone just downloaded our whole database with unencrypted personally identifiable information’).

In other words:

Never let a good crisis goes to waste

— Winston Churchill

An image with Winston Churchill making a peace sign, and his quote from above repeated

Given that lots of wheels will come off under exceptional load, testing your app for serious load is always a good idea. The day your company makes the headlines, you want to be well prepared.

Ensure you analyse your load tests over time. You will want to know how the bottlenecks evolve over time. And ensure you triage these bottlenecks; so you get a feel for what’s acceptable (page load going from 2ms to 5ms maybe?) and what is unacceptable (a database that needs a full restart given it depleted all its resources maybe?)

And those issues may be caught even before you do a load test - just make sure you do track your client and server errors, and also have a write-up about your outages (also see above, regarding the incident log).

Apart from your development and production environments, you may have one or more quality assurance (aka staging) environments. This is not per se problematic, until these environments go down. Having to fix a non-production, non-development environment is a cost that has almost zero rewards. Either these platforms can be brought up with the run of some code, so the fix is as simple as recommissioning your platform, or you have a few hours of non-productivity staring you in the face.

When your platform has proper support for feature flagging, you can ditch many, if not all QA-ish environments between your development laptops and the production offering. This ensures your time is better spent than fixing non-critical environments.

Let’s talk about your engineering potential

With these risk-assessment topics out of the way, let’s look at the things that make you smile if they go well. Your stack’s potential is based on your teams continuity, composition and culture. What you’re aiming for is the exact opposite of what 10x engineers have to offer.

Your executive leadership team should definitely set goals. They may even be stretch goals (for the non-initiated, that’s business speak for aiming for the impossible).

And that’s exactly how far their influence should go. The tactical and operational decisions should be formed in the teams where the lack of these decisions present themselves clearest; the teams doing the actual work making these goals.

A bottom-up culture will help you grow

To solve the problems that enable the organisation to meet its goals, some work needs to be performed. But how should that work be done, and by whom?

Up until the goal was unveiled, everything was hunky dory in the team. But this new goal creates a tension. Note: If there is no tension, possibly the goal is already reached, yay!

The tension for the team can be converted into a role (or many roles). A role is a collection of both purpose and accountabilities. The member in your team that has the biggest tension, is usually the member that can clearly express the role, given they are already invested in it. Probably the role is even something they want to own, alongside any other roles they’re owning.

In order to help them grow in their roles, ensure that both the purpose and accountability are visible in the whole organisation, but also the fact that this specific person, or collection of individuals are engaging in addressing the tension. They may change the role every time a mismatch between the status quo and the needed team configuration is found. No one needs a manager for this; the manager can facilitate finding the proper purpose and accountabilities, but that’s as far as they should go.

Ensure that on a bi-weekly basis, you iterate on your roles. Not all of them at the same time, but things change, and your teams’ notion of its composition should adjust accordingly. New colleagues may find this a bit uneasy, but after they connect purpose and accountability to their daily routine, they will follow the rest of the team. Their influence in altering adjusting the role when needed, makes it an even simpler buy-in.

An example

If you’re having a team member that has too much to do, and not enough time? They may want to recruit their new colleague. If this role does not exist, make the write down the purpose and/or accountability, while joining the hiring from job opening creation to onboarding. Probably you will find multiple accountabilities and purposes that do not fit in a single role. Create as many as needed. Know that a single individual can energise very many different, but related roles.

Hiring & Training

Eventually you have a set of roles that will or may reflect reality. Some roles do not get the attention they need. That role will help you grow your team - the purpose and accountabilities make fantastic prose when hiring for exactly that role. And roles that are unclear, or even with enough role owners still go a bit unnoticed, may be a fantastic drive for your team’s (self) education.

Regarding hiring new colleagues to your team. You will want to use a standardised set of questions for applicants, and rate their responses in a standardised way. This ensures interviewer bias is easier to spot. The alternative to this way of interviewing is that you may end up with a standardised set of colleagues - everyone you hire looks like the interviewer.

Hiring and training - the text above paraphrased as 'Use a standardised set of questions in your hiring process. Otherwise you may end up with a standardised set of colleagues.' Additionally we see a fractal-like (but real) photo of 6 very alike looking white men, same hair do, same toothpaste smile, sitting in a stadium

The price of education may be high. And not only (external) trainers that help you and your colleagues understand what you didn’t know before. Consider downtime, incidents, lack of role fit in your team as education costs as well.

Now that cost feels even worse if you’ve just trained your colleague (quite expensive, given they’ve just caused and solved an outage of your web shop during Cyber Monday) and they leave… Lots of knowledge will leave your company, and you paid for this knowledge…

The alternative is clearly that you don’t train your colleagues - and they stay.

People will join your company because they want to grow. They may probably want to excel. That intrinsic motivation to get better, doesn’t stop after they’ve signed their initial job contract. Ensure they get to grow in your company for both your benefit.

Let’s talk action (also known as agility)

Runners sprinting profusely, with the text 'Action is something that can be done with Agility'

Action is what you do. Agility is a way to adjust quickly to a new situation. A colleague in a former job found ‘action’ so addictive, that - when in a meeting that went on and on - their spirit was:

When in doubt, always take action.

— an engineer in a slow deciding organisation

I’ve heard about another profession, wood working to be precise, that this approach is lacking two important steps:

Think thrice, measure twice, cut once.

— Woodworkers lore

In other words: ‘Action’ may sound like the thing you need (cutting in this example), but a proper process is the thing you need (thinking and adjusting before taking action).

With processes, I don’t mean overpaid middle managers in meetings with notes and action-urging bullet-points. I mean the text outlined throughout this document: team members connecting roles to the organisations’ goals, making it visible to everyone. Or procedures around backup and restore, with team members whose accountabilities are on the line, improving it. Or logging incidents in such a way that everyone gets to learn.

Here goes: “When in doubt formulate a process.”

Wait, that’s clunky.

Let’s iterate on it.

Process over Actions

— Arno Fleming

a soccer trainer coaching their pupils, with the text 'Process over Actions'

If step 1 is to remove skeletons from the closet, that means that step 0 is to create processes to scale your culture. And iterate on these processes. That prevents a build-up of skeletons.

To stay in the original metaphor; ensure your soil is rich and healthy, as the trees that form your organisation will blossom on it.

Thanks for reading.

post scriptum

  1. I know you read the whole thing. Now go watch it (or share it with your colleagues)! screenshot from the talk

  2. Any questions? Ask them on the netherlands ruby slack.

Read more posts about talks tech leadership management