The battle for technical investment…

It’s almost planning time… a battle is about to begin.

• Engineering Leads across the organisation sharpen their technical business cases, ready to repel any attacks
• A well oiled commercial organisation readies the cavalry, set to flatten any efforts to divert attention from immediate targets
• Product Leads assume the high ground, positioned to oversee proceedings and steer skirmishes

The three sides clash. Part way through the battle a clearing emerges, creating the space for two champions to do battle and decide the fate of the roadmap. A senior technical zealot from engineering steps fourth, cladded in impregnable system knowledge, armed with premonitions of future service failure, backed by sacred industry best practice.

A commercial champion steps forth, horse-blinkered by growth, tenacious in the pursuit of targets, strengthened by a fervent sense of duty to quash distraction in the name of success.

The show-down is tense, blows are landed on both sides, but with the fatal swing of a business-critical deal at risk, the fight is over. There will be no refactors this day... Bruised and embittered, the two sides retreat back to the safety of their domains.

The theory of a ‘healthy tension’

Ok… so reality might be a little less dramatic (and I’ve possibly watched LOTR a few too many times..), but the underlying conflict is all too real.

This is often an intentional setup, based on a theory of ‘healthy tension’ between domains. The theory goes:

• Engineering owns ‘engineering concerns and decisions’ (e.g. reliability, security)
• The commercial team owns ‘commercial concerns and decisions’ (e.g. growth targets)
• Product is the ultimate authority, owning forward product strategy and the final call on roadmaps

Good balance is (supposedly) achieved through negotiation and bartering, each area vying for their respective concerns.

Why this theory fails

1. There is often an imbalance in domain leadership

Decisions are typically weighted by the structure of the leadership team. In my experience, this is often a slant that aligns with the vocational background of the CEO, which I have seen swing both ways.

2. There is too much information to process effectively.

Planning this way, all possible priorities must be considered and compared. For each priority, there might be multiple delivery options regarding level of investment / quality / starting capabilities etc that map to varied business outcomes on different scales e.g. level of security risk / potential increase in sales / customer NPS. This forms an amorphous 3D puzzle that has to be pieced together while comparing apples and oranges. To make this possible, priorities are simplified into big blocks with simple outcomes: “Deliver feature X or we lose customer Y”, “Refactor service Z or the system will fail”.

3. This relies on (and breaks) honesty and trust between domains

When you pit two sides against one another with conflicting priorities, they will invariably start to use negotiation tactics to ‘win’ (even subconsciously). Over time, trust breaks down and bad decisions start to be made based on bad data:

• Exaggerated outcomes: “The service will fail!” / “The customer will leave!”
• Bundled priorities: “Let’s squeeze in this refactor… It’ll never get prioritised otherwise”
• Optimistic / pessimistic estimates: “This rewrite would only take a week” / “That feature would take 3 months”

The cost of failure

Bad decisions

When decisions are irrationally weighted, oversimplified and/or based on bad data, you will get one of two consistent outcomes:

• Consistent under-engineering (Eventual slowdown of development speed / low service quality / loss of talent etc)
• Consistent over-engineering (Slowed efforts to find market fit or deliver new features / inefficient investment in security etc)

Both can be terminal for a business, if the right balance isn’t struck.

The technical investment ‘black market’

Prohibition in the US created an unregulated black market for alcohol, the people found a way to get their drink. The same thing happens in engineering teams when you erode all trust and ‘outlaw’ technical investment. A ticket to change some text on the landing page suddenly becomes a 1000-line refactor taking 3 days. I’ve seen it happen, and I’m afraid to say the local sheriffs are usually in on the hussle... Investment becomes sporadic, based on the personal priorities of individuals rather than the business.

Domain bubbles and low team cohesion

Pitch your domains against one another and you will create friction. This destroys trust and cohesion between teams. This in turn perpetuates the breakdown at planning and can spiral into an adversarial, rather than collaborative, organisation.

The root issue

1. Decoupled accountability and control

Laying out the conflict in simple terms, you have individuals who are accountable for goals. Reviews, bonuses, promotions and even job security can rest on those goals, they are very personal.

Those goals depend on the same pool of limited resource, allocation of which is not directly controlled by those individuals. This creates a sense of powerlessness and sets up conflict.

2. Split accountability for commercial / product / technical decisions

Accountability is split by domain. If there is a security breach, it is the relevant Engineering Leader who is held solely accountable. If a commercial target isn’t met, it is a Product / Commercial Leader on the line.

When this is the case, why would an engineering leader choose to compromise on security investment for the sake of a commercial target? Why would a Product Lead opt for investment in service quality over feature delivery, when they are only on the line for the delivery of capabilities?

At the end of the day, if the buck ends with the domain lead alone for problems relating to their domain, they will only ever be motivated to prioritise their domain.

This ownership split and conflict can be summarised as follows.

An alternative way to handle decision-making

We need a different ownership model, one that:

1. Sets an authority responsible for achieving the right balance.
2. Maintains accountability and control for domains.

Rather than splitting ownership by Domain, this can be achieved by splitting ownership by Options / Decisions cutting across all domains.

In this setup:

• Engineering is responsible for producing an optimal set of technical options for priorities, then accountable for delivering on the selected option
• The Product Team is accountable for business decisions and associated outcomes, given the options presented

Illustrative Example 1) Security investment

In the typical setup, security is owned, decided, prioritised and promoted by Engineering in relative isolation (pushing for the necessary capacity to meet it). Instead Engineering would provide a few sensible options for investment in security ranging from low to high, with an idea of the associated risk. It is then on Product to make the business decision about acceptable risk profile v.s. competing business requirements. Should a security breach happen due to the high risk profile, it’s on the decision maker. If the agreed level wasn’t met at the time of breach, it’s on the team responsible for delivery.

Illustrative Example 2) Feature delivery and quality

Service quality is typically decided by engineering in isolation (defaulting to: maximum possible given timeframe negotiated). In this setup, again a few options would provided from low to high quality, with the associated speed of delivery / limitations / subsequent requirement for investment etc. It is then on Product to choose the appropriate option given the business implications communicated. This might mean a severely limited MVP that takes a week to build (with a necessarily rebuild after), or a more stable option that takes a month.

Why this works

1) Domain authority & expertise better aligns to ownership

The Engineering Team is the authority on technical capabilities, capacity and the implications of technical choices. Engineers should not be making the call on what an acceptable security position is, because they do not decide the acceptable risk profile of the business, that is a business decision.

The Product Team is the authority on business and customer needs. They cannot dictate what level of investment is required to achieve a given security position, or what the associated risk is. That is a technical position.

This approach aligns ownership, authority and accountability with a set of clean boundaries. In this way, Engineers get better at considering and communicating all options available, while Product leaders get better at making smarter choices with better information.

2) Cross-cutting accountability removes conflict

With cross-cutting accountability for decisions across all domains, there is now no conflict of interest between departments.

With no accountability for the business decision, there is a lesser incentive for Engineers to warp information to fulfil their own agenda. The better the options, the better the decisions that can be made.

With a better understanding of the business implications of given choices, as well as direct accountability that covers all concerns, there is a real incentive for Product leaders to get the balance right.

3) Smart decisions and effective execution build trust

With transparent choices and clear outcomes, it is significantly easier for decision makers to explain their decisions and align the team.

Those choices were produced and are owned by the Engineering team, with no external input to hide behind. This unfettered ownership drastically increases the incentive and likelihood for Engineers to meet proposed timelines.

These both ultimately act to build trust in the team.

Three ways this can still go wrong

No system is born perfect, this takes work and leadership to get it right. Here are three common pitfalls:

1. Boundary Crossing

It can be hard to let go of decision making. Business decisions might not align with team opinion e.g. the fast/cheap option always being selected. This can be uncomfortable and requires a real investment in alignment to explain reasoning.

In the same way, the business might want to challenge options seeking have-their-cake-and-eat-it solutions.

This relies on good options and good decisions, if one side is missing then that needs to be fixed. This approach actually helps to decouple concerns and identify the breakdown.

2. Failing to pay the piper

Options will invariably come with caveats, like “Refactor required after X time”. If these commitments aren’t met, all trust is lost. When this happens, the system starts to be gamed and ‘cheap’ options inflated.

For this to work, there needs to be trust and honesty, and for that to happen commitments need to be met.

3. Blame culture

When you split accountability by decision / implementation rather than domain, the line of accountability is a little more blurred. E.g. was a major outage a result of poor implementation or a more holistic service weakness due to underinvestment?

This type of dispute arises inside organisations with a blame culture, resulting in more conservative options, antagonistic decisions and effort wasted on maintaining records.