Companies who could afford in-house engineering teams often have tendencies to build custom in-house solutions no matter how prevalent that solution already is. To highlight the absurdity, let me give an example: Jupyter Notebook. Despite the space being extremely competitive, Google has Colab, Amazon has Sagemaker, Azure, Databricks, Domino, Binder all offer similar services and the list goes on, Product or Technology still cannot resist the urge to build one (if not multiple) in-house. Not to mention some notable companies spent fortunes on building fully integrated notebooks on their platforms.
Everyone who works with me knows my strong stance on building something new - if there are existing solutions to a similar problem, one should not attempt to build without fully exploiting the existing ones first.
Engineering’s tendency to build something new is sometimes understandable:
I challenge with everyone who wants to build something new to answer these:
Do not misunderstand this as opposing changes and new technology. As a matter of fact, I am often the one that brings new technologies into an organization and encourage people to try it out (almost to an annoying level). Let me elaborate more.
One should not “re-fix” a problem, instead, one should constantly try to identify better problems to solve and “re-fit” solutions (existing or new) through the lenses of problems.
The questions are often not asked: what is it that your problem so special from others? Sure, when things are not working, people are frustrated, instead of framing the right problems to solve, perhaps it’s easier to solve them first.
Let’s take the notebook example (90% based on mixes of true stories):
Let’s see how engineering will build when stories are rolled out as above without product’s help to frame the issues into a better problem.
Very logical progression, other than not asking the right question in the first place, engineering can hardly be blamed for wasting their time solving problems that’s been solved over and over by the internet. Some of them might eventually raise questions after they found out just how many people have approached product’s unique problem on google. I highly recommend to keep in touch with those engineers who dare to ask these questions, they will at some point prevent teams from wasting $$$$ on useless projects down the road.
Custom solutions can rarely compete with existing vendor offerings. From the ones built by consultants to the home grown enterprise tools, people rarely have good things to say about them.
Home grown solutions rarely respond to broader market needs but are sourced from problems shared across teams within the company. They rarely have the luxury to step beyond requirements and respond to visions or market trends.
If IPhone was developed by listing out a complete list of problems they want to solve instead of visioning how the future of communication on handheld devices, we will never have IPhone.
If a truly great product can never come from custom solutions, why is it so prevalent?
When existing solutions don’t exist or don’t meet the needs, it’s easier to opt for building a custom one. When I was looking for ML frameworks, only a handful of them were relatively mature, and we still need to add a number of features for it really be useful. That being said, it’s too often product “misunderstands” what they need doesn’t exist or what exists doesn’t meet their needs. Politics often pushes them to build something anyways instead of calling for surveying on existing solutions.
Perhaps a more justified cause is that outdated systems make it difficult to integrate existing solutions. There’s only enough budget to build custom features to meet the needs but not enough to rebuild the whole system so that engineering can easily extend upgraded systems with existing solutions. Due to both cost and resource constraints, it’s quite common to update only one part of the system while keep the remaining functioning as is so one could keep the customers happy and the cost reasonable. In this case, engineering will often build custom solutions until it’s possible to integrate with more permanent solutions.
Real issues that could come with custom solution in a relative near term:
On one hand, every company is complaining there’s not enough tech talent; on the other, most people or company I talked to want to build similar but slightly different solutions - it got me thinking, what could be the reasons that most companies wants to build something that’s already there?
I reflected on different organizations and teams I worked for, and I realized, there’s indeed biases towards custom solutions without evaluating whether it’s really worth it?
If anything I’ve learned from this wasteful approach to building custom solutions:
Anyways, happy coding. Focus on what the company does the best, not building one off custom solutions no one likes.