One lesson I’ve learnt so far working in a e-commerce outfit …

Software products are like babies - they have to be brought up enough attention while in nursery lest they grow up to become monstrosities you cannot recognise - worse still, you will be held at ransom to give in every whim and fancy because you can’t back out from SLA with clients.

All requests to external systems should have a timeout. It is better to amputate an arm than dying of gangrene

One can find plenty of good advice like this, https://goinnn.github.io/zen-of-high-load-and-high-availability-backend/index.html But you only have the luxury of practicing good hygiene while your product is in nursery.

In other words, while developing proof-of-concept (POC) and prototypes, you must keep a complete record of makeshift compromises and shortcuts for the sake of expediency. They must be redone properly before the POC graduate to production use. At the very least, guard against input which violates your POC design constraints with assertions.