Today, I'll explain from another angle why the pattern and meta executable will always be greater than doing things by hand.
I love to use real life analogies. They are amazing, you can explain very advanced concepts using terms every day Joe can understand. After all, every decent system relies on universal principles. For instance, father of the household is like God to his family (why God calls himself Father? So we'd understand him with earthly analogies of what father is?). Country with king is also same as God and his followers. Every natural divine system follows principle of masculine, godlike control plane and feminine, submissive plane that simply enjoys the fruits godlike control plane that thought everything out logically to be used.
Leftist cucks, since they went without divine wisdom in engineering for a long time, only produced abominations ever since by deviating from this truth. Well, not necessarily deviating, but rather being ignorant and never aware of that. Relying on their own pathetic weasel understanding, producing needless complexity everywhere without end and extreme toils and hard work for future generations to come maintaining that garbage.
What you have now in typical software development world, if you need a database, someone deploys database and you use it. If you need queues, someone deploys kafka. You need them working together? You deploy some other garbage. There is simply no system to connect them all together, so that all would be working towards the same purpose.
Unless, there is a divine all knowing meta executable from the pattern that knows about all these relationships and enforces them
If you see the whole, not just parts separately, you can go wild. You can build extremely sophisticated abstractions that would otherwise be impossible. For instance, you can make sure database is reflected in clickhouse from postgres. Someone might say, "wait, doesn't debezium stack do that?". Yes it does, if you provide schema registries and all the dynamic garbage. Debezium stack will never check beforehand if the target table in clickhouse has the same schema is source postgres table. All this is left to a poor developer along with myriad of other runtime problems he will encounter in his lifetime.
You know what can easily check if such abstractions are valid? A meta executable that knows the following:
- What kafka clusters are running
- What databases and what tables are declared in postgres
- What databases and what tables are declared in clickhouse
Having all this information in this place we can easily check the following:
- Postgres and clickhouse and wanted kafka instances exist
- Postgres schema matches clickhouse schema
And checking that we can provide a perfect error message, that says exactly what's wrong, that schemas don't match, instead of finding out about that somewhere in the logs in production
Not only that, we can generate the following:
- Any remap from any row with ends up in clickhouse
- Possibly remap to another typesafe kafka topic with custom typesafe code
- Maybe use clickhouse materialized views to generate another realtime stream to kafka topic
Possibilities are endless, because we have all information about everything as data in meta executable. With this approach we have a full body at our control, a body, that can use all of its muscles to perform single coordinated action, like striking a blow with a sword, do a pushup or dance ballet. The choice is ours. Did you notice, how seamlessly our bodies, under normal conditions, can make all the muscles perform unified action according to one mind? Or how unified the army marches? This is a divine pattern - everything is connected and coordinated to perform the same purpose.
The bodies of software development today work like this. An arm gets told to scratch the ass. The arm goes up to the ass and sees the ass is sitting. The arm goes to head to ask permission for ass to stand up. Permission granted, ass raises up. Hand finally goes to ass and scratches it. Imagine how bizarre this looks in your head - this is not normal. This is nonsense, paganism. All of these problems are trivial to resolve for a greenhorn developer if all the systems are declared as data in one place. Yet they create all sorts of nonsense these days, dynamic schema registration, separate schema evolutions in database, separate applications written to map stream to stream with no typesafe enforcement. Its extremely limited, because you inevitably stumble into the dreaded sentence "I don't know what schemas are in that and that subsystem, so, I'll use generic json and burden CPU with needless generic json parsing".
Things could be so much better than that. As I wrote in assumptions post, the more assumptions you make, the simpler your solutions will be. And the ONLY WAY to make these assumptions is to have all the data about your infrastructure in one place. Just like kingdom cannot be divided and stand, so is information about your entire project. If it is divided, there are bound to be conflicts and confusions. But if it is united, all in one place, and checked for consistency on a very high level - such project can stand.
And if a kingdom be divided against itself, that kingdom cannot stand. - Mark 3:24
Conclusion: software development paganism, division and confusion is cancer and causes huge complexities in software projects. Divided projects without strong interconnections between them are summed parts. Whole, where all your infrastructure is defined as data in one place is greater than sum of its parts and opens unlimited possibilities for correctness checking and code generation.
