My Coding Cult

https://github.com/cultleader777/EdenDB

So, this is a brand new language that focuses on... Simply defining data. Like YAML, but not garbage, typesafe, no nulls or NaNs, with schemas and blazing fast to query. Compiles and emits outputs in milliseconds and relentlessly checks logical errors to disallow funny stuff in production.

Let's have a taste, shall we?

Say we have a table of servers

TABLE server {
  hostname TEXT PRIMARY KEY,
  ram_mb INT,
}


They can have disks, as children

TABLE disks {
  disk_id TEXT PRIMARY KEY CHILD OF server,
  size_bytes INT,
}


Looks similar to SQL, who needs another SQL dialect? Ok, let's see how we can define server with disks, in three ways:

DATA server {
  my-precious-epyc1, 4096;
  my-precious-epyc2, 8192;
}


As you can see, much less verbose and straight to the point as opposed to INSERT INTO ... statement.

There is no concept of insert statement because data is immutable.

Let's add some disks 1TB and 1.5TB disks that are children to server

DATA disks(hostname, disk_id, size_bytes) {
  my-precious-epyc1, root-disk, 1000000000000;
  my-precious-epyc2, root-disk, 1500000000000;
}


The column order (hostname, disk_id, size_bytes) is explicit here but as we've seen with server default tuple order is assumed if ommited.

This is rather verbose of adding separate rows for every server we want into separate place. Also, we redundantly repeat disk hostname.

Could we have everything in one place, so that it would look structured?

We can, with the WITH statement, let's define the same but in more concise way:

DATA server {
  my-precious-epyc1, 4096 WITH disks {
    root-disk, 1000000000000;
  },
  my-precious-epyc2, 8192 WITH disks {
    root-disk, 1500000000000;
  },
}


And here's the magic. We can define child or foreign key elements with WITH keyword and key of parent element will travel to children. There can be many levels of children and WITH statement can also be nested arbitrarily deep. There's a bunch of logic to check for errors, you can read test suite.

But long story short, this is not the garbage of `.yaml` where you can write anything we want, then you run tests and pray it will work. All fields are checked of being appropriate type, all fields must be defined, no null values are allowed and all the stuff like that is checked. Consider this typesafety as strong as defining struct in Rust or OCaml and knowing it will never have nulls or funny values.

Also, there's another variation of `WITH` so that our elements begin to look completely like traditional nested maps in programming languages, this statement is also equivalent:

DATA STRUCT server [
  {
    hostname: my-precious-epyc1, ram_mb: 4096 WITH disks {
        disk_id: root-disk,
        size_bytes: 1000000000000,
    },
  },
  {
    hostname: my-precious-epyc1, ram_mb: 4096 WITH disks [{
        disk_id: root-disk,
        size_bytes: 1500000000000,
    }]
  }
]


So, we represented the same data in three ways and all this will end up as columns for us to analyze.

Aight, what can we do, how about our data correctness?

We can write SQL (thanks embedded sqlite) to prove anything we want about our data with good ol' SQL.

Let's write a proof that we don't have smaller disks than 10GB.


PROOF "no disks exist less than 10 gigabytes" NONE EXIST OF disks {
  SELECT rowid
  FROM disks
  WHERE size_bytes < 10000000000
}


How do we prove we have nothing of certain kind? By searching for it and finding none. This is important, because if this proof in eden data language fails, it will return rowid of the offending rows which can be printed out to user so he could figure out where to fix his data.

Okay, SQL proof might be a little overkill on this, we have also column checks (thanks embedded lua).

Here's how we could have defined the disks table to have same thing checked with lua:

TABLE disks {
  disk_id TEXT PRIMARY KEY CHILD OF server,
  size_bytes INT,

  CHECK { size_bytes >= 10000000000 }
}


Also, lets have a lua computed column for size in megabytes of the disk:


TABLE disks {
  disk_id TEXT PRIMARY KEY CHILD OF server,
  size_bytes INT,
  size_mb INT GENERATED AS { size_bytes / 1000000 },

  CHECK { size_bytes >= 10000000000 }
}


Lua snippet must return an integer or compiler yells at you. No nulls are ever allowed to be returned by computed columns either.

But wait, there's more!

Datalog is the most elegant query language I've ever seen my life. But for datalog to be of any practical use at all, we cannot have any queries that refer to non existing facts or rules. I've found a decent datalog implementation for rust https://crates.io/crates/asdi , unfortunately, it does not support `NOT` statements or arithmetic operators of less, more and etc. So, I'll show two datalog proofs against our data, one that works today (even though it is meaningless) and one that would be practical and would work as soon as asdi starts supporting less than operator.


PROOF "no disks exist less than 10 gigabytes, doesn't work today" NONE EXIST OF disks DATALOG {
  OUTPUT(Offender) :- t_disks__size_mb(Size, Offender), Size < 10000.
}

PROOF "just check that no disks with name 'doofus' exists, works" NONE EXIST OF disks DATALOG {
  OUTPUT(Offender) :- t_disks__disk_id("doofus", Offender).
}


As you can see, our computed column size_mb is available in datalog to query.

But wait, there's more! EdenDB exports typesafe Rust or OCaml code to work with. Basically, once everything is verified we can generate typesafe code and analyze our data with a programming language if we really want to perform some super advanced analysis of our data.

Let's run this:

edendb example.edl --rust-output-directory .


On my computer this compilation literally took 6 milliseconds (thanks rust!). It should never appear in a profiler in a compilation process, unlike most compilers these days...

Let's see the rust source that got compiled!


// Test db content
const DB_BYTES: &[u8] = include_bytes!("edb_data.bin");
lazy_static!{
    pub static ref DB: Database = Database::deserialize(DB_BYTES).unwrap();
}

// Table row pointer types
#[derive(Copy, Clone, Debug, serde::Deserialize)]
pub struct TableRowPointerServer(usize);

#[derive(Copy, Clone, Debug, serde::Deserialize)]
pub struct TableRowPointerDisks(usize);


// Table struct types
#[derive(Debug)]
pub struct TableRowServer {
    pub hostname: ::std::string::String,
    pub ram_mb: i64,
    pub children_disks: Vec<TableRowPointerDisks>,
}

#[derive(Debug)]
pub struct TableRowDisks {
    pub disk_id: ::std::string::String,
    pub size_bytes: i64,
    pub size_mb: i64,
    pub parent: TableRowPointerServer,
}


// Table definitions
pub struct TableDefinitionServer {
    rows: Vec<TableRowServer>,
    c_hostname: Vec<::std::string::String>,
    c_ram_mb: Vec<i64>,
    c_children_disks: Vec<Vec<TableRowPointerDisks>>,
}

pub struct TableDefinitionDisks {
    rows: Vec<TableRowDisks>,
    c_disk_id: Vec<::std::string::String>,
    c_size_bytes: Vec<i64>,
    c_size_mb: Vec<i64>,
    c_parent: Vec<TableRowPointerServer>,
}


// Database definition
pub struct Database {
    server: TableDefinitionServer,
    disks: TableDefinitionDisks,
}

// Database implementation
impl Database {
    pub fn server(&self) -> &TableDefinitionServer {
        &self.server
    }

    pub fn disks(&self) -> &TableDefinitionDisks {
        &self.disks
    }

    pub fn deserialize(compressed: &[u8]) -> Result<Database, Box<dyn ::std::error::Error>> {
        // boring deserialization stuff
        ...
    }
}

// Table definition implementations
impl TableDefinitionServer {
    pub fn len(&self) -> usize {
        self.rows.len()
    }

    pub fn rows_iter(&self) -> impl ::std::iter::Iterator<Item = TableRowPointerServer> {
        (0..self.rows.len()).map(|idx| {
            TableRowPointerServer(idx)
        })
    }

    pub fn row(&self, ptr: TableRowPointerServer) -> &TableRowServer {
        &self.rows[ptr.0]
    }

    pub fn c_hostname(&self, ptr: TableRowPointerServer) -> &::std::string::String {
        &self.c_hostname[ptr.0]
    }

    pub fn c_ram_mb(&self, ptr: TableRowPointerServer) -> i64 {
        self.c_ram_mb[ptr.0]
    }

    pub fn c_children_disks(&self, ptr: TableRowPointerServer) -> &[TableRowPointerDisks] {
        &self.c_children_disks[ptr.0]
    }

}

impl TableDefinitionDisks {
    pub fn len(&self) -> usize {
        self.rows.len()
    }

    pub fn rows_iter(&self) -> impl ::std::iter::Iterator<Item = TableRowPointerDisks> {
        (0..self.rows.len()).map(|idx| {
            TableRowPointerDisks(idx)
        })
    }

    pub fn row(&self, ptr: TableRowPointerDisks) -> &TableRowDisks {
        &self.rows[ptr.0]
    }

    pub fn c_disk_id(&self, ptr: TableRowPointerDisks) -> &::std::string::String {
        &self.c_disk_id[ptr.0]
    }

    pub fn c_size_bytes(&self, ptr: TableRowPointerDisks) -> i64 {
        self.c_size_bytes[ptr.0]
    }

    pub fn c_size_mb(&self, ptr: TableRowPointerDisks) -> i64 {
        self.c_size_mb[ptr.0]
    }

    pub fn c_parent(&self, ptr: TableRowPointerDisks) -> TableRowPointerServer {
        self.c_parent[ptr.0]
    }

}


To get rows from table we can only get them through TableRowPointer special type for every single table. This type is basically wrapped usize and is a zero cost abstraction in rust. But, if you were to use raw usize you'd have a chance of passing TableRowPointerDisks to the server table, which is impossible here. TableRowPointer is just an index to arrays of our column or row stored data.

There are only two ways we can get TableRowPointer for every table:
1. We do a seq scan through entire table to get valid pointers
2. Pointers to other tables may exist as a column in a table. For instance, we can get generated .parent from disk row, or children_disks column generated for server table

Database is immutable and can never be mutated. User in rust can only ever get immutable references to columns. Meaning, you can run thousands of proofs in parallel against this data without ever worrying about locks/consistency and etc. Once your database is compiled it is a done deal and it will never change.

So we don't have nasty garbage in our api like, does this row exist? I better return an Option! No, if you ever have a TableRowPointer value it will ALWAYS point to a valid row in a one and only unique table. You can clone that pointer for free, store it in million other places and it will always point to the same column in the same table.

Of course, we should prefer using column api for performance, getting entire table row is added for convenience.

Deserialization assumes certain order in the include_bytes!() binary data and does not maintain data about tables already read/etc - we just assume that we wrote things in certain order in edendb and read them back in the same order. Also, binary data for rust is compressed with lz4 compression.

OCaml api is analogous and gives the same guarantees. Immutable data, separate types for table pointers for each table and so on.

Okay, let's do something practical with this, let's prove that no intel disks are never next to some cheap crucial disks to not ever humiliate them like that!

We'll change schema a little bit, let's add, what is effectively, a disk manufacturer enum:


TABLE disk_manufacturer {
  model TEXT PRIMARY KEY,
}

DATA EXCLUSIVE disk_manufacturer {
  intel;
  crucial;
}


DATA EXCLUSIVE says, that this data can be only defined once. Otherwise, we could keep filling up this enum with many statements in different places, this effectively makes this table an enum.

Also, let's add our disks to contain the make column with this data:


TABLE disks {
  disk_id TEXT PRIMARY KEY CHILD OF server,
  size_bytes INT,
  size_mb INT GENERATED AS { size_bytes / 1000000 },
  make REF disk_manufacturer,

  CHECK { size_bytes >= 10000000000 }
}


REF disk_manufacturer means this column has to refer by primary key to a single row in disk_manufacturer, you can never add invalid disk manufacturer value in the disk table or compiler yells.

Now our data is redefined with disk manufacturer:

DATA STRUCT server [
  {
    hostname: my-precious-epyc1, ram_mb: 4096 WITH disks {
        disk_id: root-disk,
        size_bytes: 1000000000000,
        make: intel,
    },
  },
  {
    hostname: my-precious-epyc2, ram_mb: 8192 WITH disks [{
        disk_id: root-disk,
        size_bytes: 1500000000000,
        make: intel,
    },{
        disk_id: data-disk,
        size_bytes: 1200000000000,
        make: crucial,
    }]
  }
]


So, how would the rust source look to prove that no intel disk is next to a cheap crucial disk? There ya go, full main.rs file:


#[macro_use]
extern crate lazy_static;

#[allow(dead_code)]
mod database;

fn main() {
    let db = &database::DB;
    db.disks().rows_iter().filter(|i| {
        // we only care about intel disks
        db.disk_manufacturer().c_model(db.disks().c_make(*i)) == "intel"
    }).for_each(|intel_disk| {
        let parent = db.disks().c_parent(intel_disk);
        let children_disks = db.server().c_children_disks(parent);

        for pair in children_disks.windows(2) {
            match (
                db.disk_manufacturer().c_model(db.disks().c_make(pair[0])).as_str(),
                db.disk_manufacturer().c_model(db.disks().c_make(pair[1])).as_str(),
            ) {
                ("intel", "crucial") | ("crucial", "intel") => {
                    panic!("Are you kidding me bro?")
                }
                _ => {}
            }
        }
    });
}


Do you miss endless matching of some/none options to check if some value by key exists in the table or endless unwraps? Me neither. We just get all the values directly because we know they will always be valid given we have table row pointers which can only ever be valid.

Full example project can be found here

So, basically, let's recap all the high level tools EdenDB gives to be successful working and building assumptions about your data:

1. SQL proofs
2. Lua checks/computed columns
3. Datalog (so far meh)
4. Analyze data directly in Rust or OCaml if all else fails

Basically, 4 ways to analyze the same data from four different languages and all data is statically typed and super fast/column based in the end.

Also, there are these features I didn't mention:
- include multiple files syntax
- include lua files
- unique constraints
- sqlite materialized views
- countless checks for correctness (check enum count in errors.rs file)

To see all features of EdenDB check out the tests, because I usually try to test every feature thoroughly.

And more will be added as I see fit.

What can I more say? Having such foundations, time to build the most productive open source web development platform of all time 😜

This posts assumes you have read the main thesis and are familiar with the concept of the pattern.

Sup, after doing the pattern for some time something inevitably happened that happens in every craftsmanship.

I happed to improve upon initial designs.

So, I want to list a few pain points I had working with the pattern v1:

1. Representing abstractions as data is kind of tedious, even in high level programming language like OCaml. You need to write functions, with default arguments and stuff. Ideally, I'd see only data when I'm working with data and not be burdened by programming language concepts.

2. Proving stuff about your data is possible, but not too convenient. For instance, it is trivial to write SQL queries against data, but if you need to iterate in loops against same data doing what SQL would do becomes tedious pretty quickly.

3. Adding data is annoying. I used to have one list in ocaml with all rest endpoints, but if you need more teams, or logical splitting across files then you need to write them by hand to be merged.

What is the solution?

I inevitably came to the conclusion that there needs to be a separate data language to what I want to do where abstractions can be represented simply and trivially.

Hereby, I introduce EdenDB with its own Eden Data Language (.edl file extension) (covered in the separate post).

So, the new architecture, instead of two layers has three layers:
1. Data platform, highest level of data defined with Eden Data Language
2. Codegen layer, to generate all typesafe abstractions from our high level data language
3. The final feminine plane layer, where the user just fills in the blanks programming against typesafe interfaces



The rest of the posts in this board will be about development and implementation of the version 2 of the pattern, it will become an open source project available to everyone.

Sup bois. Let's talk about some principle which explains why democracy cannot work ever work and inevitably one platform will emerge which will be unbeatable compared to others and will get all the userbase in the entire world eventually.

Things among things cannot ever be equal as explained in the world of inequality thesis. Also, in assumptions thesis I've demonstrated that more information equals more assumptions and more assumptions equals simplicity.

From this logic flow comes inevitable conclusion that to create utter and perfect simplicity you have to have all the information. From this we reach conclusion that we must have all of our information in one place so we could use it in any way imaginable, to check any assumptions and design decisions we want in our data. This is how the pattern was born.

This is all theoretical, and makes perfect sense once you understand the pattern, but someone who hasn't tried the pattern might not see it clearly at the first. This is why I love using real world examples in my posts to connect these theoretical ideas to reality to make them easy to understand for average developer. Let's talk about systemd.

The curious case of systemd

So, in a nutshell, unix bois were using init scripts to start their background services. They look horrible, are hard to maintain and are based on shell scripts. Thankfully, I've never written init.d scripts and thanks to systemd I will never have to.

Then came some guy which said the obvious thing, which you could easily deduce independently if you know the pattern and that assumptions lead to simplicity: if we know dependencies between services we can make boot order faster than with generic sysvinit scripts which start stuff at who knows when. So, basically, systemd units define services declaratively, as data. This is still far from the power of the pattern, which encompasses all infrastructure and every service within it (similarly like Nixops does, but not dynamically typed and easier to maintain and understand).

There was a huge pushback against systemd from linux bois. And usually I couldn't care less about what linux monkeys think about anything. They cannot understand to this day why they still don't have a normally functioning desktop linux distribution that would be adopted in the mainstream and why year of linux desktop is still a dream in a soyfaggot's mind. The problem in linux monkey's mind is that they cannot conceieve value in all things working together. For them it is always "this and this small service must work independently, doesn't know anything about anything, and somehow magically integrate to the system and stuff should somehow randomly work".

Anybody who used any linux desktop for some time usually brick their system after upgrades, installing debian, yum or other packages and God forbid they try to do any funny stuff with drivers once they start working. A package is expected to be thrown in the environment it knows nothing about, it knows nothing how system is already optimized and what is done to it and it somehow needs to figure it all out how to install itself without corrupting the system, and there is no downgrade option. That's why I think in linux no distro besides NixOS, which has downgrades, latest software, reproducible builds, has any right to exist and waste everyone's time.

The monkey minds who think linux desktop is great, except when it stops working after a year or two (depending on how tech savvy you are and how much you demand from your system) made a huge pushback against systemd, which solved practical problem of spinning up services in a decent way and faster boot order with dependencies. From the standpoint of the pattern, systemd is a super tiny step towards defining everything as data, has practical benefits of practically improving everyone's life with efficiency and service management. Systemd is definitely an improvement, yet degenerate linux pagan monkeys abhorred it, without even knowing why. Like dogs without understanding that bark at any passing person.

What I want to say with this? Systemd is a super small subset of the pattern, gives a few improvements in a single system. Good that it was adopted. But the endgame is the pattern and having all of the single companies data, as services, apis, databases, typed queues defined as data.

What will happen when an open source system will emerge that will have all the abstractions software developers need defined as data in a single codebase? I only do pattern for myself and don't open source it, because I wouldn't want imbeciles filling out pull requests for 100 features, making it generic and horrible - it is now perfect for me.

But inevitably, a day will come, where there will be one open source distribution of the pattern (similarly to serverless framework, but typesafe and not javascript garbage) supporting one database, one queue management system, one monitoring framework, one logging solution, one scheduler and one underlying operating system distribution (likely NixOS and its declaration files would be used as low level assembly in the pattern solution).

The day this day comes, 90% of devops/sre yaml and spreadsheet masturbators will become useless, 99% of pre deployment logical errors (like we have to have three or five zookeepers for quorum, or this data structure upgrade is not backwards compatible) will be caught by analyzing the data in the pattern. What will happen to worthless dynamically typed languages like ruby and javascript which must have monkeys maintaining them?

Incremental thinking

I remember some talk about go, improving garbage collection latency or something like that. Some guy spake about a cycle: we make efficient hardware. Then software becomes slow. We improve software. Then hardware becomes slow. We improve hardware. And so on and so on. We don't have to be as stupid as them. All that we need to do, to make drastic improvements and to see the endgame clearly - let's have everything as data. If we have everything as data, we become a small tech god in our own limited universe and we have ultimate power with the pattern. We can restrict any service talking to anything, its cores, what interfaces it sees - simply at compile time. No need to dynamically check access - it is restricted in the language, hence more efficient. If we have all as data, 90%+ abstactions/devops/infrastructure tools no longer make sense. They are hugely complex, fighting lack of information, they do incremental improvements at best, yet they cannot holistically take all the information and use all that power to get the best possible experience.

Linux monkeys are so far behind apple, which optimized their entire m1 chip to WORK TOGETHER with their software and give the best experience that it is not even funny. MuH fReE sOfTwArE mAdE bY dIvIdEd iMbEcIlE mOnKeYs wHeRe mUh sLiDeS dOnT eVeN wOrK aT tHiS tImE.

Don't be a monkey, don't zone into the small part of the system, treat it as a low level irrelevant swappable assembly (I'll make separate post about this assembly), think holistically, of making everything work together and you'll be miles ahead of every "eVeRyThInG iS a fIlE, i hATe sYsTeMd" unix imbecile.

Enter ye in at the strait gate: for wide is the gate, and broad is the way, that leadeth to destruction, and many there be which go in thereat: Because strait is the gate, and narrow is the way, which leadeth unto life, and few there be that find it. - Matthew 7:13-14

I, as usual, was arguing my ideas with some people. Recently I argued with a literal faggot (literal faggot).

The argument from my side is very simple and I'll repeat it 100th time - if you have your problems defined as data in one place you can analyze them any way you want and catch a lot of otherwise very hard to catch errors before production. You should always strive to represent your problems as data if you want to produce correct and working software from the get go.

And the arguments of the faggots always boil down to such things: we'll, some developer might not want to write SQL. Or, some developers prefer ruby. Or, data in database changes, statistics change or what not.

That is, they can always find some edge case, as if it is justification to throw out overall solidity of the system for one minute detail that ought to be brought down to our control eventually anyway.

Let me give you an example of what they do. For instance, Rust, as long as you write safe code you cannot have segfaults. Faggot can say "we'll, you write unsafe code and have a segfault in Rust, hence the idea of Rust is invalid!". You see what these people do? They would dismiss the entire idea of Rust because of edge case of writing unsafe code which ought to be avoided. They dismiss the idea of the pattern, which catches 90% of trivial errors before deploying something to production because of some edge case that can still sneak into 10%, which we didn't cover yet. But, given we know enough data about our domain, we could finish even that in the long run.

Let's entertain the analogy of a kings house. House of the king is clean, spotless, everyone is dressed in good clothes and everyone smells nice. Interior has very expensive furniture. Then comes some poor fellow from the street, with dirty clothes, dirty feet and smells like hell. What will the king do to such a man? To go in front of the king you need to look appropriately and follow the rules of the king's house.

The pattern, meta executable with our domain problems defined as data is the house of the king. We have certain rules. We enforce our contracts with typesafety so we wouldn't see these errors in production. We perform custom static assertions and analyses on our data. Hence why I use OCaml and Rust in our king's house. If we want to bring in the likes of Ruby in our house, this would be equivalent of bringing in the smelly, dirty bozo which takes a dump on our golden floor of our royal house. We cannot have that. How can we ensure that Ruby upholds our type signatures we set for database queries and queues if Ruby itself cannot enforce its own types? Ruby is a dirty bozo that doesn't even go into the shower to wash his clothes. Same with Python, Javascript. Why should we bring such languages into our royal house, and make our house smell like shit because dynamically typed languages cannot enforce the basic contracts, which we could generate, but which would never be enforced?

Hence, the argument - some developers are familiar with Ruby, hence we need to support it, is worthless. I could say, some bozo sleeping near the dumpster wants to drive a lamborghini. So what? The crowd of dynamically typed ruby/python/javascript faggots are not to be considered when making decisions, their votes mean nothing. If they wanted a vote in the system of the pattern, where all honourable participants of the system honour their contracts, they need to wash themselves (in javascript, for instance, they'd must use typescript), wash their clothes, humble themselves and ask admittance to the king's house.

Current software companies are like third world countries. No law, no order. Most developers are dirty, smelly bozo's. They schedule meetings about how to make two or more components work together, because typesystem is in the mind of the bozo developer and he has to remember if there will be errors (which problem you'd never have with the pattern, as compiler is thinking for you). Hence the development is slow and everyone is frustrated.

You see, it is very easy to defile yourself. A virgin girl can lose her virginity drunk to some school bad boy and become defiled for the rest of her life, whom no self respecting man will want to marry. Also, sluttery destroys morality of the nation as well (Do not prostitute thy daughter, to cause her to be a whore; lest the land fall to whoredom, and the land become full of wickedness. - Leviticus 19:29). It is very easy to spray graffiti on the wall of your house making it look like it was vandalized by irresponsible teenagers that didn't receive whooping when growing up. It is very easy to do things wrong and go out of the way and make everyone else around you to suffer. Like Jesus said, broad is the way of destruction, but narrow is the way that leadeth unto life and few will find it.

Hence, we must separate ourselves from the world and its insanity, same principle for the church and same for the software development.

Wherefore come out from among them, and be ye separate, saith the Lord, and touch not the unclean thing; and I will receive you. - 2 Corinthians 6:17

Love not the world, neither the things that are in the world. If any man love the world, the love of the Father is not in him. For all that is in the world, the lust of the flesh, and the lust of the eyes, and the pride of life, is not of the Father, but is of the world. And the world passeth away, and the lust thereof: but he that doeth the will of God abideth for ever. - 1 John 2:15-17

Here's the illustration of what I mean. There's the sea of wrong, and developer, if he wants to develop in the sea of wrong he uses services like swagger declarations of http requests and then writes them by hand in his service. He uses non typesafe rails database, writes tests, hopes it works. He uses random data format when interacting with queues - millions of places where trivial mistakes can be made.



Below, in the sea of wrong, we have our royal built house with the pattern. We have few abstractions, like typesafe interactions with postgres database, typesafe interactions with queues, typesafe calls to other REST API services. We label queries if they are mutating the database, or are readonly. We can have stress testing of certain postgres queries which will be generated since we have our queries as data. We automatically create prometheus variables for every single query we have. We store every single query ever executed against the database, with their arguments, in clickhouse table for analyzing. We have our typesafe, efficient, binary format logging system, where we could generate how those will be represented in the frontend.

In our house, there's one law, one order, one way to do things. We only allow typesafe languages in our house. We have fewer, but super solid abstractions that ALL HAVE TO WORK TOGETHER in the same ecosystem and we don't need to support hundreds of databases/queues and so on. We do not write yamls (unless generated from typesafe config). Our configurations are typesafe enums where, for instance, I don't specify a port of kafka service, I specify with enum which cluster of kafka service talks to and all ports must be defined for every cluster and are checked if they are valid, cannot clash on the same machine and so on.

To build the house of the king took time, but once it is built, you cannot imagine you were a dirty bozo once living in the street and digging dumpsters once.

Have a nice day bois.

.. is by saying it is generic. And we'll cover a lot of examples that reveal these hidden inefficiencies to the naked eye.

Lets start with foundations, a CPU. x86 or ARM is a generic compute machine. Generic compute machines will never be as optimal for special tasks as dedicated ASICs. Bitcoin mining first happened on CPUs. Then, since GPU aligns more specifically with the problem of massive parallel computation people mined with GPU's. GPU's are, again a generic compute device rather made for rendering games, so the next step were creating ASIC's. Well, there's a bus stop I between GPU and ASIC I skipped called Field Programmable Gate Arrays (FPGA), but I don't know if anyone tried programming FPGA's to mine Bitcoin. Anyway, the moral of the story is that CPUs/GPUs are a generic devices that will always be beat by specialised hardware. I just spent a paragraph to say that the sky is blue, so what?

Think about a world before GPU's, FPGA's and ASIC's. It wasn't that obvious then was it? And rest assured, we will challenge a lot of de facto ideas that we will predict disintegrate in the future.

Memory allocators.

Today a lot of people spend time developing memory allocators. ME's are implementations of a generic interface of malloc and free. Think of all the possible questions you have to ask yourself when implementing malloc. Does user want a big chunk of memory or small? How often is memory freed? Does memory travel between threads? The implementors cannot have the slightest idea, because such information is not passed to the function, hence they have to make generic approximations. That is, allocators code is full of if's and else's draining precious CPU cycles trying to fight this lack of information with dynamic analysis. Dynamism is cancer too, like I mentioned in another post.

There are standard operating system allocators, then jemalloc, now mi-malloc seems to be competing for the title of the "fastest" allocator. In reality they're just better approximating to the needs of the current time real world software.

But, for instance, if a routine uses small objects we could implement a simpler small object allocator and beat the generic one. Same with big objects. Same with objects that don't need to ever be freed. Specialised allocators will always beat generic ones.

Today in Rust you pick one allocator for everything and tada... I predict that in future compilers will perform static analysis like a skilled programmer and will pick specialised allocators for routines instead of one generic allocator for everything like jemalloc, mi-malloc and so on.

So, we just murdered the today's untouchable holy cow of memory allocators and this truth will eventually become obvious in a few decades to everyone. What's next?

SQL databases. Let's pick on generic solution of query planning. What happens today is, there's dynamic table of metadata about the data in the DB. If someone makes a query the DB has to parse that statement, analyse validity with the metadata, optimise the query with table, stats and so on.

Accepting a string is a generic solution. We must spend time parsing, analysing, handling errors in our precious runtime.

How about we move information about DB tables and queries to the compile time? After all ruby monkeys write tests to be reasonably sure about DB interactions before deployment to production.

What if database tables wouldn't be binary blobs in some generic filesystem format but rather a programing language constructs?
What if your DB was a generated code of a single object where tables were hash map arrays and your planned queries are simply exposed methods that you call and have typesafe results?
What if by defining the queries you need beforehand you could eliminate all queries that don't hit indexes and never deploy that?
What if, after analysing queries for tables you would determine that multiple tables are never queried together and can be serviced by multiple cores without even needing any generic locking mechanisms?

I'd love to use such DB instead and I will probably implement one with the pattern...

Another sacred cow is dead. Let's pick on Haskell faggots!

Smug Haskell academic weenies say "everything must be pure". Guess what? Making everything pure is a generic solution. Haskell crap because of that has to copy gazillions of objects and be inefficient, this greatly complicates the development of software projects with this impractical puzzle language.

Think of Cardano, now soon to be dead shitcoin that didn't deliver anything of practical value since 2017, having four years plus of a headstart versus projects like Avalanche, Polkadot and Solana, which use practical languages like Go or Rust and already have huge ecosystems. Meanwhile ADA Haskell faggots are just beginning to optimise their 8 tps smart contract platform. Even if they do optimise and become sufficiently usable (but will never reach speeds of Rust of Polkadot and Solana) their competitors will already be huge and nobody will need yet another blockchain.

So much for imposing a generic solution of purity to your project and destroying your productivity and develoment efforts.

The more specific solution would be controlled side effects. You mark a function pure and it can only use pure functions. I imagine a programming language will emerge, or maybe provable ADA already does this, where you can query and prove the code, that it does modify variable, is pure and so on.

Third cow done. Let's get serious: Operating systems.

Think of linux and the crazy amount of generic stuff it supports. I can guarantee that there is a ton of performance left on the table because of generic linux interfaces.

For instance, Terry Davis (RIP, you genius hacker soul!) wrote temple OS which specialised in games and in his OS he could perform a million context switches versus thousands of linux. That was because he coded a specific solution and circumvented lots of wasteful abstractions that are simply not needed in his context.

So far linux seems to suffice most applications with adequate performance, but I predict more and more unikernels will appear that will function as faster databases, load balancers, search engines and so on and so on. And unikernels will also eliminate many of generic system attack vectors, such as unintended shell access.

We could go on forever and ever about things that are generic and now are costly because of that, but I don't have infinite time. Simply, if you see inefficiency, or encumberance, ask yourself, is it because, for instance, XML, JSON or YAML is generic? If something is, how could it be made specific? Maybe these could be specific, typesafe language constructs?

And you'll see angles you have never considered before.

By the way, nature seems to be created with all hardware, no generic intel chips. Specific bones, eyes, nails to every animal kind. So that speaks volumes of the infinite wisdom of our Creator!

Have a nice day

I want to talk today about the responsibility and competence spectrum permeating all of the nature below:



Okay, so, on the very left there is utter masculinity. Think from the God's perspective, absolute power, absolute wisdom. God knows everything about everything and doesn't need anyone, everything that someone gets from God is by his grace toward that someone. This is the pefect, most possibly masculine image of God.

On the utter right there is utter femininity. A beautiful, submissive woman that is led by a masculine husband from the very left of the spectrum. If we talked about God this would be submissive followers, but principles are the same. Designs on this earth are fractal and self similar to patterns in heaven. What is on earth is a smaller copies of what is in heaven.

On the middle of the spectrum, where, in ideal world, no one should be, are either masculine, self sufficient women (disgusting) or weakling, faggity, feminine men (disgusting). These examples occour in nature, for instance, certain spiders where small, weak, feminine man is worthless and is eaten by a female after sex. Needless to say, God potrays these examples of how things ought not to be done with man which is made in the image of God.

Why is that? Why such opposite extremes? Why much more is required from masculine spectrum, to lead and know, to have strength and take care of the feminine spectrum?

Like we talked about generic vs specific thesis, to make the simpliest possible solutions you have to have all assumptions. To have the most possible assumptions you have to have all the information. Only such entity, that has all the information and all assumptions can make perfectly simple design decisions and impose simplest possible interfaces for the feminine plane to use them.

That is what meta executable in the pattern achieves - it has all the high level information about the project. REST Endpoints, queries, periodic tasks, all databases, their indexes, frontend elements etc. etc. Once we have all this data in meta executable, what would otherwise be a complex flow of getting/approximating this dynamic data from something somehow (like querying a database of executed query logs to analyze if queries are hitting the indexes - we're way too late, we could already be writing post mortems by now) now becomes trivial to solve as we have all the data we need defined in the project compile time in the meta executable.

All the dynamic metrics analysis systems from our standpoint are just generic abominations, which will never do the perfect work as we spake in generic vs specific thesis. They're neither fully encompassing, masculine, godlike enough, so they could provide everyone with perfect solution without extra integration code (they could, if they were part of the masculine plane executable in the pattern), neither they are perfectly simple, where you could provide typesafe API for them as consumers of the feminine plane. They're neither here nor there, hence, cannot be in the perfectly simple and beautiful feminine plane (like a beautiful, feminine woman), neither can they belong to the perfectly rational and all knowing and encompassing, infinitely powerful masculine plane.

Consider logging systems of today, elastic search + kibana. For the application logging all that is given is text. Output a line, then parse, possibly using complex regex who knows what. These will end up as separate parsed fields in kibana. Then make a kibana dashboard for every new crap app that comes out interpreting those plaintext messages of anything. And the work is never ending and certainly not perfect. This is because today's leftist faggots software devs cannot think straight, they provide generic output from the application and generic way to analyze that data which brings in bunch of complex caveats because it is not a specific solution.

Let's think about this problem from our perfect godlike meta executable, what we could accomplish with it instead?

1. Let's define every log message by type for every app in the masculine plane
2. Let's generate CapnProto schemas for every log entry and log only very small binary data that is the essence of the log message, using very little bandwidth in Kafka.
3. No more text, only possibly description next to log entry in the meta executable. Or, in meta executable we could specify formatting message how to visualize it for humans.

   Say, log message is how many eggs for what price someone bought in a store:
 
  struct LogMessageOfBuyingEggsInAStore {
    user_id @0 :UInt32;
    number @1 :UInt32;
    price @2 :Float32;
  }

  And we could provide a template string in the meta executable, with log entry of how to format this minimal, non text data in some GUI output:
  "User {user_id} bought {number} eggs in a store for price {price}"

  We could check that all of that is logical and works in compile time, in the meta executable.

  We could check that template variables exist for this log message, that we don't use all message variables, and so on and so on.

4. We could generate frontend to perfectly display our log messages according to the message types, no need to parse text.
5. Messages logged with perfectly typesafe function from the interface user in the feminine plane:

api.log_buying_eggs_in_a_store(price: 1.23, quantity: 7)


6. We could even attach alerting if we wanted to the log entries, if some error is really bad, we'd get an alert.
7. Since messages are CapnProto we'd spend 0 CPU time parsing them when dumping all them to ClickHouse

See how much more powerful solutions we can come up with that no soy leftist faggot could ever think of just by giving more power to the masculine plane where it rightfully belongs? And simple, feminine plane would only get typesafe functions on their api to log these entries? And leftist faggots tend to have the idea that something is either efficient or elegant, here we get both, efficiency with CapnProto and elegance of defining very high level abstraction for log entries where we don't even need to write more code once we developed it and things just work by adding more log entry types!

Just like a beautiful feminine woman could feel weak, doesn't need to have much wisdom inside the arms of a masculine husband who takes care of her and the family.

Like I mentioned, today, most software developers stand in the middle. They know little a bit of masculine plane, technical details, yet they feign utter ignorance when matter comes to making all things work together. For them it is unforgivable sin that database schema, queries and transactions should live together in a single codebase where they could be perfectly analysed together! Such faggots spend decades in meetings, figuring out the most complex possible flows in a rails codebase, where queries need to be checked for correctness, but also all the logic of interacting with the database must live with rails too because muh precious ruby faggots say "it would be unproductive to develop if we don't know database schemas". As if productivity is possible in the long run when developing with dynamically typed programming language. And craziest abominations are conceived, ultimate job securities that are then implemented for multiple years with no end in sight because more and more dynamic problems keep popping up.

The alternative is the pattern, all database schemas live in meta executable, all queries and transactions live in meta executable, everything is checked before the production with possibly enabled parallelism for performance and only if everything is ok, only then typesafe user API's with implementations are generated and everyone lives happily ever after. But no, let's create unmaintainable, fragile, complex abominations instead.

Like I mentioned time and time again, paganism is a mental disorder. God mocks feminine men in nature, think of all the spiders where male is small and female is huge and eats spider after sex. Such men, such faggots, if they don't fulfill the natural responsibility of calling the shots and leading the family have no purpose as nature shows us and do deserve to get cheated on, do deserve a used up slut of a wife, do deserve to ignorantly raise not their own children conceived by a cheating wife, do deserve to be only used for their money and do deserve the divorce court. These faggots are trying to enjoy in front of their women what only feminine plane was intended to enjoy - being weak, being incompetent and women ruling over them. And woman, who tries to stick her nose into a masculine plane, being the bread earner, calling the shots, being semi-smart (just a little bit smarter than feminine woman in reality, but will never reach the levels of the greatest inventors) does deserve to get smashed and dashed, does deserve to get her heart broken by the chad train that runs through her day and night in her youth, does deserve faggity, worthless, unattractive, feminine husband at the end of her youth and does deserve bearing weak and unstable children, her daughters being raised as worthless sluts and her sons raised feminine faglet losers.

None of these two variants, feminine man or masculine woman will ever be a good solution. Most software developers today fall in the middle, in the abomination which shouldn't exist. Hence, innovations are slow. They are a little bit smarter than the general population, but from the standpoint of a masculine plane all of them are insignificant small turds who will never develop anything decent.

Just think of Nikola Tesla who invented our entire modern electrical grid. After Tesla electricity innovation has stopped, same things that were used back then when he invented them are used today. Tesla knew it all about electricity, he was the hardcore innovator and brought most of these advancements to us. No wonder his labs were raided by secret agencies. But think about what that means - today software developers are brainwashed with the idea that no, you can't be the single innovator, innovations come when lots of "smart" people put their stuff together out of pieces and everyone is supposedly needed to make breakthroughs. Reality is much less romantic, you get miles ahead of every imbecile, you become prodigy, and obvious, beautiful, simple, much more advanced solutions just keep coming to you because everyone around you is a braindead and brainwashed monkey damaged by a few years of academia.

When I was in a university, there came a subject about God object that knows everything and it was told that people shouldn't do that. You're instantly told you should not strive to be similar to God. Would God be pleased if people, who he claims were created in his own image, would try to be similar to God? Of course he would be pleased if every man tried to fulfill his image. Yet, for monkeys in university it is told to do just the opposite! What are the odds! The pattern, which could easily solve most complex problems in the most concise and elegant way, given that you represent all your problems as single database of data (the dreaded God object), where you could easily analyse and check everything for consistency is discouraged from being used so software devs would keep being miserably stupid with 99 programming languages they have to learn and drowning in problems of diverse dynamic garbage interoperating with each other.

That being said, make sure masculine plane figures everything out, so there would be utter simplicity and beauty in the feminine plane, just as it is a natural and the perfect divine way to do this as is evident in all of creation.

Good day bois!

Today I want to talk about another issue leftist software developers today mess up. The wording "Proudly found elsewhere", "Humble thyself and reuse", "Trust libraries written by experts" etc.

I want to explain why this approach becomes meaningless the more mature and old your project is.

You see, when you want to start a project you are attacked by countless choices for a web framework, databases, queues, infra management tools and what not. Examples seem sweet, "Look at this common case, so little code, so shiny! Use me, this framework and be productive and happy!". Every framework offers its own abstractions which it deals with and shows these as beautiful.

Devil is in the details. Now ask yourself, are developers making their frameworks make them to you? No, they make it for a group of people. And that group of people can vary wildly in what they use. Hence, any third party framework can only be generic to the lowest common denominator which satisfies a subset of people. We discussed in specific vs generic thesis how generic things will never be perfectly simple since they are not specific enough for the problem at hand. You can cut salami, wood and bang nails with an axe, but axe would be a generic tool. What you want specifically for the most convenient job is a knife for salami, axe for wood and hammer for nails.

So, unsuspecting developers get lured into yet another web framework with all its subtelties and another baggage of knowledge needs to be established in the brain until you become productive. To make matters worse, if the company becomes successful, it becomes painstainkingly obvious that the framework was never intended for multimillion lines of code, like ruby on rails. Having to maintain millions of lines of code written with dynamically typed language would cause me to become suicidal. But it just becomes too late, and people start huge projects, to split their dear monolith, no refactorings are ever guaranteed to succeed in ruby and the little framework that started so shiny with "look ma, so little code" becomes unbearable, bug ridden burden to the entire company, simply because, people who started with the framework didn't know any better, had no hindsight, and the entire company has to suffer because of early poor technology choices.

What is the alternative you ask? Glad you asked! The alternative, is going to the complete opposite spectrum, which is the pattern. Owning most of your abstractions, like infra management tools, your web framework, your frontend reactive framework. And that sounds crazy at first glance, to someone who doesn't know the power of the pattern and how it makes all of your abstractions much simpler then you'll get anywhere else just because they solve more specific problem to your needs.

The only reason I'd ever pick a generic framework like rails is for short term projects, or prototyping, if you want to get up to speed to see if your project can be successful to begin with. Then, as soon as I know that we're dealing in the long term game I'd start working on the pattern solution for the company ASAP.

You see, it is not obvious to start with the pattern, to own your abstractions. But once you get there, and you have a successful company with the pattern, and you deliver stuff so fast and correctly that everyone's head is spinning and you solve such mountains of problems which little strokes of code that work with your well defined data - no generic framework will ever be able to compete with that.

Which generic framework will offer you at a stroke, in few lines of code to, say:
1. declare a new application
2. declare which typesafe queues it will use
3. declare which typesafe database interactions it will perform
4. have it instantly all monitored according to the abstractions the app uses
5. deploy it in production and it interacts with the rest of your ecosystem correctly the first time

The answer is none, all needs to be done by hand if you take up some used up third party bycicle. And it is error prone and unproductive. The pattern can do all that, but your typesafe rock solid abstractions have to be built, which takes some time for your specific solution, but the benefits will be enjoyed forevermore. You can even have your team names in the pattern and assign enum on which team is responsible for what service. The sky is the limit of what can be done with the pattern.

And every reasonably sized company ought to have their own meta executable with the pattern with their specific use cases. For instance, the pattern of a 3d game developer will be radically different from the pattern of web application shop company. But in their own right, since their meta executable is specific to them, you are then free to not bend to the wills any framework tries to impose on you, but what will be best for the company.

Imagine checking just in one executable that entire network that spans any number of datacenters that no switches have loops, that there are no duplicate ips, that there are no duplicate ports on any server you have, that on any given server you do not exceed declared RAM capacity for what applications will be running. Imagine enforcing via iptables rules that service a can only talk to service b only with typesafe interfaces. All ran in few milliseconds, without spinning up servers in the cloud, just for the fact that you have your problems defined as data and you can reap all the benefits of having complete and total control and knowledge about your entire infrastructure. Don't like some abstraction? Delete and refactor your entire infrastructure at once and be done! No need to synchronize and think about 5 different endpoints that will not work once you shutdown the service.

This is unbeatable. People narrow down their projects like nutjobs, they create service to control dns but it doesn't work with anything else. Then they create more crappy, narrow scope services for orchestration, for network proxy, for CI and nothing ever works together and everyone needs to patch that garbage day and night and praying everything will work.

Why not have all that data, about your services, about your network, infrastructure, databases, their tables and their queries in one place where meta executable could ensure everything is working perfectly together? Leftist degenerate mindsets of today will never figure it out on their own.

So, for any longer term project, you should build the pattern with specific abstractions to you and have perfectly bonded infrastructure to your company. The initial cost will be greater than starting with out of the box framework, but once you get through a steep start of building abstractions you'll have absolute control over your project and abstractions that blow out any third party garbage out of the water. The start with the pattern is slower but the rewards are infinite.

I'll just illustrate what productivity looks with the pattern as the project ages and matures versus taking some garbage off the shelf which was never customly tailored for your company specifically



Have a good day, bois.

This topic has been moved to Development Tips.

https://mycodingcult.com/index.php?topic=16.0

This topic has been moved to Development Tips.

https://mycodingcult.com/index.php?topic=15.0

This topic has been moved to Development Tips.

https://mycodingcult.com/index.php?topic=17.0