Recent Posts1
The pattern V2 / Eden DB Improvements 3
« Last post by CultLeader on April 27, 2024, 10:29:26 AM »Sup bois, I've added extremely powerful feature into EdenDB: OCaml data modules. What this allows to do is to define data in OCaml using typesafe generated constructs.
I did this a while ago in this commit https://github.com/cultleader777/EdenDB/commit/9270c3e42ab28410e8ba3018f76b4eea0b586da4 , just didn't get around to describing it yet until now.
Motivation? Eden data language is fast to parse but will never be as powerful as standard typesafe programming language.
For instance, if I want to define 10 similar nodes in a cloud, doing it with OCaml is trivial, run a loop and you're done.
People write an utter abominations like jsonnet, or they add hacks like count variable in terraform or for_each in terraform.
In reality we just need a good programming language to define our data and we don't need to learn useless garbage like jsonnet, or how to program in terraform (which is nightmare). Terraform output is just an assembly compiler later, we don't deal with it directly.
Here's an example.
Say, we have main.edl file
And we have OCaml file at some-module/bin/implementation.ml
So OCaml allows us to define table data in OCaml. We didn't write mk_test_table function (generated). define_data function is required in the generated code. There's a some_text column that has default value if unspecified. If your OCaml file implementation mismatches schema it's a compile time error.
To someone who's used to the standard implementation of compilers that do very boring stuff, linking executables from explicit outputs this might not be trivial how it works. In fact, I don't recall any compilers that do anything similar to what is done here. If codegen step is performed, it is usually performed in some Makefile task before compilation.
So, this is how it works:
1. EdenDB compiler parses all .edl sources with includes
2. We first process table schema and their interconnections, any error there (non existing foreign key column and etc.) is EdenDB compiler error
3. Now we know all the schema that will be in EdenDB, we just didn't insert any data yet and didn't check foreign keys, constraints and etc.
4. At this point we actually run OCaml codegen for the OCaml data module defined at some-module.
We generate: dune, dune-projec, context.ml, context.mli, db_types.ml, implementation.mli, main.ml and dummy implementation.ml if it doesn't exist yet.
User is responsible for modifying implementation.ml file with his data, he can have other OCaml files with utility functions and everything.
We assume that we use dune for OCaml.
5. Now we insert the EdenDB data that is defined in EDL
6. At this point, EdenDB compiler runs dune exec for data modules, reads json dump of defined data, and inserts data into EdenDB state according to the schemas
7. The rest of EdenDB checks are performed, like checking duplicate primary keys, foreign key existance on all of the data, whether it is defined in EDL, Lua or OCaml
I haven't seen any compiler do something like that so I thought it is interesting design. And I'm 99% sure, being 70k lines of code into Eden platform that this is how I will define my data at the end of the day.
So, Current eden platform pipeline:
1. Run EdenDB compiler, which runs OCaml data modules and defines data of all of our infrastructure. Basic database correctness checks are performed here
2. Eden platform analysis step, here we perform all the checks regarding our infrastructure, say if defined node ip belongs to the DC subnet, that we have 3 or 5 consul servers per DC and so on
3. Codegen step runs which compiles the rest of our project, including provisioning, postgresql migrations, nix configs, and gazillion of other stuff. There are files that user can edit manually in Eden platform in their project directory (mainly business logic of EdenDB applications).
Have a good day bois!
I did this a while ago in this commit https://github.com/cultleader777/EdenDB/commit/9270c3e42ab28410e8ba3018f76b4eea0b586da4 , just didn't get around to describing it yet until now.
Motivation? Eden data language is fast to parse but will never be as powerful as standard typesafe programming language.
For instance, if I want to define 10 similar nodes in a cloud, doing it with OCaml is trivial, run a loop and you're done.
People write an utter abominations like jsonnet, or they add hacks like count variable in terraform or for_each in terraform.
In reality we just need a good programming language to define our data and we don't need to learn useless garbage like jsonnet, or how to program in terraform (which is nightmare). Terraform output is just an assembly compiler later, we don't deal with it directly.
Here's an example.
Say, we have main.edl file
Code: [Select]
DATA MODULE OCAML "some-module"
TABLE test_table {
id INT PRIMARY KEY,
some_text TEXT DEFAULT 'hello',
}And we have OCaml file at some-module/bin/implementation.ml
Code: [Select]
open! Context
open! Db_types
let define_data () =
mk_test_table ~id:123 () |> def_test_table;
mk_test_table ~id:123 ~some_text:"overriden" () |> def_test_table;
()
So OCaml allows us to define table data in OCaml. We didn't write mk_test_table function (generated). define_data function is required in the generated code. There's a some_text column that has default value if unspecified. If your OCaml file implementation mismatches schema it's a compile time error.
To someone who's used to the standard implementation of compilers that do very boring stuff, linking executables from explicit outputs this might not be trivial how it works. In fact, I don't recall any compilers that do anything similar to what is done here. If codegen step is performed, it is usually performed in some Makefile task before compilation.
So, this is how it works:
1. EdenDB compiler parses all .edl sources with includes
2. We first process table schema and their interconnections, any error there (non existing foreign key column and etc.) is EdenDB compiler error
3. Now we know all the schema that will be in EdenDB, we just didn't insert any data yet and didn't check foreign keys, constraints and etc.
4. At this point we actually run OCaml codegen for the OCaml data module defined at some-module.
We generate: dune, dune-projec, context.ml, context.mli, db_types.ml, implementation.mli, main.ml and dummy implementation.ml if it doesn't exist yet.
User is responsible for modifying implementation.ml file with his data, he can have other OCaml files with utility functions and everything.
We assume that we use dune for OCaml.
5. Now we insert the EdenDB data that is defined in EDL
6. At this point, EdenDB compiler runs dune exec for data modules, reads json dump of defined data, and inserts data into EdenDB state according to the schemas
7. The rest of EdenDB checks are performed, like checking duplicate primary keys, foreign key existance on all of the data, whether it is defined in EDL, Lua or OCaml
I haven't seen any compiler do something like that so I thought it is interesting design. And I'm 99% sure, being 70k lines of code into Eden platform that this is how I will define my data at the end of the day.
So, Current eden platform pipeline:
1. Run EdenDB compiler, which runs OCaml data modules and defines data of all of our infrastructure. Basic database correctness checks are performed here
2. Eden platform analysis step, here we perform all the checks regarding our infrastructure, say if defined node ip belongs to the DC subnet, that we have 3 or 5 consul servers per DC and so on
3. Codegen step runs which compiles the rest of our project, including provisioning, postgresql migrations, nix configs, and gazillion of other stuff. There are files that user can edit manually in Eden platform in their project directory (mainly business logic of EdenDB applications).
Have a good day bois!
