Eden DB Improvements 1

[CultLeader]

Sup bois, some improvements were made to the EdenDB to allow it to be much more flexible than standard SQL.

Basically, I was experimenting of how to represent domain problems as data, and I had to start with the servers.

Code Select Expand


TABLE server {
  hostname TEXT PRIMARY KEY,
}


Then I wanted to solve the infinite problem - how to check that no server has duplicate ports.

But ports are used by many abstractions, say, docker containers, systemd services and so on.
How do we ensure that different abstractions that use ports do not take duplicate port on the same server?
I kinda fell little short.

Let's see how reserved port table looks.

Code Select Expand


TABLE reserved_port {
  number INT PRIMARY KEY CHILD OF server,
}


Port is child of server. Now, say, we have docker containers. How do we refer to the reserved ports?

Before improvements shown in this post this is invalid

Code Select Expand


TABLE docker_container {
  name TEXT PRIMARY KEY CHILD OF server,
}

TABLE docker_container_port {
  port_name TEXT PRIMARY KEY CHILD OF docker_container,
  reserved_port REF reserved_port,
}


We couldn't refer to a reserved port because there's no unique key.

But then I thought, wait a minute, if both elements are ancestors of server, why the uniqueness context for port must be global?

If you think about it, the uniqueness context for such table that is child of anything ought to be the common ancestor. And if we imagine that every tables common ancestor is imaginary root table, then it makes perfect sense - everything is unique in the root table context.

This is how current topology looks visualized:


So, basically, the concept of a foreign key would be much more useful, if REF to another table would refer to the uniqueness context of the parent of the referred table.

So, having many different abstractions we can reserve ports in the single server so they would never clash and we find out about this before production when some systemd service can't start!

Let's define some fictional docker container running on some server with reserved port:

Code Select Expand


DATA server {
  epyc-1
}

DATA STRUCT docker_container {
  hostname: epyc-1,
  name: postgres
  WITH docker_container_port {
    port_name: main_port,
    reserved_port: 5432
  }
}


This so far does not work because of an error that reserved port does not exist in server:

Code Select Expand


NonExistingForeignKey { table_with_foreign_key: "docker_container_port", foreign_key_column: "reserved_port", referred_table: "reserved_port", referred_table_column: "number", key_value: "5432" }


We must reserve the port also.

Code Select Expand


DATA reserved_port {
  epyc-1, 5432
}


Now stuff compiles. And we cannot add more `reserved_port` with same value or stuff fails, if we copy reserved_port statement once again, boom:

Code Select Expand


FoundDuplicateChildPrimaryKeySet { table_name: "reserved_port", columns: "(hostname, number)", duplicate_values: "(epyc-1, 5432)" }


However, there's an issue that we would need to repeat reserved_port statement along with defining docker containers. Which leads to another improvement done to EdenDB: add rows with lua.

Basically, instead of defining this data by hand, now we can run lua code and insert to any table we want:

Code Select Expand


INCLUDE LUA {
  data('reserved_port', { hostname = 'epyc-1', number = 5432 })
}


And, of course, you must refer existing table and you must only define existing columns or compiler yiels at you.

`data` internal edendb function implementation is so trivial that I'll simply post it here:

Code Select Expand


__do_not_refer_to_this_internal_value_in_your_code_dumbo__ = {}
function data(targetTable, newRow)
    local queue = __do_not_refer_to_this_internal_value_in_your_code_dumbo__

    if queue[targetTable] == nil then
        queue[targetTable] = {}
    end

    -- we simply accumulate values to insert in lua runtime and then process
    -- them in one go in rust
    table.insert(queue[targetTable], newRow)
end


We simply have a global queue of rows to be inserted from lua and we iterate this map and insert data defined in lua in one go.

So, now, to avoid repetition we can simply define lua function that defines a docker container by inserting into all appropriate places we want:

Code Select Expand


INCLUDE LUA {
  function docker_container(hostname, containerName, portName, portValue)
    data('docker_container', {
      hostname = hostname,
      name = containerName,
    })

    data('docker_container_port', {
      hostname = hostname,
      name = containerName,
      port_name = portName,
      reserved_port = portValue
    })

    data('reserved_port', {
      hostname = hostname,
      number = portValue,
    })
  end
}


Data order doesn't matter, all checks are processed only once all data has been inserted into tables.

And we just call this function once in lua of defining postgres container, while encapsulating complexity, and deleting previous data definitions for this docker container:

Code Select Expand


INCLUDE LUA {
  docker_container('epyc-1', 'postgres', 'main_port', 5432)
}


Of course, I wrote a terse funtion as an example, in production likely there will be named map arguments with many default parameters provided internally in the function to make this more practical.

To be more practical, we include lua files for library stuff (this feature already existed):

Code Select Expand


INCLUDE LUA "lib.lua"


Full working gist can be found here https://gist.github.com/cultleader777/175fb37cb2ce1c9cb317ce2d187a926b

So, these are the two recent improvements that gave eden data language quite more power:
1. allowing foreign keys to refer to other child rows in the entity
2. defining data from lua, you can script anything you possibly want

Good day bois.