My Coding Cult

In this post I will reveal most valuable programming pattern I know. It's extremely solid, versatile and opens limitless possibilities. I've discovered this independently myself, because I knew that is the way the LORD had to create everything. There is no other way to do things (everything before this is a wrong way), and it will be clear why. It is so deep, its core goes to the foundation of both sexes, male and female, masculinity and femininity and what and why that is at all.

Show me any programming language feature you know without which you cannot live and it all pales in comparison to this pattern. It is irrelevant and useless and has no meaning compared to this. In fact, I can do any language feature you ever knew with this pattern while host programming language only needs basic features.

You don't even need to practice it, it will be obvious - there is nothing that you will not know about this pattern that I know just reading this post.

If you only read single post in this forum, it better be this post.

Okay, hype speech is over, let's go.

Represent abstractions as data

What does that mean? Sounds vague marketing sales bullshit, let me explain it very simply. Instead of connecting to the database and sending the query from code, we define query as data in a program that will generate typesafe call to a function. Here is OCaml example of what I mean:

let res = execute_write ~params:[|string_of_int voting_id|] pg "
  UPDATE voting_session
  SET signatures_count = signatures_count + 1
  WHERE voting_id = $1
  AND NOT is_scheduled
  AND NOW() < petition_deadline
  " in

This code block executes SQL query in Postgres and you pass parameters and convert them to strings by hand.

Every single time you write such block you are risking of making a mistake. Not nice. Ruby monkey answer to this? "Just write tests!". Oh yes, If I was idiot Ruby monkey I'd see no issue with writing tests for every single most trivial code block for everything.

Here is the alternative:

let db_queries = [
...

  mk_db_mutator
    FtVoting
    "increment_voting_signatures_count"
    {|
    -- voting_id:15
    UPDATE voting_session
    SET signatures_count = signatures_count + 1
    WHERE voting_id = <voting_id:int>
    AND NOT is_scheduled
    AND NOW() < petition_deadline
    |};
...
]

I defined exact same query as data in a list of other queries. mk_db_mutator function just returns struct of this query.

This data resides in a meta executable that generates OCaml code for a typesafe function:

let dbm_increment_voting_signatures_count (pg: Postgresql.connection) ~(voting_id: int) =
  incr dbm_increment_voting_signatures_count_init_count;
  let timestamp = now () in
  send_db_query_record ~timestamp ~query_label:"dbm_increment_voting_signatures_count" ~duration:(-1.0) ~rows_affected:0 ~rows_returned:(-1) ~arguments_keys:["voting_id"] ~arguments_values
:[string_of_int voting_id];
  let res = execute_write ~params:[|string_of_int voting_id|] pg "
    UPDATE voting_session
    SET signatures_count = signatures_count + 1
    WHERE voting_id = $1
    AND NOT is_scheduled
    AND NOW() < petition_deadline
    " in
  let duration = now () -. timestamp in
  let rows_affected = int_of_string_opt res.result#cmd_tuples |> Option.value ~default:(-1) in
  incr dbm_increment_voting_signatures_count_success_count;
  incr_float dbm_increment_voting_signatures_count_duration_sum duration;
  send_db_query_record ~timestamp ~query_label:"dbm_increment_voting_signatures_count" ~duration ~rows_affected ~rows_returned:0 ~arguments_keys:["voting_id"] ~arguments_values:[string_of_int voting_id];
  res

So, compilation of codebase goes like this:
1. Meta executable runs, performs all checks, and if everything is okay emits code
2. The rest of my codebase compiles, and uses typesafe functions

Not a big deal? Okay, let's go through all the things that one small query has checked when it is defined as data:

1. It has to work, mutate data in the table, it is run with example data against test database in mind
2. We check that it does mutate a row
3. <voting_id:int> is substituted and added to a typesafe function, I can add arguments in any places I want
4. Code is generated to save EVERY SINGLE QUERY in Clickhouse WITH THEIR ARGUMENTS in a NICE MAP. I have every single query ever executed against the database, well compressed, with their parameter names and values in Clickhouse.
5. Prometheus global variables are generated and incremented and automatically exposed to prometheus.
6. Every single query is assigned to some feature enum, so, if I delete features logically I have to go through all the code and see where it is used
7. If I change query signature, I will have to refactor code where I use this function.

Imagine, I just write very little code, of defining query as data and all this code is generated and all these things happen on autopilot. I have confidence that queries will work and things work 99% of the time. I barely write any tests either.

My mistake factor on producing invalid queries is now drastically reduced, and I can write very complex queries and know things will mostly work.

Imagine you're at work, and someone says, we have to label and monitor all queries in the database. Find places where we query database and make sure all is tracked.

Well, you could be lucky, attach to the framework, but what if you have lots of code that just uses SQL library which has no concept of tracking? Do you track in database?

Here, what I did, SINCE I HAVE ALL QUERIES DEFINED AS DATA, I added few lines of code to:
1. generate prometheus global variables
2. generate function that dumps all these metrics to prometheus string
3. few lines add increment to global variables in their special global variables

Wow, so simple anyone can do it. And you can do anything you'd want when you generate code from data. Sooooooooo simple!


Database tables

I also define them as data. Not as raw SQL.

let db_tables = [
  ...

  mk_db_table
    11
    "chat_message_likes"
    [
      mk_field 1 "message_id" Int ~index:true;
      mk_field 2 "user_id" Int ~index:true;
      mk_field 3 "time_liked" Timestamp ~default:"CURRENT_TIMESTAMP" ~index:true;
    ]
    ~uniq_constraints:["message_id"; "user_id"]
    ~test_data:[
      ["112"; "10"; "NOW()"];
      ["113"; "12"; "NOW()"];
    ]
  ;

  ...
]

And the RAW SQL is just generated.

mk_field 1 - field number, is unique identifier for field that does not figure in SQL, so, I could rename fields in SQL if I want to.

So, I can rename a column, keep same number, and migration is generated that renames column.

Guess what? If I change schema in the data my queries break, I have to fix that. Everything has to work together.

Which is nicer:

- Having raw SQL to define schemas, maybe parse SQL Ast and try figure out what it does, and write migrations by hand also
- Define schema as data and just generate all migrations with simple code generation?

I choose second, and it paid dividends time and time and time and time again.

I have single test that checks that migrated schema from previous version must be same as freshly initialized, so, I can tolerate errors in migration function, because they are detected early and never reach production.

What I could also do, since I have schema as data:
- Generate code for automatic monitoring of int overflows
- Say that table has logical unique constraint, but don't enforce it because of performance and generate code to monitor it
- Maybe flag some tables that need to be mirrored to Clickhouse for better performance and generate Clickhouse code that will keep refreshing table in Clickhouse

Possibilities of this approach to solve problems are endless.

REST endpoints

Rest endpoints? I have them as data. Here's an example:

let rest_endpoints = [
  ...

  mk_rest_endpoint
    FtForum
    "/api/post_to_chat_room"
    LoggedInUser
    (CustomAction "post_to_chat");

  ...
]

As you can see, I even have enum, whether endpoint is accessible to logged in user or to outsider.

What is custom action you ask? I have all custom actions defined as data:

let custom_actions = [
  ...

  mk_custom_action
    FtForum
    ~return_type:"post_to_chat_return_result"
    "post_to_chat"
    [
      "chat_room_id", Int;
      "contents", String;
    ];

  ...
]

They have type signatures and arguments. What type signature the rest endpoint has depends on the custom action. For instance, now rest endpoint receives json with body of `chat_room_id` and `contents` - it comes from custom action. It could database query mentioned earlier instead, and rest endpoint would accept different argument.

How do I call this rest endpoint? I'm glad you asked, you just use typesafe OCaml function that under the hood gets results of this custom action and you only provide typesafe function. In fact, we receive defined return type:

type post_to_chat_return_result =
  | CAR_PostMsg_MessageTooLong
  | CAR_PostMsg_EmptyMessage
  | CAR_PostMsg_UserDoesntBelongToRoom
  | CAR_PostMsg_Ok
[@@deriving yojson]

No json parsing by hand, I just call generated function in the frontend:

rest_post_post_to_chat { chat_room_id = chat_room.room_id; contents = contents } (fun res ->
    match res with
    | CAR_PostMsg_Ok -> (
        materialize_toast "Hoooray, posted!"
    )
    | CAR_PostMsg_MessageTooLong -> (
        materialize_toast "Message to long"
    )
    | CAR_PostMsg_EmptyMessage -> (
        materialize_toast "Message empty"
    )
    | CAR_PostMsg_UserDoesntBelongToRoom -> (
        materialize_toast "You do not belong to this chat room"
    )
)

If I add more enum outcomes in the backend I MUST also fix the frontend to account for such case.

I don't deal with raw json. I don't deal with null pointer exceptions. Everything is typesafe and usually works first time and there are no surprises. I don't understand why people would ever want to code frontend in plain javascript, suicide!

Let's look at all code that was generated to serve/call this rest endpoint:


  post "/api/post_to_chat_room" begin fun req ->
    wauth_succ "backend_rest_ca_post_to_chat" req (fun session ->
      App.string_of_body_exn req |> Lwt.map (fun json ->
        match Yojson.Safe.from_string json |> gen_type_7557852441389680360_of_yojson with
        | Ok args -> (
            let with_connection = with_connection ~ray_id:(get_ray_id_from_request req) in
            let api = mk_custom_action_auth_api "post_to_chat" with_connection session req.request in
            let result = ca_post_to_chat api (args.chat_room_id) (args.contents) in
            result |> post_to_chat_return_result_to_yojson |> Yojson.Safe.to_string |> respond_json_str_lwt
          )
        | Error err -> respond_json_str_lwt "PHEIL"
      )
    )
  end;


All of the tricky stuff, about authenticating, parsing right type, right arguments, and returning type as json is generated.
The function that is not generated is ca_post_to_chat and is a perfectly typesafe function that just receives appropriate labeled arguments:


let ca_post_to_chat ~(api: custom_action_auth_api) ~(chat_room_id: int) ~(contents: string) =
  if String.length contents > 1000 then (
    CAR_PostMsg_MessageTooLong
  ) else if String.is_empty contents then (
    CAR_PostMsg_EmptyMessage
  ) else (
    with_connection "post_message_to_chat" (fun c ->
        let res = dbq_does_user_belong_to_chat_room c
            ~user_id:api.session_data.user_id ~room_id:chat_room_id in
        if res.(0).user_belongs > 0 then (
          dbm_post_to_chat_room c
            ~user_id:api.session_data.user_id
            ~chat_room_id ~contents |> ensure_pg_ok;
          CAR_PostMsg_Ok
        ) else (
          CAR_PostMsg_UserDoesntBelongToRoom
        )
      )
  )


As you can see, I don't deal with anything that is HTTP related, I declare rest endpoint, state custom action with its arguments, its return type (that will be perfectly reflected in frontend, bye bye faggitty HTTP return error codes), and I have to provide typesafe function to fulfill that contract and if anything is wrong, compiler will error out and I cannot ship to production.


This is generated frontend code, so I could call this rest endpoint just as any other function with result callback:

let rest_post_post_to_chat (args: gen_type_7557852441389680360) with_result =
  let url_var = Printf.sprintf "/api/post_to_chat_room" in
  let body = gen_type_7557852441389680360_to_yojson args |> Yojson.Safe.to_string in
  post_ajax_url_json_wcallback url_var "frontend_ca_post_to_chat" body (fun str_res ->
      Yojson.Safe.from_string str_res |> post_to_chat_return_result_of_yojson |>
      function
      | Ok result -> (
          with_result result
        )
      | Error err -> (
          print_endline "Cannot deserialize custom action"
        )
    )


I don't deal with knowing what is the right URL to call this - all generated from same source, and both perfectly match!

Plus, I make these checks of the top of my head:
1. Custom action must exist to be attached to REST endpoint
2. You cannot call REST endpoint with non existing arguments, because same generated type (without nulls) is both in frontend and backend, and same serialization function parses them
3. There cannot be duplicate REST endpoints, even if I have thousands of them (and they grow very fast, because adding many of them is now trivial due to code generation) duplicate paths will be detected
4. It is generated that every REST endpoint with its label is logged and monitored in prometheus automatically.
5. If I break query in the backend, and type signatures change I must refactor the frontend too


Thoughts

What framework offers you that? Because everything I see is a joke compared to what I do myself. And that is for the very simple fact:

DEFINE YOUR PROBLEMS AS DATA

If I had to write separate calls to rest endpoints and separate calls in backends everything would become a mess very quickly. If I had no typesafety, things would become a mess very quickly.

TYPESAFETY IS A MUST IF YOU WANT TO DO SUCH THIGS.

I rip for everyone I meet about typesafety. Most people can't understand it because they never tried to do something as powerful as this pattern, so, they think its okay to push non working code, someone notices and push some other garbage code to fix first garbage code. But if you want to do things like this pattern, try doing it with NodeJS or Ruby and we'll see how much you'll last. You'll start drowning in infinite amount of runtime errors very quickly. Hence, we have to use statically typed language, that will check our generated code integrity or these exploits become practically impossible.

Now I hope everyone understands why I don't take any dynamically typed scripting languages seriously, be it python or javascript or ruby. I don't need them. I do much more powerful things, with much greater correctness and typesafety. I barely spend time debugging code, because it just usually works. Ruby developer has to write 2x code, implementation and tests because he has no choice. I write 90%+ percent implementation code on rock solid abstractions and very little tests, for pesky things like making sure regex is parsing what I expect. I couldn't imagine writing trivial tests for every REST endpoint, or database query - what a waste of time. Things like that work from the first time in vast majority of the cases and I can focus on the application.

Sure, I wish some features were present in OCaml that ruby has, like decent string interpolation, but again, I did not find it issue enough yet, but, if I will, I will just make typesafe template abstraction in meta executable.

Who forbids me of defining templates like that? This code does not exist, but I'm just fantasizing:


let templates = [
  ...
  mk_template "some_template_ini" {|
    ip_address = <the_ip:string>

    open_port = <port:int>

    <iterate peers { peer_port: int, peer_ip: string }
    connection = <peer_ip>:<peer_port>
    <iterate end>
  |};
  ...
];


And then just generating typesafe function with all these arguments:

let template_string = tplt_some_template_ini ~the_ip:"127.0.0.1" ~port:8080 ~peers:[{ peer_port = 1234; peer_ip = "1.1.1.1" }] in
...


What forbids me from doing that? Hence, I don't care about programming language features.

I only need:
1. Decent typesafety (like OCaml and Rust provides, Java with random null pointers is way too weak)
2. Meta executable with my domain problem defined as data

Other benefits of generating code yourself in meta executable:
1. Your IDE will be grateful, it can parse trivial generated language code, instead of having to support very complex language features
2. You have a reference point of how generated code looks, so, you don't need to wonder, like in Lisp what macros have done and why stuff doesn't work
3. Ultimate flexibility to have cross context consistency, imagine, generating language bindings to your rest endpoints for huge amount of different languages
4. Unlike C/Rust/Lisp macros, which are very weak and just do local context this allows you to evaluate global context of everything working together from all different sides (you can check that for instance, a queue exists but is not used by anyone)
5. Since your problem is defined as immutable data, you can test your data very fast, mostly in memory consistency tests for your domain (avoiding pesky integration tests that require spinning up machines)

This is so simple anyone could do this. You don't need PhD's (people in university are overrated anyway), only common sense. If I taught a greenhorn coder this, he would probably beat the living crap out of 99% seasoned developers in productivity and correctness.

That's it. Not interested in Rust macros, Lisp macros, Rebol, Red, Scala, Go, V or any of 100 languages that will come out next year and forever. I can do everything on my own and generate anything I ever wanted. Practical applications of this pattern are endless and productivity gains are immeasurable. No wonder I consider everyone else around me an idiot now :/

I will also explain the logical reasoning and philosophy behind this. We will touch sexes, why even women are attracted to a certain behavior in a man, feminine and masculine planes and all creation.