Software Maturity

Started by CultLeader, Today at 04:22:21 PM

Previous topic - Next topic

CultLeader

I made a few improvements to Eden platform as it will be almost two years that I'm running in production. No matter how many compile time errors you add not everything is testable in compile time. Say, for me it took half a year to get lets encrypt certificates working correctly. Not that I was debugging half a year, but one certificate issuance is valid for three months I think and not all steps in the pipeline succeeded in updating the certificate, hence it didn't refresh automatically. So after couple of failed auto refreshes and painful debugging sessions certificates have been working automatically for I think at least a year now and I forgot them.

And that's the path of most software. Regardless of the programming language. If you put n iterations to your library/package/executable it will have less and less issues.

Well, why use Rust then? For Rust path to maturity is much faster. Even though Eden platform executable and EdenDB compiler are written in Rust they do generate a lot of shell scripts. Needless to say there can be bugs which then I stamp out. But just like with automatic certificates example I get less and less of these and everything getting better and better.

I remember even with some crappy scripts for some company I maintained reasonably complex shell scripts before, which is a pain needless to say. However, after bloodied knuckles and hundreds of iterations of provisioning eventually things become quite stable.

Say, this forum, I didn't look for an implementation of forum in Rust. Simple machines has a lot of battle scars and frankly I don't need anything super complex to publish my thoughts. The blood and sweat and tears which made simple machines forum are other people, not mine.

That being said, I'd never start any new and ambitious project like Eden platform in any other language but Rust. Since this is a compiler that should blazingly test a lot of stuff I had to use Rust to never be limited by the language's performance ceiling. I knew I would get to my goals a lot faster by using typesafety of Rust. Even now when adding features, as I can't keep everything in my head, I forget the parts of code but once I browse around, all the types are clear and I can keep building stuff. If I add enum invariant I'll have to cover it. If I use non existing field that is a compile time error.

I can't imagine writing Eden platform in something like ruby. Just using the tooling of the language makes me feel disgusted. If you install a bunch of packages with opam in OCaml you just feel quality, a bunch of green lines finish building stuff and off you go. Same with cargo in Rust. When I had to use Ruby at work before I was appalled - some package needs some native dependency, or some config is incorrect and I get cryptic errors. Even worse it becomes when I actually need to use the tooling and I had to call a ruby priest who memorized all the errors by heart so he could translate it into actions of what was needed to be done to make it work. Ruby people are absolutely crazy.

Hence I'm also a little bit ashamed because there are still rough edges in Eden platform itself. However, these days mostly shipping product and from time to time I do get to iron out issues which make life better overall.

For instance, I incorrectly assumed that Grafana Loki and Grafana Tempo are good products. After operating them in production turns out they are horrible products, they use too much bandwidth, processes OOM crash often and the products themselves are slow. Turns out the only good product grafana labs ever made was... grafana. You see, I don't have every tiniest detail figured out. I hope to get there, but I don't know everything about underlying components. But I do know one thing for sure - I want to open grafana and see my goddamn logs and traces to work reliably. That is my vision. That is not some brilliant insight - I just want things to work. That's why I plan in future to replace loki and tempo with VictoriaLogs and VictoriaTraces, same guys that made VictoriaMetrics with which I never had any issues. I made a rule in my mind to just run away from any garbage that claims "bottomless storage" (code word for s3 lock in and inefficient querying).

Eden platform is a complete attempt at just providing everything lazy developer needs to run his infrastructure for any project where he wants to serve http requests to the user. Lazy developer (like I want to be) is assisted by compile time errors and if infrastructure compiles it will have to work. I don't want to hire infra guys to do the work of the compiler. Lazy developer just wants high availability, knowing that any server can go down and stuff will still work. My idea is that all of the services provided are just software and we shouldn't pay anything extra to predators like AWS or Vercel. We should just have a few bare metals at start and have everything we need already, deployments, backend services, monitoring, tracing, logs, typesafety, queues, databases and so on.

I imagine only adding very few more features now upon the top of everything that is working already:
  • Self hosted email
We have S3 and we have Postgres. We own our DNS. All these services are highly available. Why should we pay extra for email providers? We shouldn't. We could just host something like stalwart email (written in Rust, yay) and be completely free from any more extra email providers might charge us.
  • Automatic OpenAPI imports, we should just import the goddamn yaml, it is parsed, analyzed and we could expose the api to our Rust services and just be able to use them in a typesafe manner. That's it. We shouldn't need to open manual reqwest calls.
  • Generate and expose data about database schemas to blackbox deployments for typesafe database querying, like if you want to use third party frameworks that would be respectful of database types
  • Scaling strategy across datacenters if we need more than two bandwidth nodes - didn't need yet, will cross that bridge once I'm out of bandwidth

I'm not sure if I'll need to add more databases to the mix, there's Postgres already. Maybe for hyperscaling FoundationDB will make sense at some point, but I don't have any scaling issues on modern hardware as of yet.

Once all these features are added then I can just keep maturing them. So, if I define scope as finite and I can stop adding new stuff at some point that means eventually I should fix all the issues as well so Eden platform could be nearly flawless (as long as underlying components used allow it to be flawless).

Going on a tangent here, I'm glad that the shitstain of ruby on rails is now losing popularity and people are massively realizing that no typesafety is killing their productivity. DHH claims this is cock shanel of languages (how much of a homo should you be to know about perfume this much?) because muh we use sharp knives and we write little code. I understand where he's coming from, I'm using lucky framework which is in crystal (typesafe and blazing fast ruby) for one project and productivity is amazing. You know when I feel most productive in lucky framework? When I define endpoint and it generates automatic frontend typescript definition for calling it and I don't need to do anything else. When I, or mostly coding agent these days does initial implementation and has to fix all the typesafety errors and things just usually work. I don't spend times in long winded debugging sessions of what dynamic variable has become now and how did it get there. That is productive.

And its very funny, because when DHH mentions that static types are no good and that ruby scales because shopify uses it, he completely "forgets" to mention these points:
  • shopify is adopting sorbet for typesafe ruby
  • shopify has roughly 4000 engineers
  • shopify codebase is 2.8 million LoC

Now, let's divide lines of code per developer 2_800_000 / 4000 = 700.

700. 700 lines of code on average per developer. DHH would never mention this, so the long haired hippy fag claims that ruby codebases can scale, that is funny.

If we're measuring length of our pride now, EdenDB implementation without examples and tests is 12k lines of code, Eden Platform lines of code without tests is 79k. So total 91k lines of Rust code by me - one developer without using LLMs, bulk of implementation before even LLMs of today existed. I only now sometimes use LLMs to generate parts or proof of concepts for scripts or regexes, but everything else is by me alone. I guess then I'm 130x more productive engineer than an average monkey in shopify, which is not suprising for me at all.

Now I want to analyze also that in Eden Platform 24.4k lines of code are for pure code generation (/codegen/ module). So, little bit more than 30% of all code in Eden Platform is a code that writes code. Imagine if 30% of code in your project were Rust macros or ruby metaprogramming - project would be unmaintainable. But because my codegen is simple, I just create strings and I can generate any code, not just rust but Makefile scripts, nomad job declarations, sql migrations, anything really, then my approach becomes a lot simpler and the only reason why Eden Platform as a project is possible at all.

Now I want to analyze my actual production project which uses Eden Platform. All the lines of code in the repo are 154719 - let's rough it down to 154k lines of code. Here's the kicker, of these 154k lines of code I wrote with my hands... 11402 lines of code. I wrote 7% of code in my eden platform project, the rest of code like nomad job declarations, provisioning scripts, nix configurations is autogenerated. I do deploy stuff out of tree of eden platform though via blackbox deployments so that source doesn't end up in eden platform project code.

So say, even if I'm already roughly 130x engineer to the average communist fag at shopify Eden platform still makes me 13x more engineer (1690x total, lol) by the code that it generates for me and everything is type safe and I have no fear deploying to production.

So, of course, there are tons of success stories of using Rust for large projects with small teams, tons of blockchains (think Solana, Polkadot) but there are very few high LoC ruby projects and all of them have the same symptoms:
  • You need to hire tons of communist fags to maintain all that mess
  • They tend to be adopting some typesafety primitives anyway because it is obvious that code is unmaintainable once it becomes large in ruby

So unless you can hire or groom thousands of commie pink haired homos that like ruby good luck building your project with rails in 2026, lol.

Have a good day bois!