Is it time for Lisp in DevOps?

We have been working on a project migrating a big Rails app from physical hardware to AWS, and I have been doing a lot of automation work.

It strikes me how we are doing the same thing over and over with different tools: reading variables, templating files and running semi-declarative logic. All with more or less broken syntax, including:

  • Terraform
  • Terragrunt
  • Ansible
  • Shell scripts
  • Packer JSON
  • Jinja2
  • CodeDeploy appspec.yml lifecycle scripts
  • Capistrano
  • Rake
  • Cron specs
  • CloudFormation
  • Endless different config file syntaxes

During the dotcom days we would laugh at the guys with title "HTML Programmer". Now we are "YAML Programmers".

Can I somehow get this variable from over here, through there into this other config file without going crazy with escaping? Can I template a JSON file file with a scripting language written in YAML?

I like Ansible because it's powerful, but everyone can understand it even if they are not doing DevOps full time. It's procedural, which means fewer surprises. It is written in a proper general purpose programming language, Python. We can easily write extensions, so there is no limit to what it can do, unlike single-purpose systems. It's the best of the bunch, but the syntax is still filled with random bits of weirdness.

This makes me think about replacing it all with a Lisp (probably Scheme), following the "code is data" and "data is code" mantra. We would have lots of parentheses, but we would have real variables with sane scoping rules, real functions, proper syntax for function calls, and macros(!). Maybe we need the YAML/JSON equivalent of SXML

Lisp is very powerful, but has never had the success to match. I learned it after hearing smart people say, "learn Lisp, not because it's practical, but because it will expand your brain and change the way you think about programming". I have certainly found that to be true. I recommend "Practical Common Lisp if you are interested in learning Lisp. (His Coders at Work is also quite good.) And there are great Scheme books like The Little Schemer and of course Structure and Interpretation of Computer Programs.

I think that part of the lack of success of Lisp was due to bad timing. When it was at its peak, we didn't have open source, we had $5000/seat licenses for Lisp compilers (and that would buy a car). Lisp was very powerful, and faced competition from people selling FORTRAN for defense contracts who played the game better. Now we have open source tools and communities, e.g. Racket.

Another part was the "smug Lisp weenies". Lisp tends to attract smart but somewhat dysfunctional people, who are condescending to newbies and love to argue. They made it impossible to make progress on the legitimate improvements needed to the language.

Lisp has minimal syntax, which is one of its biggest strengths, but tends to make it difficult for newcomers. The killer was a combination of funny syntax, too much flexibility, a bad "out of box" experience, and poor community.

There are interesting things beyond Lisp these days. I mainly program in Elixir, which has the power of Lisp-style macros, approachable syntax and a great community. The pendulum is swinging away from dynamic scripting languages. The action is in functional programming and powerful type systems, e.g. Haskell. But there is something to be said for taking a step back from the cutting edge of type theory and making a practical, approachable language. So OCaml and Reason are interesting, particularly as an antidote to the intentionally dumbed-down Golang.

I could write a Scheme syntax for Terraform. Or maybe I will take a swing at a Terraform clone in Elixir...