Which programming language is best suited for economic planning?
Python is the most general purpose but almost certainly there is a compiled language which is orders of magnitude faster. I know that R has many functions written in fortran and C which are blazingly fast but having never worked with any compiled languages myself I have no idea how fast they are compared to R.
what are other people's experiences?
Hmm I couldn't talk about which language would be best in general, but I have recently discovered that the data.table library in R will speed up functions that manipulate large datasets to an incredible degree.
@casperadmin that library was my main reason for making this post actually! I've been using it for years for work and used it in college. To my knowledge, it actually runs everything in C.
It's a pretty ingenious idea; have a language which is easy to program and it's basically a safe interface with a much quicker language like C. My statistics professors in college who taught courses in R would often say to never write a loop in R since someone has written a function that will run it in fortran or c that will do it faster and you'll save time by just finding what it is haha. this is what the bracket and $ notation on data.table objects do.
and the way fread and fwrite work is also incredible.
R seems to be particularly well suited actually. the function for inverting matrixes ultimately is written in fortran. the pacma library's inv() function returns the inverse of a matrix but it's source code is just a nice wrapper for the solve() function in base R. the solve function "is an interface to the LAPACK routines DGESV and ZGESV." which are both written in fortran.
source code available here: https://github.com/Reference-LAPACK/lapack/tree/master/SRC
if one were going to implement Leontief's method, this inverse matrix part seems like it would be the most computationally demanding and it's already written for us in an incredibly fast language. the only better thing we could hope for would be a C or C++ or Lisp implementation but I don't know if those would necessary be faster. I know nothing about those languages tho.
There are two separate questions: "what is the best programming language for people trying to do research in this area?" and "what will the planning software of the future be written in?" As for the first question it's really a matter of what people are familiar with / has the best data science tools out of the box / personal preference. As for the latter question, when it comes to large compute problems the thing that matters the most is writing correct concurrent code. When people program targeting distributed computers (a.k.a. "supercomputers") this is the main concern. Rust I think stands out in the current field for writing correct concurrent code. Haskell is a personal favorite for these kinds of problems as well. Since clock speeds have stopped speeding up and processors are now having to rely on just increasing core count this is becoming a more and more popular area of PL research, so I think the future will yield programming languages much better suited to this problem with nice primitives for concurrent programming.
"Best suited" very much depends. For the lowest-level number crunching stuff it's typically FORTRAN, C and assembly. Stepping up one level you need control over memory allocation and the ability to ask the kernel various things so as to avoid unnecessary copying and reallocation. Most often in HPC this is done in C or C++. That takes care of most of the heavy lifting, so whatever you put above that likely depends on whatever gets the job done with the team you have.
R seems useful for research, but I am not familiar with it. I tend to use GNU Octave.
@jules Does Rust have tools for formally verifying code? C does, and last time I looked Rust didn't. But maybe that has changed? Python isn't really suitable when it comes to writing correct code due to its anemic type system. Numpy/scipy were a pain last time I used them.
@jules those two questions are an important distinction. great point.
i think this really goes well with what i said previously about R. it's easy for researchers and lay people to write and check planning programs written in it while the actual machine-level stuff happens in another language altogether.
i'll see if there are libraries written in Rust for R which might be doing similar operations that we would want.
As a sidenote, i have a primary source that many of Google's supercomputers are actually running python. I know for a fact (primary source, alum of my alma mater) the patent department at google is running basically only python. they own so many patents that they don't even know what they own and their internal patent team write programs to analyze and categorize their patents entirely in python. I asked the guy why python if it's so slow and he said that with the data storage and computing power they have the team really don't have to worry about either thing at all. crazy to think, isn't it?
@Joe This actually doesn't surprise me at all. Often times there is some work done to take advantage of parallelism safely or do fast numeric computing, and then there is some library that allows you to take advantage of this work in a language like python. This is how the whole machine learning world operates. Machine learning is a field that especially relies on compute power and parallelism. They get away with using python by writing libraries in some other language to do the nasty bits and then interfacing with it using pytorch or something.
@thardin Rust has no where near the ecosystem of verification tools that C has, although it can do a lot of the stuff required to verify the correctness of parallel programs using affine types which I think is of particular importance (more so than full program verification, termination checking, etc)
@jules Sounds like affine types are mostly useful for avoiding copies, and presumably for correctly moving data between threads. Do they work with things like MPI in the mix? I spent quite a bit of time in the FmiGo project locating and eliminating allocations and copies in the MPI part of the system. Eventually we got latency down to a few microseconds.
I think @wpc has done some work in the area of PL design for distributed computing. Most of the time the "compilation" of distributed algos to say C or Rust is done by hand. It would be useful if say chained matrix multiplications and system solves were automagically MPI-ified with optimal chaining handled by the compiler. At the moment this seems to be more of an art.
@Joe This actually doesn't surprise me at all. Often times there is some work done to take advantage of parallelism safely or do fast numeric computing, and then there is some library that allows you to take advantage of this work in a language like python. This is how the whole machine learning world operates. Machine learning is a field that especially relies on compute power and parallelism. They get away with using python by writing libraries in some other language to do the nasty bits and then interfacing with it using pytorch or something.
woah! that makes total sense actually. so it it's exactly like what i imagined would be beneficial about R for planning. split up the task from:
theory -> program implementation -> hardware instructions
to
theory -> interpreted lang implementation -> compiled lang operations -> hardware instructions
if i'm understanding correctly? that way, for example, the google employees on that team who are way more familiar with law can focus on applying the theory and methods into and easy to use language like python and another team (probably some publicly maintained library if we're being honest) figures out how to make the functions in more efficient languages work with this python "interface".
looking it up, it seems like there are quite a few python libraries which use underlying C which makes total sense.
@jules Looks interesting. Makes me wonder how this compares to more out-there languages like E.