DDFacet (new code)

I'm installing a new piece of the LOFAR code called DDFacet, the repo is still private, the code is in my home, I can try to install it manually for the first time, then we can see if/how to add it to the script. The first missing package is: python3-pybind11

would it be possible to install it system-wide?

I was thinking for future reference it might be good to keep a file in this repo with all the external packages that you had to install to fulfill all the requirements (such as astropy, pytables...)

I am installing it by hands as a test, other requirements I noted:

• psutil
• pyfftw
• SharedArray
• ephem
• prettytable
• cpuinfo
• Polygon
• pylru
• deap

How did the testing go? I guess we'd need to start a fresher environment, based on pkgsrc-2020Q2 at least. What's your preferred Python version right now? 3.8 works?

I see pybind11 in current pkgsrc: http://cdn.netbsd.org/pub/pkgsrc/current/pkgsrc/devel/py-pybind11/README.html

(What the heck … you need Boost-Python and extra bindings for C++11 and Python? This is not programming, it's a game of bondage! Standing on the shoulders of giants … here, it's riding on an army of dwarfs:-/)

Those other requirements are all Python packages? I don't see how such a growing list can be kept current in a general-purpose collection like pkgsrc. We should focus on the tricky stuff (like Boost, correctly intertwined with the compiler and other libs) as base and then the lot of hopefully simpler packages on top, installed by the script if not present.

There should be a common scheme to get all the python stuff in, right? So we could write down a list of deps and use pip or the like to install them into our prefix. My emphasis would be on ensuring the exact versions, though, as the environment should be reproducable.

Things seems to work fine, all these dependencies are for a big python code (DDFacet) that we recently started using - they are all python packages and theoretically we can easily install them with pip (I did not have problems installing them)

It may make sense to draw a line for the easily-installable stuff that builds on things like numpy that are more involved.

As I did not work much with virtualenv and friends myself, is it easy to install the Python packages (with defined versions, of course) via pip in the build script and install them into the prefix? Other packages we build install python modules, too.

At some point we can define the list of packages we expect in the environment and those that are going to be installed anyway by the build script. Some testing of the expected packages being present makes sense.

I guess pip-installing the packages looks identical for the lot of them, so we can just add a loop to the script installing a list of packages with proper version into their prefixes inside the LOFAR software tree.

You got an example for e.g. installing psutil vX.Y.Z into $LOFAR_PREFIX/psutil?