new dppp incompatible with current patch

This is going to be a recurring problem... how do we flow the patch directly into dppp source?

Is there any hurdle in suggesting the patch to upstream (as a pull request, perhaps)? From my PoV, it could just be merged.

A first step perhaps is just to open an issue with the current patch as attachement and asking for any opinion on why someone would not like those changes. Perhaps upstream wants to make the changes their own way.

Important points are:

1. Support Boost::pythonXY component as opposed to Boost::python or Boost python3. This is the present/future.
2. Use current (less deprecated) cmake ways to detect python.
3. Less redundancy in Python checking in CMakeLists at deeper level.
4. Add support for BLAS_LIBS and LAPACK_LIBS, as this is customary in HPC installations to choose the (optimized) BLAS implementation in an environment that offers multiple options.

If they agree that these are worthy goals, they could just merge the patch file, or ask (you?) to fork their repo and a prepare a pull request, as it is customary in the git(hub, lab) world nowadays.

This is my (ugly) fix: DP3-buildfix.patch

but I am not sure it still does exactly what you wanted, it seems to work though...

Shall I open the issue or would you prefer to do it? (I will likely not be able to answer any question anyway...)

Well, if it works … you can just commit it. Makes it easier for me to judge your changes, as I see the diff of the diff;-)

What's the upstream issue tracker for DPPP?

(My fixes are also ugly … I'm just cargo-coding this CMake stuff like anyone else;-)

For DPPP we use: https://github.com/lofar-astron/DP3/issues

Hi Thomas, the new version of DP3 brakes our installation script again, I will put together an updated patch. Did you manage to start propagating your fixes to the DP3 repo?

cheers, fra

Am Thu, 28 May 2020 14:20:18 +0200 schrieb "Gasperin, Prof. Dr. Francesco" gitlab@gitlab.rrz.uni-hamburg.de:

Hi Thomas, the new version of DP3 brakes our installation script again, I will put together an updated patch. Did you manage to start propagating your fixes to the DP3 repo?

I wanted to today … you got your update ready yet?

yes but it's just for a branch that still needs to be merged, it will take hours/days for them to check and merge it

fra

So our current patch is for DP3 master branch and I should just post it to them?

yes, the current one is for the master

https://github.com/lofar-astron/DP3/pull/238

I'm off for vacation for a week … I'll see if I can offer some comments, but maybe you can watch this, too.

I've seen the tickets, thanks a lot, I'll follow it up! :)

cheers and have a nice vacation Fra

I updated the patch, matching the current pull request. Confirm?

Hi @BAQ0506, while trying to build DP3, I noticed that there still seems to be some issue :/

I get the following build logs:

build.log

Installing DP3...
Cloning into 'presrc'...
Patching build (for BLAS, Boost, Python, usually) ...
patching file CMake/FindCasacore.cmake
Reversed (or previously applied) patch detected!  Skipping patch.
2 out of 2 hunks ignored -- saving rejects to file CMake/FindCasacore.cmake.rej
patching file CMake/PythonInstall.cmake
Reversed (or previously applied) patch detected!  Skipping patch.
4 out of 4 hunks ignored -- saving rejects to file CMake/PythonInstall.cmake.rej
patching file CMakeLists.txt
Reversed (or previously applied) patch detected!  Skipping patch.
2 out of 2 hunks ignored -- saving rejects to file CMakeLists.txt.rej
patching file PythonDPPP/CMakeLists.txt
Reversed (or previously applied) patch detected!  Skipping patch.
2 out of 2 hunks ignored -- saving rejects to file PythonDPPP/CMakeLists.txt.rej

And the CMakeList.txt.rej is:

CMakeList.txt.rej

--- CMakeLists.txt
+++ CMakeLists.txt
@@ -44,13 +44,14 @@ endif(IDGAPI_LIBRARIES AND IDGAPI_INCLUDE_DIRS)
 
 find_package(CFITSIO REQUIRED)
 
-find_package(PythonLibs 3 REQUIRED)
-find_package(PythonInterp REQUIRED)
-message(STATUS "Using python version ${PYTHON_VERSION_STRING}")
+find_package(Python3 COMPONENTS Interpreter Development)
+message(STATUS "Using python version ${Python3_VERSION}")
+
+include_directories(${Python3_INCLUDE_DIRS})
 
 #Prevent accidentally finding old BoostConfig.cmake file from casapy
 set(Boost_NO_BOOST_CMAKE ON)
-set(BOOSTPY ${PYTHON_VERSION_MAJOR}${PYTHON_VERSION_MINOR})
+set(BOOSTPY ${Python3_VERSION_MAJOR}${Python3_VERSION_MINOR})
 find_package(Boost COMPONENTS date_time filesystem python${BOOSTPY} numpy${BOOSTPY} program_options system unit_test_framework REQUIRED)
 include_directories(${Boost_INCLUDE_DIR})
 
@@ -236,7 +237,7 @@ set(EXTRA_LIBRARIES ${EXTRA_LIBRARIES}
   ${CASACORE_LIBRARIES}
   ${Boost_LIBRARIES}
   ${EXTRA_LIBRARIES}
-  ${PYTHON_LIBRARIES}
+  ${Python3_LIBRARIES}
   ${IDGAPI_LIBRARIES}
   Threads::Threads)

Do you have any idea what the issue might be this time? Looks like again something with the python stuff?

-Henrik

Well, this just looks like the DP3 patch has been fully merged upstream, see https://github.com/lofar-astron/DP3/pull/238 . So we can drop the patch. I don't see any hunks left, presuming your log is complete. You might try preparing a commit for that yourself.

Hm, I guess I would achieve that by just removing the lines containing patch -Np1 < "$scriptdir/DP3-buildfix.patch".

But it seems that now there is a new dependency for the development branch, called EveryBeam (see here)

CMake Error at CMakeLists.txt:79 (message):
  EveryBeam include file (EveryBeam/load.h) not found: please install
  https://git.astron.nl/RD/EveryBeam

I guess this is to replace LOFARBeam in the future. I will wait until they merge this into master...

Henrik, you should be able to push to the repo now. You could remove the patch (or just make it empty). The EveryBeam thing … it's only in the dev branch, right? Maybe you can work on supporting that one, too.

It is only on the dev branch, yes. But afaikt this is the only branch where the your pull request was merged. So for the current master we would still need the patch.