Installation as a pip package broken

Hello,

It is no more possible to install TheengsGateway as a pip package (in venv or not doesn’t change anything)
I tried and I have reproduced the same behavior on a debian 11 and debian 12 vm, both fully upgraded

Actually, it is TheengsDecoder that is causing issue:

python3 -m pip install TheengsDecoder

=>

Collecting TheengsDecoder
  Using cached theengsdecoder-1.8.5.tar.gz (304 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: TheengsDecoder
  Building wheel for TheengsDecoder (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for TheengsDecoder (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [84 lines of output]
      WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
      Traceback (most recent call last):
        File "/tmp/pip-build-env-_6g27v2d/normal/lib/python3.9/site-packages/setuptools_scm/_integration/pyproject_reading.py", line 36, in read_pyproject
          section = defn.get("tool", {})[tool_name]
      KeyError: 'setuptools_scm'
      
      
      --------------------------------------------------------------------------------
      -- Trying 'Ninja' generator
      --------------------------------
      ---------------------------
      ----------------------
      -----------------
      ------------
      -------
      --
      CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
        Compatibility with CMake < 3.10 will be removed from a future version of
        CMake.
      
        Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
        to tell CMake that the project requires at least <min> but has been updated
        to work with policies introduced by <max> or earlier.
      
      Not searching for unused variables given on the command line.
      
      -- The C compiler identification is GNU 10.2.1
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- The CXX compiler identification is GNU 10.2.1
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done (0.3s)
      -- Generating done (0.0s)
      -- Build files have been written to: /tmp/pip-install-yxl8h5iv/theengsdecoder_1ba096835b2a492fac582573e0e6cddc/_cmake_test_compile/build
      --
      -------
      ------------
      -----------------
      ----------------------
      ---------------------------
      --------------------------------
      -- Trying 'Ninja' generator - success
      --------------------------------------------------------------------------------
      
      Configuring Project
        Working directory:
          /tmp/pip-install-yxl8h5iv/theengsdecoder_1ba096835b2a492fac582573e0e6cddc/_skbuild/linux-x86_64-3.9/cmake-build
        Command:
          /tmp/pip-build-env-_6g27v2d/overlay/lib/python3.9/site-packages/cmake/data/bin/cmake /tmp/pip-install-yxl8h5iv/theengsdecoder_1ba096835b2a492fac582573e0e6cddc -G Ninja -DCMAKE_MAKE_PROGRAM:FILEPATH=ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-yxl8h5iv/theengsdecoder_1ba096835b2a492fac582573e0e6cddc/_skbuild/linux-x86_64-3.9/cmake-install -DPYTHON_VERSION_STRING:STRING=3.9.2 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-_6g27v2d/overlay/lib/python3.9/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/home/jeedom/gw_venv/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.9 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.9.so -DPython_EXECUTABLE:PATH=/home/jeedom/gw_venv/bin/python3 -DPython_ROOT_DIR:PATH=/home/jeedom/gw_venv -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.9 -DPython3_EXECUTABLE:PATH=/home/jeedom/gw_venv/bin/python3 -DPython3_ROOT_DIR:PATH=/home/jeedom/gw_venv -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.9 -DCMAKE_MAKE_PROGRAM:FILEPATH=ninja -DCMAKE_BUILD_TYPE:STRING=Release
      
      CMake Error at CMakeLists.txt:1 (cmake_minimum_required):
        Compatibility with CMake < 3.5 has been removed from CMake.
      
        Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
        to tell CMake that the project requires at least <min> but has been updated
        to work with policies introduced by <max> or earlier.
      
        Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.
      
      Not searching for unused variables given on the command line.
      
      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "/tmp/pip-build-env-_6g27v2d/overlay/lib/python3.9/site-packages/skbuild/setuptools_wrap.py", line 660, in setup
          env = cmkr.configure(
        File "/tmp/pip-build-env-_6g27v2d/overlay/lib/python3.9/site-packages/skbuild/cmaker.py", line 354, in configure
          raise SKBuildError(msg)
      
      An error occurred while configuring with CMake.
        Command:
          /tmp/pip-build-env-_6g27v2d/overlay/lib/python3.9/site-packages/cmake/data/bin/cmake /tmp/pip-install-yxl8h5iv/theengsdecoder_1ba096835b2a492fac582573e0e6cddc -G Ninja -DCMAKE_MAKE_PROGRAM:FILEPATH=ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-install-yxl8h5iv/theengsdecoder_1ba096835b2a492fac582573e0e6cddc/_skbuild/linux-x86_64-3.9/cmake-install -DPYTHON_VERSION_STRING:STRING=3.9.2 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-_6g27v2d/overlay/lib/python3.9/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/home/jeedom/gw_venv/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.9 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.9.so -DPython_EXECUTABLE:PATH=/home/jeedom/gw_venv/bin/python3 -DPython_ROOT_DIR:PATH=/home/jeedom/gw_venv -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.9 -DPython3_EXECUTABLE:PATH=/home/jeedom/gw_venv/bin/python3 -DPython3_ROOT_DIR:PATH=/home/jeedom/gw_venv -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.9 -DCMAKE_MAKE_PROGRAM:FILEPATH=ninja -DCMAKE_BUILD_TYPE:STRING=Release
        Source directory:
          /tmp/pip-install-yxl8h5iv/theengsdecoder_1ba096835b2a492fac582573e0e6cddc
        Working directory:
          /tmp/pip-install-yxl8h5iv/theengsdecoder_1ba096835b2a492fac582573e0e6cddc/_skbuild/linux-x86_64-3.9/cmake-build
      Please see CMake's output for more information.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for TheengsDecoder
Failed to build TheengsDecoder
ERROR: Failed to build installable wheels for some pyproject.toml based projects (TheengsDecoder)

my investigations so far make me think that the version of cmake defined (3.3) is the cause as compatibility with cmake<3.5 has been dropped but I don’t know exactly since which version)

I could reproduce the same error if I try to install from source so I tried to update CMakeLists.txt files but I’m confused on which one should be use exactly between /CMakeLists.txt (on the root) and /python/CMakeLists.txt

If I adapt MIN cmake version in the second, I don’t have the error above anymore but others pop regarding src files not found / invalid path; seems like target_include_directories() is not taken into account.

I don’t know anything about cmake so I’m bit lost;

Is this an issue you spotted already? Any advice to move fwd?

Hi @Mips

Admittedly not a CMake expert here either, but could you check which CMake version is installed with your Debian 11 and Debian 12.

Then also try to change the syntax with min and max in the CMakeLists.txt files to

cmake_minimum_required(VERSION 3.3...3.xx)

with 3.xx (4.0.0. ??) being your latest version on either Debian 11 or 12.

Let us know how this iw working for you.

Hi DigiH,

thanks for your reply.

The default cmake version available via apt package is:

  • on deb11: 3.18.4-2+deb11u1
  • on deb12: 3.25.1-1

But I don’t think this is relevant because via pip, various other version are available (same on deb11 & deb12);
I tried to install various one in a venv before installing TheendDecoder but same result as above because during the build a “build-venv” is created in /tmp and a dedicated version is installed and used, see extract of above trace:

/tmp/pip-build-env-_6g27v2d/overlay/lib/python3.9/site-packages/cmake/data/bin/cmake

So I suspect that the last version of cmake (4.0.0) is installed hence causing the issue.
I cannot check the version of cmake there because the tmp dir is deleted at the end;


I already tried to update /python/CMakeLists.txt with the proposed syntax but then I had error on missing source (just giving the end because is quite long):

     CMake Error at CMakeLists.txt:6 (add_library):
        Cannot find source file:
      
          src/decoder.cpp
      
        Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm
        .ccm .cxxm .c++m .h .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90
        .f95 .f03 .hip .ispc
      
      
      CMake Error at CMakeLists.txt:6 (add_library):
        No SOURCES given to target: _decoder
      
      
      CMake Generate step failed.  Build files cannot be regenerated correctly.
      Traceback (most recent call last):
        File "/tmp/pip-build-env-4iidavig/overlay/lib/python3.11/site-packages/skbuild/setuptools_wrap.py", line 660, in setup
          env = cmkr.configure(
                ^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-4iidavig/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 354, in configure
          raise SKBuildError(msg)
      
      An error occurred while configuring with CMake.
        Command:
          /tmp/pip-build-env-4iidavig/overlay/lib/python3.11/site-packages/cmake/data/bin/cmake /opt/decoder/python -G Ninja -DCMAKE_MAKE_PROGRAM:FILEPATH=ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/opt/decoder/python/_skbuild/linux-x86_64-3.11/cmake-install -DPYTHON_VERSION_STRING:STRING=3.11.2 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-4iidavig/overlay/lib/python3.11/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/opt/TheengsGateway/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.11.so -DPython_EXECUTABLE:PATH=/opt/TheengsGateway/bin/python3 -DPython_ROOT_DIR:PATH=/opt/TheengsGateway -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.11 -DPython3_EXECUTABLE:PATH=/opt/TheengsGateway/bin/python3 -DPython3_ROOT_DIR:PATH=/opt/TheengsGateway -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.11 -DCMAKE_MAKE_PROGRAM:FILEPATH=ninja -DCMAKE_BUILD_TYPE:STRING=Release
        Source directory:
          /opt/decoder/python
        Working directory:
          /opt/decoder/python/_skbuild/linux-x86_64-3.11/cmake-build
      Please see CMake's output for more information.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for theengsdecoder
Failed to build theengsdecoder
ERROR: Failed to build installable wheels for some pyproject.toml based projects (theengsdecoder)

so we can see the error:

        Cannot find source file:
      
          src/decoder.cpp

which somehow I found logical because src/decoder.cpp is not a sub-directory of /python/ where CMakeLists.txt is; so in it that scenario, the relative path would be ../src/decoder.cpp

but if tried to update source path, then I got error about c header files missing (which is also logical somehow because files aren’t in that same directory:

add_library(_decoder MODULE TheengsDecoder/_decoder.cpp ../src/decoder.cpp)

=>

      _modinit_prefix:PyInit_
      -- Configuring done (0.1s)
      -- Generating done (0.0s)
      -- Build files have been written to: /opt/decoder/python/_skbuild/linux-x86_64-3.11/cmake-build
      [1/4] Building CXX object CMakeFiles/_decoder.dir/opt/decoder/src/decoder.cpp.o
      FAILED: CMakeFiles/_decoder.dir/opt/decoder/src/decoder.cpp.o
      /usr/bin/c++ -D_decoder_EXPORTS -I/usr/include/python3.11 -I/opt/decoder/python/src/arduino_json/src -I/opt/decoder/python/src -O3 -DNDEBUG -fPIC -MD -MT CMakeFiles/_decoder.dir/opt/decoder/src/decoder.cpp.o -MF CMakeFiles/_decoder.dir/opt/decoder/src/decoder.cpp.o.d -o CMakeFiles/_decoder.dir/opt/decoder/src/decoder.cpp.o -c /opt/decoder/src/decoder.cpp
      In file included from /opt/decoder/src/decoder.cpp:22:
      /opt/decoder/src/decoder.h:26:10: fatal error: ArduinoJson.h: No such file or directory
         26 | #include "ArduinoJson.h"
            |          ^~~~~~~~~~~~~~~
      compilation terminated.
      [2/4] Building CXX object CMakeFiles/_decoder.dir/TheengsDecoder/_decoder.cpp.o
      FAILED: CMakeFiles/_decoder.dir/TheengsDecoder/_decoder.cpp.o
      /usr/bin/c++ -D_decoder_EXPORTS -I/usr/include/python3.11 -I/opt/decoder/python/src/arduino_json/src -I/opt/decoder/python/src -O3 -DNDEBUG -fPIC -MD -MT CMakeFiles/_decoder.dir/TheengsDecoder/_decoder.cpp.o -MF CMakeFiles/_decoder.dir/TheengsDecoder/_decoder.cpp.o.d -o CMakeFiles/_decoder.dir/TheengsDecoder/_decoder.cpp.o -c /opt/decoder/python/TheengsDecoder/_decoder.cpp
      /opt/decoder/python/TheengsDecoder/_decoder.cpp:4:10: fatal error: decoder.h: No such file or directory
          4 | #include "decoder.h"
            |          ^~~~~~~~~~~
      compilation terminated.
      ninja: build stopped: subcommand failed.
      Traceback (most recent call last):
        File "/tmp/pip-build-env-v4dawjpo/overlay/lib/python3.11/site-packages/skbuild/setuptools_wrap.py", line 668, in setup
          cmkr.make(make_args, install_target=cmake_install_target, env=env)
        File "/tmp/pip-build-env-v4dawjpo/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 696, in make
          self.make_impl(clargs=clargs, config=config, source_dir=source_dir, install_target=install_target, env=env)
        File "/tmp/pip-build-env-v4dawjpo/overlay/lib/python3.11/site-packages/skbuild/cmaker.py", line 741, in make_impl
          raise SKBuildError(msg)
      
      An error occurred while building with CMake.
        Command:
          /tmp/pip-build-env-v4dawjpo/overlay/lib/python3.11/site-packages/cmake/data/bin/cmake --build . --target install --config Release --
        Install target:
          install
        Source directory:
          /opt/decoder/python
        Working directory:
          /opt/decoder/python/_skbuild/linux-x86_64-3.11/cmake-build
      Please check the install target is valid and see CMake's output for more information.
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for theengsdecoder
Failed to build theengsdecoder
ERROR: Failed to build installable wheels for some pyproject.toml based projects (theengsdecoder)

And I stopped there because all these does not make sense to me;
I mean: it was working, so that code should be correct, and obviously I don’t understand everything so I’m probably on the wrong way.

Also, I supposed that the directive target_include_directories() in CMakeLists.txt should be there to instruct which directories should be used for include.
I guess the best to do would be to upgrade that CMakeLists.txt file to be compatible with a more recent version of cmake but as I don’t know how it is supposed to work; I mean: am I doing right trying to build from /python dir, it is difficult to make it work;
I don’t know if the errors I encountered are caused by a miss-configuration / usage of myself or by the fact that cmake version is not the same.

@Mips

Could you try with the latest recent release 1.9.1, which should install with

pip install theengsdecoder

Good job, it works perfectly!

I tested on debian 11 & 12; I can install a raspberry pi to test on it as well later this week if you need but I’m sure it will be fine as well.

Thanks a lot for your work and reactivity!