The first set of error messages suggest that you have two installation of the Qt libraries on your computer, one in the directory /usr/local/opt/qt/lib/Qt*, the other in /Users/3sd/Qt/5.11.1/clang_64/lib/Qt*. Both libraries version are being pulled in during compilation of OVITO leading to a version conflict.
It's important to make sure that one and the same Qt version is being used during compilation of OVITO and during execution. You can control which Qt installation is being used during compilation by setting the CMAKE_PREFIX_PATH variable in CMake, e.g.
cmake -DCMAKE_PREFIX_PATH=/Users/3sd/Qt/5.11.1/clang_64/ ...
The Python interface of Ovito uses the PyQt5 module, which provides the Python bindings for the Qt library functions. At runtime, the statement "import PyQt5.QtGui", which appears in the error message, will load the PyQt5 module and with it the native Qt libraries. My guess is that at this point the libraries from the second Qt installation directory get loaded, leading to a version conflict with the Qt libraries that are needed by the Ovito core, i.e. the ones from the first Qt installation that was specified at build time.
Probably the resolution to this problem is to build the PyQt5 module form source instead of using a precompiled or prepackaged version. When building the PyQt5 python module, you need to specify the path to the same Qt library installation that is also used to build Ovito. At least this is what I do to build the official binaries for Ovito for Mac. I download the PyQt5 sources, build them against the same Qt version as Ovito and install them in the Python interpreter (see
here).
May I ask why intend to build Ovito/ovitos from source on macOS in the first place instead of using the prepackages binaries?