Building SciPy on AIX

[sawyl]

A little while ago, I tried to get SciPy to build on AIX using IBM's XL series of compilers and the ESSL library. The results were disappointing. I couldn't get SciPy to link correctly and the only procedure I was able to find online was fiendishly complicated. After spend some days working on the problem, I gave up, put it aside and shifted my attention to more important things.

So I wasn't exactly delighed when SciPy's priority increased sufficiently to put it back on my radar. But, full of brio after the holidays and in search of an interesting puzzle to crack, I decided to give it another go. And what do you know? It only took me a few hours to come up with a solution, saved here for posterity and for any other AIX admin foolish enough to want to give this a try.

---

Compile BLAS

Download the netlib BLAS distribution and unpack it. Change the macros in make.inc to the following:

  FORTRAN   = xlf
  OPTS      = -O3 -qmaxmem=-1 -qextname
  NOOPT     = -qextname
  LOADER    = xlf
  LOADOPTS  = -qextname
  BLASLIB   = libblas_.a

Build the library in the usual way and copy to a central location, e.g. /opt/netlib/lib/libblas_.a.

---

Compile LAPACK

Download the netlib LAPACK distribution and unpack it. Copy INSTALL/make.inc.xlf (or make.inc.RS6K in older distributions) to make.inc and change the macros in make.inc to the following:

  FORTRAN  = xlf
  OPTS     = -O3 -qmaxmem=-1 -qextname
  NOOPT    = -qextname
  LOADER   = xlf
  LOADOPTS = -qextname
  TIMER    = EXT_ETIME

Build the library in the usual way and copy to a central location, e.g. /opt/netlib/lib/liblapack_.a.

---

Patch NumPy

Install NumPy if it isn't already installed. Patch the module numpy.distutils.fcompiler.ibm to fix preprocessor handling by adding a new _compile() method to the IBMFCompiler class as follows:

  def _compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts):
    cc_args = map(lambda s: s.replace("-D", "-WF,-D"), cc_args)
    pp_opts = map(lambda s: s.replace("-D", "-WF,-D"), pp_opts)
    FCompiler._compile(self, obj, src, ext, cc_args, extra_postargs, pp_opts)

This should ensure that pre-processor options such as "-DNO_ATLAS_INFO=1" are replaced with "-WF,-DNO_ATLAS_INFO=1"

---

Build SciPy

Unpack the SciPy distribution and run setup.py as follows:

  LDFLAGS="-L/opt/netlib/lib -lessl -lblas_ -llapack_" \
  LAPACK=/opt/netlib/lib/liblapack_.a \
  python setup.py build

If everything goes according to plan, the build process should complete without reporting any linking errors and it should possible to install using:

  LDFLAGS="-L/opt/netlib/lib -lessl -lblas_ -llapack_" \
  LAPACK=/opt/netlib/lib/liblapack_.a \
  python setup.py install

If you see errors of the form:

• ld: 0711-317 ERROR: Undefined symbol: srotmg_ then you've failed to include the -lblas_ flag or you failed to build your version of BLAS with the -qextname flag
• ld: 0711-317 ERROR: Undefined symbol: slamch_ then you've failed to add the -qextname flag to the NOOPT macro in LAPACK's make.inc

Onwards to the horrible — and more important — mess that is netCDF...

Comments

Cant complie LAPLACK with XLC or GCC

[tamer awad (from livejournal.com)]

Thanks you for your great blog here.
I cant make LAPLACK lib file. I dont have xlf I have xlc and gcc installed on AIX 6.1.

I am also trying to install conda but can compile pycosat

Re: Cant complie LAPLACK with XLC or GCC

[sawyl.livejournal.com]

I'm afraid I can't provide definite help - we retired our AIX systems in September 2015 - but I believe the GNU fortran compiler is not compatible with the IBM ESSL libraries.

It may best to build ATLAS with the GNU compilers and use this to build SciPy and NumPy. This worked when I investigated back in 2012 but we decided not to use it because we were concerned about how long ATLAS took to build and optimise itself on the Power 7.