Do I Need 32bit Libxml2 For Python On Snow Leopard?

i'm having a hell of a time installing scrapy on my sl mbp. it requires libxml2, so i set about installing that. installing it from macports doesn't seem to pull down the python bi

Solution 1:

From the paths in your traceback, it appears you have installed and are trying to use the python.org python 2.6.4 (installed to /Library/Frameworks/Python.frameworks ...). That python is 32-bit only. By default on 10.6, MacPorts tries to install 64-bit versions of packages. You can change that for most MacPorts packages by specfying the +universal variant on your MacPorts install commands or by adding it to /opt/local/etc/macports/variants.conf. But, since you need a variety of packages, if you just need this for your own machine you'll likely find it much easier to just install a complete python2.6 64-bit solution using MacPorts. Other than scrapy itself, you should find nearly all of the packages you need already there. You'll need to modify your $PATH to add /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin and /opt/local/bin before any other directories with Python in them. Something like this should work (untested!):

sudo port selfupdate  # ensure you have the latest ports file information
sudo port install py26-libxml2 py26-twisted py26-openssl py26-simplejson py26-setuptools python_select
sudo python_select python26  # optionally make /opt/local/bin/python -> python2.6
sudo /opt/local/bin/easy_install-2.6 scrapy
# or install manuallycd /path/to/scrapy
sudo /opt/local/bin/python2.6 setup.py install

EDIT: For 10.6, due to some issues with lack of Tk support in 64-bit mode, Tk defaults to the X11 version which you probably neither want nor need. I add the following variants to /opt/local/etc/macports/variants.conf for 10.6 (not all of them are applicable to python and friends):

+bash_completion +quartz +ssl +no_x11 +no_neon +no_tkinter +universal +libyaml -scientific

EDIT: If MacPorts is installed properly, you should see something like this on 10.6:

$ ls -l /opt/local/bin/python2.6
lrwxr-xr-x  1 root  wheel  73 Oct 28 20:25 /opt/local/bin/python2.6@ -> /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
$ file /opt/local/bin/python2.6
/opt/local/bin/python2.6: Mach-O universal binary with 2 architectures
/opt/local/bin/python2.6 (for architecture x86_64): Mach-O 64-bit executable x86_64
/opt/local/bin/python2.6 (for architecture i386):   Mach-O executable i386

That's if you've added the +universal variant, otherwise you may just see a single x86_64 or i386 architecture.

When using relative paths, make sure your shell $PATH has the two MacPorts directories first:

$ echo$PATH
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin ...

You may need to edit your ~/.bash_profile or other shell file to make that change "permanent". If you want the bare python command to refer to the MacPorts Python, make sure to run the python_select command shown above. If you are still having problems, a nice thing about MacPorts is it is easy to delete and start over again; just:

$ sudo rm -r /opt/local

and download the MacPorts 10.6 installer. But you really shouldn't have to do that. And I think it would really be in your best interests to get this fixed and working as it will likely save you lots of headaches in the future.

Solution 2:

I have a small note to contribute. While attempting to follow-along on this post, I began with three installs of Python:

1. The default Apple Python 2.5.1 located at: /usr/bin/python
2. A version located: /Library/Frameworks/Python.framework/Versions/2.7
3. And a Macport version located: /opt/local/bin/python2.6

My trouble was that:

$ python

would always default to the 2.7 when I needed it to use the Macports version. The following did not help:

$ sudo python_select python26

I even removed the 2.7 version which caused only an error.

I figured out I needed to change the default path to the Macports version using the following:

$ PATH=$PATH\:/opt/local/bin ; export PATH

And then reinitiate the ports, etc.

Finally, I was not able to reference the scrapy-ctl.py file by default through these instructions so I had to reference the scrapy-ctl.py file directly

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py

UPDATE

A quick addendum to this post with instructions to create the link, found on the Scrapy site (#2 and #3).

Starting with #2, "Add Scrapy to your Python Path"

sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scrapy

And #3, "Make the scrapy command available"

sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py /usr/local/bin/scrapy

Solution 3:

Trying to install but still struggling. When I:

sudo /opt/local/bin/easy_install-2.6 scrapy

I get error message and it stops

AttributeError: 'NoneType'object has no attribute 'get'

Instead I build Scrapy from scratch which works fine.

Then I try what you write

cd /path/to/scrapy
sudo /opt/local/bin/python2.6 setup.py install

Where is /path/to/scrapy? Is that:

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py

or

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Scrapy-0.8-py2.6.egg

or is somewhere else?

Solution 4:

Credit to @Ned Deily

These steps seem to work if you want to run Scrapy 0.8 on OS X 10.6. It uses Macports install of Python 2.6 rather than the one bundled with the OS. Steps assume Macports is not installed yet.

Get latest MacPorts installer from here and install:

http://www.macports.org/install.php

sudo port install py26-libxml2 py26-twisted py26-openssl py26-simplejson py26-setuptools python_select

sudo /opt/local/bin/easy_install-2.6 scrapy

Change your ~.profile to:

exportPATH=/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:$PATH