Scrapy环境安装

由于scrapy目前只能运行在python2上，所以先更新centos上面的python到最新的 Python 2.7.11

更新centos6.8系统和开发工具集到最新

# yum update
# yum groupinstall "Development tools" 

安装python2.7.12

# wget https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz
# tar zxvf Python-2.7.12.tgz 
# cd Python-2.7.12
# ./configure
# make && make install

然后备份原来的python，并把python2.7做软连接到新的位置

# mv /usr/bin/python /usr/bin/python.bak
# ln -s /usr/local/bin/python2.7 /usr/bin/python

#python -V    //验证python   

修改yum指向python目录

# vi /usr/bin/yum
调整#!/usr/bin/python为#!/usr/bin/python2.6

安装pip

# wget https://bootstrap.pypa.io/get-pip.py
# python get-pip.py
# pip -V　　#查看pip版本

先安装一些依赖软件

yum install python-devel
yum install libffi-devel
yum install openssl-devel

然后安装pyopenssl库

pip install pyopenssl

安装xlml

yum install python-lxml
yum install libxml2-devel
yum install libxslt-devel
或者
yum install -y zlib-devel bzip2-devel openssl-devel xz-libs wget

安装service-identity

pip install service-identity

安装twisted

pip install scrapy

安装scrapy

pip install scrapy -U

测试scrapy

scrapy bench

最终成功，太不容易了