【爬虫之路】批量下载5sing上一位歌手的全部歌曲

*16-3-27更新，原方法已失效，仅供参考

实现思路是这样的：
1.访问songer的所有作品列表页，例如：
http://5sing.kugou.com/marblue/fc/1.html
显示的是songer的翻唱作品第一页，修改链接访问所有作品列表页即可。

2.正则表达式抓取作品id号与歌曲名字。

3.以id为14041908的翻唱作品为例，利用id号与歌曲类型生成链接：
http://5sing.kugou.com/fc/DownFile.aspx?SongID=14041908&SongType=fc

4.利用requests模块模拟登陆5sing网站，访问3生成的链接，得到重定向链接，如3例子中的作品，重定向后链接是
http://data1.5sing.kgimg.com/T1EZZPB4YT1R47IVrK.mp3

5.利用4得到的链接与urllib.urlretrieve即可下载歌曲至本地，歌曲名字可以用先前2抓取的名字命名。

ps.由于这是单线程爬虫，下载起来很慢，于是我想了个高效的使用该爬虫的方法，即生成两个表：下载链接表，下载文件名与歌曲名字对应表。利用迅雷读取下载链接表即可。因为下载下来的文件都是如T1EZZPB4YT1R47IVrK.mp3一类的形式，因此可以利用批处理命令与对应表修改mp3文件名便大功告成。

这里以歌手“三无marbule”为例：

#coding:utf-8import requestsimport urllibimport urllib2import reimport osdef redirect(songid , songtype):    params = {'SongID' : str(songid) , 'SongType' : str(songtype)}    headers = {    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',    'Accept-Encoding':'gzip,deflate,sdch',    'Accept-Language':'zh-CN,zh;q=0.8',    # 'Cache-Control':'max-age=0',    'Connection':'keep-alive',    'Cookie':'niceScrollTop=0; _ga=GA1.2.68528838.1409202005; bdshare_firstime=1416837992203; wsp_volume=0.2; wsp_ismuted=0; liveWindow=1; CNZZDATA4559556=cnzz_eid%3D144953663-1417489637-http%253A%252F%252Fopenapi.qzone.qq.com%252F%26ntime%3D1428148526; PHPSESSID=euhdsfqlb5lgqkvd36tn2gt337; 5sing_ssid=euhdsfqlb5lgqkvd36tn2gt337; CNZZDATA5758545=cnzz_eid%3D1849251994-1416835424-http%253A%252F%252F5sing.kugou.com%252F%26ntime%3D1429798455; 5sing_auth=9wyi1oEy05Th8rSQwAXb+K61FrHwZfsPNHoltUMCB+BYjcohtSdq+w==; 5sing_user_info=a%3A3%3A%7Bs%3A7%3A%22wsingId%22%3Bi%3A33169395%3Bs%3A8%3A%22username%22%3Bs%3A9%3A%22%E4%B9%B1%E5%BD%B1%E7%83%9B%22%3Bs%3A6%3A%22avatar%22%3Bs%3A53%3A%22http%3A%2F%2Fimg10.5sing.kgimg.com%2Fm%2FT1MUEPB4AT1RXrhCrK.jpg%22%3B%7D; Anonymous=68ebeb3fff0b4cde817d60830d9b1a7f; area=%E5%85%A8%E9%83%A8; CNZZDATA983423=cnzz_eid%3D1505656430-1414072487-%26ntime%3D1429963731',    'Host':'5sing.kugou.com',    'User-Agent':'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0'    }    url = 'http://5sing.kugou.com/'+ str(songtype) + '/DownFile.aspx'    r = requests.get(url , headers = headers , params = params)    # print r.url    return r.urldef getinfo(url):    html = urllib2.urlopen(url).read()    id_str = '''<a href="http://5sing.kugou.com/fc/(.*?).html" title=".*" target="_blank">'''    id_pat = re.compile(id_str)    id_list = id_pat.findall(html)    name_str = '''<a href="http://5sing.kugou.com/fc/.*.html" title="(.*?)" target="_blank">'''    name_pat = re.compile(name_str)    name_list = name_pat.findall(html)    url_list = []    down_name_list = []    i = 0    for x in id_list:        down_url = redirect(x , 'fc')        url_list.append(down_url)        # down_name = '''5sing.kgimg.com/(.*?).mp3'''        # down_name_pat = re.compile(down_name)        # down_name_list.append({'downname' : down_name_pat.findall(down_url)[0] , 'songname' : name_list[i]})        down_name_list.append({'downurl' : down_url , 'songname' : name_list[i]})        i += 1    return id_list,name_list,url_list,down_name_listdef download(down_name_list):    for x in down_name_list:        path = 'D:\mp3\三无marbule' + str(x['songname']) + '.mp3'        print path + ' downling~\n'        urllib.urlretrieve(str(x['downurl']),path)        print path + ' downloaded~\n'page_num = 4songtype = 'fc'x = 0id_list = []name_list = []url_list = []down_name_list = []while x < page_num:    x += 1    url = 'http://5sing.kugou.com/marblue/' + str(songtype) + '/' + str(x) + '.html'    # print getinfo(url)    id_list += getinfo(url)[0]    name_list += getinfo(url)[1]    url_list += getinfo(url)[2]    down_name_list += getinfo(url)[3]    print down_name_listpath_name1 = 'D:\down_url.txt'path_name2 = 'D:\downurl_and_songname.txt'f = open(path_name1 , 'w+')for x in url_list:    insert_str = x + '\n'    f.write(insert_str)f.close()f = open(path_name2 , 'w+')for x in down_name_list:    insert_str = str(x['downurl']) + str(x['songname']) + '.mp3' + '\n'    f.write(insert_str)f.close()#download(down_name_list)

附1：下载链接表：

http://data1.5sing.kgimg.com/T1EZZPB4YT1R47IVrK.mp3http://data8.5sing.kgimg.com/T11adPB4_T1R47IVrK.mp3http://data1.5sing.kgimg.com/T1kGxnBXZT1R47IVrK.mp3http://data10.5sing.kgimg.com/T1JhKKBmET1R47IVrK.mp3http://data10.5sing.kgimg.com/T10uh5B4_T1R47IVrK.mp3http://data4.5sing.kgimg.com/T10.JKBXWT1R47IVrK.mp3http://data4.5sing.kgimg.com/T1s8DKBC_T1R47IVrK.mp3http://data10.5sing.kgimg.com/T1iPd7BCDT1R47IVrK.mp3http://data8.5sing.kgimg.com/T1cQA5B4YT1R47IVrK.mp3http://data1.5sing.kgimg.com/T1VuxPB4ET1R47IVrK.mp3http://data10.5sing.kgimg.com/T1dPD7B5DT1R47IVrK.mp3http://data3.5sing.kgimg.com/T1eKK4B4hT1R47IVrK.mp3http://data1.5sing.kgimg.com/T1vubXBXLT1R47IVrK.mp3http://data1.5sing.kgimg.com/T1zjCbBXYT1R47IVrK.mp3http://data2.5sing.kgimg.com/T1axdXBXdT1R47IVrK.mp3http://data7.5sing.kgimg.com/T1InAbBXdT1R47IVrK.mp3http://data4.5sing.kgimg.com/T14eVQB4ZT1R47IVrK.mp3http://data9.5sing.kgimg.com/T1ROKXBXCT1R47IVrK.mp3http://data10.5sing.kgimg.com/T1RhVXBXZT1R47IVrK.mp3http://data9.5sing.kgimg.com/T1fRhnByKT1R47IVrK.mp3http://data3.5sing.kgimg.com/T1Q9VLB4DT1R47IVrK.mp3http://data2.5sing.kgimg.com/T1wSELB4JT1R47IVrK.mp3http://data9.5sing.kgimg.com/T10IhLBXZT1R47IVrK.mp3http://data5.5sing.kgimg.com/T1klEPBXdT1R47IVrK.mp3http://data2.5sing.kgimg.com/T1Yp_LByhT1R47IVrK.mp3http://data10.5sing.kgimg.com/T1CKCTB4ET1R47IVrK.mp3http://data6.5sing.kgimg.com/T1gmdTB4KT1R47IVrK.mp3http://data1.5sing.kgimg.com/T176KTB4ZT1R47IVrK.mp3http://data2.5sing.kgimg.com/T1AExTB4LT1R47IVrK.mp3http://data10.5sing.kgimg.com/T1CiEPB4_T1R47IVrK.mp3http://data7.5sing.kgimg.com/T1T.CPB4DT1R47IVrK.mp3http://data9.5sing.kgimg.com/T1nydTByCT1R47IVrK.mp3http://data6.5sing.kgimg.com/T1JQVeBXKT1R47IVrK.mp3http://data3.5sing.kgimg.com/T1vjbeBybT1R47IVrK.mp3http://data8.5sing.kgimg.com/T1knCeBybT1R47IVrK.mp3http://data6.5sing.kgimg.com/T1g7KeBXWT1R47IVrK.mp3http://data4.5sing.kgimg.com/T1a2EeBydT1R47IVrK.mp3http://data10.5sing.kgimg.com/T1c4xeBXJT1R47IVrK.mp3http://data3.5sing.kgimg.com/T1G_K7BXYT1R47IVrK.mp3http://data5.5sing.kgimg.com/T1LJW7ByAT1R47IVrK.mp3http://data4.5sing.kgimg.com/T1cKx7BybT1R47IVrK.mp3http://data2.5sing.kgimg.com/T153_KBXWT1R47IVrK.mp3http://data7.5sing.kgimg.com/T1oBb5BXCT1R47IVrK.mp3http://data5.5sing.kgimg.com/T1DlA5BXAT1R47IVrK.mp3http://data8.5sing.kgimg.com/T1SnJ5ByDT1R47IVrK.mp3http://data6.5sing.kgimg.com/T1EJD5BXZT1R47IVrK.mp3http://data10.5sing.kgimg.com/T1gIW5BX_T1R47IVrK.mp3http://data10.5sing.kgimg.com/T1AibmBXhT1R47IVrK.mp3http://data1.5sing.kgimg.com/T14JYmB4ET1R47IVrK.mp3http://data9.5sing.kgimg.com/T1soxmByJT1R47IVrK.mp3http://data6.5sing.kgimg.com/T1eI_4BX_T1R47IVrK.mp3http://data7.5sing.kgimg.com/T10M_bByAT1R47IVrK.mp3http://data6.5sing.kgimg.com/T141dCByLT1R47IVrK.mp3http://data9.5sing.kgimg.com/T1KDZCBy_T1R47IVrK.mp3http://data10.5sing.kgimg.com/T1dShCBXhT1R47IVrK.mp3http://data8.5sing.kgimg.com/T1GgDyBXZT1R47IVrK.mp3http://data10.5sing.kgimg.com/T174dCByWT1R47IVrK.mp3http://data10.5sing.kgimg.com/T1TMKyByAT1R47IVrK.mp3http://data6.5sing.kgimg.com/T1a6ZyBXDT1R47IVrK.mp3http://data9.5sing.kgimg.com/T1qnVyByZT1R47IVrK.mp3http://data1.5sing.kgimg.com/T1JBEsByYT1R47IVrK.mp3http://data6.5sing.kgimg.com/T1OXhXBybT1R47IVrK.mp3http://data5.5sing.kgimg.com/T1VthXByxT1R47IVrK.mp3http://data1.5sing.kgimg.com/T1SLCsBXdT1R47IVrK.mp3http://data1.5sing.kgimg.com/T1IcAsByKT1R47IVrK.mp3http://data2.5sing.kgimg.com/T1d5LsBXET1R47IVrK.mp3http://data9.5sing.kgimg.com/T1T6KsByLT1R47IVrK.mp3

附2：下载文件名与歌曲文件名对应表（记得要删除文件名中的非法字符）：

rename T1EZZPB4YT1R47IVrK.mp3 【ilem】今天没吃药 with.ilem.mp3rename T11adPB4_T1R47IVrK.mp3 【填词】化作星辰.mp3rename T1kGxnBXZT1R47IVrK.mp3 【梅とら】一騎当千.mp3rename T1JhKKBmET1R47IVrK.mp3 【ilem】阴阳先生.mp3rename T10uh5B4_T1R47IVrK.mp3 【ilem】怪物大暴走.mp3rename T10.JKBXWT1R47IVrK.mp3 【V家】神のまにまに/神的随波逐流 with.Midaho.mp3rename T1s8DKBC_T1R47IVrK.mp3 【ilem】僵尸舞.mp3rename T1iPd7BCDT1R47IVrK.mp3 【匀之家】想象森林 【盖盖Arrange ver.】.mp3rename T1cQA5B4YT1R47IVrK.mp3 【V家】Crazy ∞ nighT ver.匀雪三五东粗爵布.mp3rename T1VuxPB4ET1R47IVrK.mp3 【umedy（梅とら）】miss you X'mas【HB to 匀子】.mp3rename T1dPD7B5DT1R47IVrK.mp3 【火影OP15】Sign （full ver.）.mp3rename T1eKK4B4hT1R47IVrK.mp3 【天堂之吻op】Lonely in Gorgeous.mp3rename T1vubXBXLT1R47IVrK.mp3 【渣麦】六等星の夜 short ver. .mp3rename T1zjCbBXYT1R47IVrK.mp3 【A/Z】aLIEz.mp3rename T1axdXBXdT1R47IVrK.mp3 【三黄云猕菠饼爽】キミがいる.mp3rename T1InAbBXdT1R47IVrK.mp3 【sekai no owari】花鳥風月 （HB to 树粗）.mp3rename T14eVQB4ZT1R47IVrK.mp3 【HB to 洛萱】Freely Tomorrow【with.克】.mp3rename T1ROKXBXCT1R47IVrK.mp3 和女神唱马鹿摸你.mp3rename T1RhVXBXZT1R47IVrK.mp3 【火影op13】CLOSER【HB to 阿布布】.mp3rename T1fRhnByKT1R47IVrK.mp3 【中文】化身孤岛的鲸.mp3rename T1Q9VLB4DT1R47IVrK.mp3 【V家】Lie 【HB to 豆浆浆】.mp3rename T1wSELB4JT1R47IVrK.mp3 【V家】脳浆炸裂ガール with.小五.mp3rename T10IhLBXZT1R47IVrK.mp3 【三无X洛萱】When They Cry-Rosin Mix-.mp3rename T1klEPBXdT1R47IVrK.mp3 【野良神OP】午夜の待ち合わせ -TV size-.mp3rename T1Yp_LByhT1R47IVrK.mp3 【V家】さよならのかわりに/再见的代替 with.爽爽.mp3rename T1CKCTB4ET1R47IVrK.mp3 【V家】おこちゃま戦争/孩子气的战争 with.布雷.mp3rename T1gmdTB4KT1R47IVrK.mp3 【V家】ロストワンの号哭-piano.ver-.mp3rename T176KTB4ZT1R47IVrK.mp3 【V家】ゴローンゴローン～フンコロガシ～ with. 夕准.mp3rename T1AExTB4LT1R47IVrK.mp3 【V家】深海シティアンダーグラウンド/深海地下都市.mp3rename T1CiEPB4_T1R47IVrK.mp3 【AAA】恋音と雨空 ver.云匀五嘟露MO三.mp3rename T1T.CPB4DT1R47IVrK.mp3 【m-flo】Summer Time Love 【克ft.me】.mp3rename T1nydTByCT1R47IVrK.mp3 【V家】番凩 with.凤冴.mp3rename T1JQVeBXKT1R47IVrK.mp3 【V家】オサナナブルー/青梅竹马的忧郁.mp3rename T1vjbeBybT1R47IVrK.mp3 【V家】ムラサキ/紫.mp3rename T1knCeBybT1R47IVrK.mp3 【芽纱岚三五凛】MUGIC.mp3rename T1g7KeBXWT1R47IVrK.mp3 【冥月XMOKONAX三无】SPLASH FREE.mp3rename T1a2EeBydT1R47IVrK.mp3 【哲♂学】恋爱フィロソフィア【173×三无】.mp3rename T1c4xeBXJT1R47IVrK.mp3 【东方】Baby，baby with.夕准.mp3rename T1G_K7BXYT1R47IVrK.mp3 【ALTIMA】one （with.小空）.mp3rename T1LJW7ByAT1R47IVrK.mp3 【V家】Bad ∞ End ∞ Night ver.匀雪三五东粗爵布.mp3rename T1cKx7BybT1R47IVrK.mp3 【菅野洋子】月光シンフォニア with.小五.mp3rename T153_KBXWT1R47IVrK.mp3 【HOME MADE家族】Looking For You.mp3rename T1oBb5BXCT1R47IVrK.mp3 【林肯公园】威风堂々【Remix】.mp3rename T1DlA5BXAT1R47IVrK.mp3 【..】我要金坷垃【速产渣慎..】.mp3rename T1SnJ5ByDT1R47IVrK.mp3 「Loveliest」with  MOKONA.mp3rename T1EJD5BXZT1R47IVrK.mp3 【V家】般若心経Ｒ＆Ｂ【halyosy ver.】.mp3rename T1gIW5BX_T1R47IVrK.mp3 【V家】威風堂々.mp3rename T1AibmBXhT1R47IVrK.mp3 【清水翔太】home.mp3rename T14JYmB4ET1R47IVrK.mp3 【V家】WAVE【 HB to 小五】.mp3rename T1soxmByJT1R47IVrK.mp3 【交响诗篇 OP2】少年ハート.mp3rename T1eI_4BX_T1R47IVrK.mp3 【天野月子】月.mp3rename T10M_bByAT1R47IVrK.mp3 【V家】rain stops ,good-bye 【HB to 佑可猫】.mp3rename T141dCByLT1R47IVrK.mp3 【V家】イカサマ カジノ（欺骗赌场）feat.布雷.mp3rename T1KDZCBy_T1R47IVrK.mp3 【魂音泉】フラワリングナイト x そんなゆめをみたの ~lonely dreaming girl~.mp3rename T1dShCBXhT1R47IVrK.mp3 【原创】東京テディベア piano short ver.FL自制慎入.mp3rename T1GgDyBXZT1R47IVrK.mp3 【V家】愛言葉 （纯人声 ver.）.mp3rename T174dCByWT1R47IVrK.mp3 【V家】muddy cloud.mp3rename T1TMKyByAT1R47IVrK.mp3 【羊之歌】Destiny~宿命~feat.匀子【HB to 扬舅】.mp3rename T1a6ZyBXDT1R47IVrK.mp3 【SH】恋人に射ち堕とされた日feat.扬舅.mp3rename T1qnVyByZT1R47IVrK.mp3 【V家中文】晓秋月明.mp3rename T1JBEsByYT1R47IVrK.mp3 【レインブック】千本桜【HB to 匀子】.mp3rename T1OXhXBybT1R47IVrK.mp3 バイバイ.mp3rename T1VthXByxT1R47IVrK.mp3 エウテルペ.mp3rename T1SLCsBXdT1R47IVrK.mp3 混沌.mp3rename T1IcAsByKT1R47IVrK.mp3 ベルベットの祈りfeat.苔藓.mp3rename T1d5LsBXET1R47IVrK.mp3 天命风流.mp3rename T1T6KsByLT1R47IVrK.mp3 The Thorn Bird 荆棘鸟。feat.小雪.mp3