[原创]论坛悬赏区的爬取和分析，了解高频问题更好挣CB

本文来源吾爱破解论坛

本帖最后由 天空宫阙 于 2019-3-9 20:43 编辑

入门级网页爬虫的简单应用，大神就不用看了
先放结果

result.jpg (303.47 KB, 下载次数: 1)

下载附件  保存到相册

2019-3-9 13:17 上传

吾爱悬赏区词频分析.jpg (114.93 KB, 下载次数: 1)

下载附件  保存到相册

2019-3-9 13:28 上传

import requests
from requests.exceptions import RequestException
from bs4 import BeautifulSoup
import re
import time
#请求头的修改
headers={
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36'
}
#请求一个网页
def get_html(url):
    try:
        response = requests.get(url,headers=headers)
        response.raise_for_status()
        #print(response.apparent_encoding)
        response.encoding = 'GB2312'
        return response.text
    except:
        print('请求失败')


def parse_html(html):
    soup = BeautifulSoup(html, "lxml")
    thread = soup.select('#threadlisttableid')[0]
    tbodys = thread.find_all(id = re.compile('normalthread'))
    for tbody in tbodys:
        title = tbody.select('a')[2].string
        try:
            with open('wuai_statistics.txt', 'a', encoding='utf-8') as f:
                f.write(title+'\n')
                f.close()
        except:
            continue
        
def main(page):
    url='https://www.52pojie.cn/forum.php?mod=forumdisplay&fid=8&specialtype=reward&filter=specialtype&specialtype=reward&rewardtype=1&page='+str(page)
    html = get_html(url)
    parse_html(html)

if __name__ == '__main__':
    for page in range(1,201):
        print('正在保存第'+str(page)+'页')
        main(page)
        print('成功')

import numpy as np
from PIL import Image
import re
import jieba
from wordcloud import WordCloud,ImageColorGenerator,STOPWORDS
import  matplotlib.pyplot as plt

# 打开存放项目名称的txt文件
with open('wuai_statistics.txt','r',encoding='utf-8') as f:
    word= (f.read())
    f.close()
'''
# 图片模板和字体
image=np.array(Image.open('ditu.jpg'))
'''
font=r'C:\\Windows\\fonts\\msyh.ttf' 

# 去掉英文，保留中文 
resultword=re.sub("[A-Za-z0-9\[\`\~\!\@\#\$\^\&\*\(\)\=\|\{\}\'\:\;\'\,\[\]\.\<\>\/\?\~\。\@\#\\\&\*\%]", "",word) 
wordlist_after_jieba = jieba.cut(resultword) 
wl_space_split = " ".join(wordlist_after_jieba) 

# 设置停用词 
sw = set(STOPWORDS) 
sw.add("一个")
sw.add("求个")
sw.add("可以")
sw.add("怎么")
sw.add("如何")
sw.add("求助")
sw.add("一份")
sw.add("或者")
sw.add("有没有")
sw.add("一款")
sw.add("帮忙")

# 关键一步
my_wordcloud = WordCloud(scale=4,font_path=font,stopwords=sw,background_color='white',
                         max_words = 100,max_font_size = 60,random_state=20).generate(wl_space_split) 

#显示生成的词云 
plt.imshow(my_wordcloud)
plt.axis("off") 
plt.show() 

#保存生成的图片
my_wordcloud.to_file('result.jpg')

wuai_statistcs.zip (1.85 KB, 下载次数: 5) 2019-3-9 13:54 上传 点击文件名下载附件
下载积分: 吾爱币 -1 CB