利用xpath建标签树以后,虽然提高了元素匹配效率,但是etree会把中文转为ASCII码,所以简单地tostring以后会有乱码。
解决方法:
import requestsfrom requests.exceptions import RequestExceptionfrom lxml import etreeheaders = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.2 Safari/605.1.15',}def get_one_page(url, headers): try: response = requests.get(url, headers=headers) if response.status_code == 200: response.encoding = response.apparent_encoding return response.text return None except RequestException: return Nonetree = etree.HTML(html)aim = tree.xpath(exp)for i in aim: content = etree.tostring(i, encoding='utf-8', pretty_print=True, method="html").decode('utf-8')