What library does Python use to write reptiles?

Python crawler, full name Python web crawler, is a program or script that automatically crawls information on the World Wide Web according to certain rules. It is mainly used to capture securities trading data, weather data, website user data, picture data, etc. In order to support the normal function of web crawler, Python has a large number of built-in libraries, mainly including several types. The following article will give you an introduction.

First, Python crawler network library

Python crawler network libraries mainly include: urllib, requests, grab, pycurl, urllib3, httplib2, RoboBrowser, MechanicalSoup, mechanize, socket, Python unique, hyper, PySocks, treq, aiohttp.

Second, Python web crawler framework

Python web crawler framework mainly includes: grab, scrapy, pyspider, cola, portia, restkit and demiurge.

Third, the HTML/XML parser?

LXML: An efficient HTML/ XML processing library written in C language. Support XPath.

●cssselect: parse DOM tree and CSS selector.

●pyquery: parse DOM tree and jQuery selector.

●BeautifulSoup: an inefficient HTML/ XML processing library, implemented in Python.

●html5lib: generate DOM of HTML/ XML document according to WHATWG specification. Now all browsers use this specification.

●feedparser: parse RSS/ATOM feed.

●MarkupSafe: provide a safe escape string for XML/HTML/XHTML.

●xmltodict: A Python module, which can make you feel that you are dealing with JSON when dealing with XML.

●xhtml2pdf: Convert HTML/CSS into pdf.

● Untangling: It is easy to convert XML files into Python objects.

Fourth, text processing.

A library for parsing and manipulating simple text.

● difflib: (Python standard library) helps to do differential comparison.

●Levenshtein: Quickly calculate Levenshtein distance and Levenshtein.

●fuzzywuzzy: Fuzzy string matching.

●esmre: regular expression accelerator.

●ftfy: Automatically organize Unicode text to reduce fragmentation.

Five, the specific format file processing

A library that parses and processes specific text formats.

●tablib: a module that exports data into XLS, CSV, JSON, YAML and other formats.

●textract: Extract text from various files, such as Word, PowerPoint, PDF, etc.

●messytables: a tool for parsing chaotic table data.

●rows: universal data interface, supporting multiple formats (currently supporting CSV, HTML, XLS, TXT: more will be provided in the future! )。