如何使用 Python 中的 BeautifulSoup 从 Body 标签中抓取所有文本?

pythonserver side programmingprogramming

Web 抓取是一种用于从网站中提取数据的强大技术。Python 中一个流行的 Web 抓取库是 BeautifulSoup。BeautifulSoup 提供了一种简单直观的方法来解析 HTML 或 XML 文档并提取所需的信息。在本文中，我们将探讨如何从 <body> 中抓取所有文本使用 Python 中的 BeautifulSoup 从网页的标签中抓取所有文本。

算法

以下算法概述了使用 BeautifulSoup 从 body 标签中抓取所有文本的步骤:

导入所需的库:我们需要导入请求库来发出 HTTP 请求，并从 bs4 模块中导入 BeautifulSoup 类来解析 HTML。
发出 HTTP 请求:使用 request.get() 函数向您要抓取的网页发送 HTTP GET 请求。
解析 HTML 内容:通过传递 HTML 内容并指定解析器来创建 BeautifulSoup 对象。通常，默认解析器是 html.parser，但您也可以使用 lxml 或 html5lib 等替代方法。
查找 body 标签:使用 BeautifulSoup 对象上的 find() 或 find_all() 方法定位 <body> 标签。find() 方法返回第一次出现，而 find_all() 返回所有出现次数的列表。
提取文本:找到 body 标签后，您可以使用 get_text() 方法提取文本内容。此方法返回所选标签及其所有后代的连接文本。
处理文本:对提取的文本执行任何必要的处理，例如清理、过滤或分析。
打印或存储输出:显示提取的文本或将其保存到文件、数据库或任何其他所需目标。

语法

soup = BeautifulSoup(html_content, 'html.parser')

此处，html_content 表示要解析的 HTML 文档，而 'html.parser' 是 Beautiful Soup 用于解析 HTML 的解析器。

tag = soup.find('tag_name')

find() 方法在解析的 HTML 文档中定位指定 HTML 标记(例如 <tag_name>)的第一次出现，并返回相应的 BeautifulSoup Tag 对象。

text = tag.get_text()

get_text() 方法从指定的标记对象中提取文本内容。

示例

以下代码将从 openai 网页的 body 标记中打印所有文本内容。输出可能会因您选择抓取的网页而异。

import 请求
from bs4 import BeautifulSoup

# 发出 HTTP 请求
url = 'https://openai.com/'
response = request.get(url)

# 解析 HTML 内容
soup = BeautifulSoup(response.content, 'html.parser')

# 查找 body 标签
body = soup.find('body')

# 提取文本
text = body.get_text()

# 打印输出
print(text)

输出

CloseSearch Submit Skip to main contentSite 
NavigationResearchOverviewIndexProductOverviewChatGPTGPT-4DALL
·E 2Customer storiesSafety standardsPricingDevelopersOverviewDocumentationAPI
referenceExamplesSafetyCompanyAboutBlogCareersCharterSecuritySearch 
Navigation quick links Log inSign upMenu Mobile Navigation
CloseSite NavigationResearchProductDevelopersSafetyCompany 
Quick Links Log inSign upSearch Submit  Your browser 
does not support the video tag. Introducing the ChatGPT 
app for iOSQuicklinksDownload on the App StoreLearn more
about ChatGPTCreating safe AGI that benefits all of
humanityLearn about OpenAIPioneering research on the 
path to AGILearn about our researchTransforming work and 
creativity with AIExplore our productsJoin us in shaping 
the future of technologyView careersSafety & responsibilityOur 
work to create safe and beneficial AI requires a deep
understanding of the potential risks and benefits, as 
well as careful consideration of the impact.Learn about 
safetyResearchWe research generative models and how to
align them with human values.Learn about our researchGPT-4Mar
14, 2023March 14, 2023Forecasting potential misuses of 
language models for disinformation campaigns and how to 
reduce riskJan 11, 2023January 11, 2023Point-E: A system 
for generating 3D point clouds from complex promptsDec 16, 
2022December 16, 2022Introducing WhisperSep 21, 2022September
21, 2022ProductsOur API platform offers our latest models 
and guides for safety best practices.Explore our productsNew 
and improved embedding modelDec 15, 2022December 15, 2022DALL
·E now available without waitlistSep 28, 2022September 28,
2022New and improved content moderation toolingAug 10,
2022August 10, 2022New GPT-3 capabilities: Edit & 
insertMar 15, 2022March 15, 2022Careers at OpenAIDeveloping
safe and beneficial AI requires people from a wide range 
of disciplines and backgrounds.View careersI encourage my 
team to keep learning. Ideas in different topics or fields 
can often inspire new ideas and broaden the potential solution
space.Lilian WengApplied AI at OpenAIResearchOverviewIndexProductOverviewGPT-4DALL·
E 2Customer storiesSafety standardsPricingSafetyOverviewCompanyAboutBlogCareersCharterSecurityOpenAI 
© 2015 – 2023Terms & policiesPrivacy policyBrand guidelinesSocialTwitterYouTubeGitHubSoundCloudLinkedInBack
to top