HTML CSS JAVASCRIPT JQUERY AI PHP SQL EXCEL LINUX C# C++ C PYTHON JAVA 教程库 参考手册 技术文章 测验 练习 HOWTO FAQ

如何在 Pandas 中查找和过滤重复行?

pythonserver side programmingprogramming更新于 2026/2/16 22:36:17

有时在数据分析过程中，我们需要查看重复行以更好地了解数据，而不是直接删除它们。

幸运的是，在 Pandas 中，我们有一些方法可以处理重复项。

.duplciated()

此方法允许我们提取 DataFrame 中的重复行。我们将使用包含重复项的新数据集。我已经从链接下载了 Hr 数据集。

import pandas as pd
import numpy as np

# 导入具有特定列的 HR 数据集
df = pd.read_csv("https://raw.githubusercontent.com/sasankac/TestDataSet/master/HRDataset.csv",
usecols = ("Employee_Name""PerformanceScore","Position","CitizenDesc"))

# 按员工姓名对值进行排序并使其永久保存
df.sort_values("Employee_Name"inplace = True)
df.head(3)



Employee_Name
Position
CitizenDesc
PerformanceScore
0
Adinolfi
Production Technician I
US Citizen
Exceeds
1
Adinolfi
Sr. DBA
US Citizen
Fully Meets
2
Adinolfi
Production Technician II
US Citizen
Fully Meets


默认情况下，duplicated() 的工作方式是通过 keep 参数，此参数将标记每个值的第一次出现为非重复值。

如果某行出现多次，此方法不会将其标记为重复，而是将第一行之后的每个后续行标记为重复。困惑吗?让我再举一个例子来解释一下，假设篮子里有 3 个苹果，此方法的作用是将第一个苹果标记为非重复，将其余两个苹果标记为重复。

示例

df["Employee_Name"].head(3)

输出

0 Adinolfi
1 Adinolfi
2 Adinolfi
Name: Employee_Name, dtype: object

示例

df["Employee_Name"].duplicated().head(3)

输出

0 False
1 True
2 True
Name: Employee_Name, dtype: bool

现在要提取出重复项(记住第一次出现不是重复项，而是后续出现的重复项，并将通过此方法输出)，我们需要将此方法传递给数据框。

df.shape



(310, 4)



df[df["Employee_Name"].duplicated()]



Employee_Name
Position
CitizenDesc
PerformanceScore
1
Adinolfi
Sr. DBA
US Citizen
Fully Meets
2
Adinolfi
Production Technician II
US Citizen
Fully Meets
3
Adinolfi
Production Technician I
US Citizen
Fully Meets
4
Adinolfi
Production Manager
US Citizen
Fully Meets
6
Anderson
Production Technician I
US Citizen
Exceeds
...
...
...
...
...
303
Wang
Production Technician II
US Citizen
Fully Meets
304
Wang
Production Technician II
US Citizen
Fully Meets
305
Wang
Production Technician I
US Citizen
PIP
306
Wang
CIO
US Citizen
Exceeds
307
Wang
Data Analyst
US Citizen
Fully Meets

79 行 × 4 列

从上面的输出中，有 310 行，其中 79 个重复项是使用 .duplicated() 方法提取的。

参数 -"LAST"

默认情况下，此方法将标记第一次出现的值作为非重复项，我们可以通过传递参数 keep = last 来更改此行为。

此参数将执行的操作是将前两个苹果标记为重复项，将最后一个苹果标记为非重复项。

df[df["Employee_Name"].duplicated(keep="last")]



Employee_Name
Position
CitizenDesc
PerformanceScore
0
Adinolfi
Production Technician I
US Citizen
Exceeds
1
Adinolfi
Sr. DBA
US Citizen
Fully Meets
2
Adinolfi
Production Technician II
US Citizen
Fully Meets
3
Adinolfi
Production Technician I
US Citizen
Fully Meets
5
Anderson
Production Technician I
US Citizen
Fully Meets
...
...
...
...
...
302
Wang
Production Technician II
US Citizen
Exceeds
303
Wang
Production Technician II
US Citizen
Fully Meets
304
Wang
Production Technician II
US Citizen
Fully Meets
305
Wang
Production Technician I
US Citizen
PIP
306
Wang
CIO
US Citizen
Exceeds

ARGUMENT - FALSE

keep 参数还将接受一个附加参数"false"，该参数将把所有出现超过一次的值标记为重复，在我们的例子中，所有 3 个苹果都将被标记为重复，而不是如上例中所示的第一个或最后一个。

注意 - 指定 false 参数时不要使用引号。

df[df"Employee_Name"].duplicated(keep=False)]



Employee_Name
Position
CitizenDesc
PerformanceScore
0
Adinolfi
Production Technician I
US Citizen
Exceeds
1
Adinolfi
Sr. DBA
US Citizen
Fully Meets
2
Adinolfi
Production Technician II
US Citizen
Fully Meets
3
Adinolfi
Production Technician I
US Citizen
Fully Meets
4
Adinolfi
Production Manager
US Citizen
Fully Meets
...
...
...
...
...
303
Wang
Production Technician II
US Citizen
Fully Meets
304
Wang
Production Technician II
US Citizen
Fully Meets
305
Wang
Production Technician I
US Citizen
PIP
306
Wang
CIO
US Citizen
Exceeds
307
Wang
Data Analyst
US Citizen
Fully Meets

Now finally, to extract the unique values from a dataset we can use the “~” (tilda) symbol to negate the values

df_unique~df["Employee_Name"].duplicated(keep=False)df[df_unique]



Employee_Name
Position
CitizenDesc
PerformanceScore
7
Andreola
Software Engineer
US Citizen
Fully Meets
25
Bozzi
Production Manager
US Citizen
Fully Meets
26
Bramante
Director of Operations
US Citizen
Exceeds
27
Brill
Production Technician I
US Citizen
Fully Meets
34
Burkett
Production Technician II
US Citizen
Fully Meets
...
...
...
...
...
276
Sweetwater
Software Engineer
US Citizen
Exceeds
277
Szabo
Production Technician I
Non-Citizen
Fully Meets
278
Tavares
Production Technician II
US Citizen
Fully Meets
308
Zhou
Production Technician I
US Citizen
Fully Meets
309
Zima
NaN
NaN
NaN

drop_duplicates()

此方法与前一种方法非常相似，但是此方法可以在 DataFrame 上而不是单个系列上进行。

注意:- 此方法在 DataFrame 的所有列上查找重复的行并将其删除。

len(df)

输出

310



len(df.drop_duplicates())

输出

290

子集参数

子集参数接受列名列表作为字符串值，我们可以在其中检查重复项。

df1=df.drop_duplicates(subset=["Employee_Name"],keep="first")df1

Employee_Name
Position
CitizenDesc
PerformanceScore
0
Adinolfi
Production Technician I
US Citizen
Exceeds
5
Anderson
Production Technician I
US Citizen
Fully Meets
7
Andreola
Software Engineer
US Citizen
Fully Meets
14
Athwal
Production Technician I
US Citizen
Fully Meets
20
Beak
Production Technician I
US Citizen
Fully Meets
...
...
...
...
...
293
Von Massenbach
Production Technician II
US Citizen
Fully Meets
295
Wallace
Production Technician I
US Citizen
Needs Improvement
300
Wang
Production Technician I
Eligible NonCitizen
Fully Meets
308
Zhou
Production Technician I
US Citizen
Fully Meets
309
Zima
NaN
NaN
NaN

我们可以指定多列并使用上一节讨论的所有保留参数。

df1=df.drop_duplicates(subset="Employee_Name""CitizenDesc"],keep=False)df1



Employee_Name
Position
CitizenDesc
PerformanceScore
7
Andreola
Software Engineer
US Citizen
Fully Meets
16
Beak
Production Technician I
Eligible NonCitizen
Fully Meets
25
Bozzi
Production Manager
US Citizen
Fully Meets
26
Bramante
Director of Operations
US Citizen
Exceeds
27
Brill
Production Technician I
US Citizen
Fully Meets
...
...
...
...
...
287
Tejeda
Network Engineer
Eligible NonCitizen
Fully Meets
286
Tejeda
Software Engineer
Non-Citizen
Fully Meets
300
Wang
Production Technician I
Eligible NonCitizen
Fully Meets
308
Zhou
Production Technician I
US Citizen
Fully Meets
309
Zima
NaN
NaN
NaN

unique() 方法

unique 方法查找一系列中的唯一值，并将唯一值作为数组返回。此方法不排除缺失值。

len(df["Employee_Name"])

输出

310



df["Employee_Name"].unique()



array(['Adinolfi', 'Anderson', 'Andreola', 'Athwal', 'Beak', 'Bondwell',
'Bozzi', 'Bramante', 'Brill', 'Brown', 'Burkett', 'Butler',
'Carabbio', 'Carey', 'Carr', 'Carter', 'Chace', 'Champaigne',
'Chan', 'Chang', 'Chivukula', 'Cierpiszewski', 'Cisco', 'Clayton',
'Cloninger', 'Close', 'Clukey', 'Cockel', 'Cole', 'Cornett',
'Costa', 'Crimmings', 'Daneault', 'Daniele', 'Darson', 'Davis',
'DeGweck', 'Del Bosque', 'Demita', 'Desimone', 'DiNocco',
'Dickinson', 'Dietrich', 'Digitale', 'Dobrin', 'Dolan', 'Dougall',
'Dunn', 'Eaton', 'Employee_Name', 'Engdahl', 'England', 'Erilus',
'Estremera', 'Evensen', 'Exantus', 'Faller', 'Fancett', 'Favis',
'Ferguson', 'Fernandes', 'Ferreira', 'Fidelia', 'Fitzpatrick',
'Foreman', 'Foss', 'Foster-Baker', 'Fraval', 'Friedman', 'Galia',
'Garcia', 'Garneau', 'Gaul', 'Gentry', 'Gerke', 'Gill', 'Gonzales',
'Gonzalez', 'Good', 'Handschiegl', 'Hankard', 'Harrison',
'Heitzman', 'Horton', 'Houlihan', 'Howard', 'Hubert', 'Hunts',
'Hutter', 'Huynh', 'Immediato', 'Ivey', 'Jackson', 'Jacobi',
'Jeannite', 'Jeremy Prater', 'Jhaveri', 'Johnson', 'Johnston',
'Jung', 'Kampew', 'Keatts', 'Khemmich', 'King', 'Kinsella',
'Kirill', 'Knapp', 'Kretschmer', 'LaRotonda', 'Lajiri', 'Langford',
'Langton', 'Latif', 'Le', 'LeBel', 'LeBlanc', 'Leach', 'Leruth',
'Liebig', 'Linares', 'Linden', 'Lindsay', 'Lundy', 'Lunquist',
'Lydon', 'Lynch', 'MacLennan', 'Mahoney', 'Manchester', 'Mancuso',
'Mangal', 'Martin', 'Martins', 'Maurice', 'McCarthy', 'McKinzie',
'Mckenna', 'Meads', 'Medeiros', 'Merlos', 'Miller', 'Monkfish',
'Monroe', 'Monterro', 'Moran', 'Morway', 'Motlagh', 'Moumanil',
'Mullaney', 'Murray', 'Navathe', 'Ndzi', 'Newman', 'Ngodup',
'Nguyen', 'Nowlan', 'O'hare', 'Oliver', 'Onque', 'Osturnka',
'Owad', 'Ozark', 'Panjwani', 'Patronick', 'Pearson', 'Pelech',
'Pelletier', 'Perry', 'Peters', 'Peterson', 'Petingill',
'Petrowsky', 'Pham', 'Pitt', 'Potts', 'Power', 'Punjabhi',
'Purinton', 'Quinn', 'Rachael', 'Rarrick', 'Rhoads', 'Riordan',
'Rivera', 'Roberson', 'Robertson', 'Robinson', 'Roby', 'Roehrich',
'Rogers', 'Roper', 'Rose', 'Rossetti', 'Roup', 'Ruiz', 'Saada',
'Saar-Beckles', 'Sadki', 'Sahoo', 'Salter', 'Sander', 'Semizoglou',
'Sewkumar', 'Shepard', 'Shields', 'Simard', 'Singh', 'Sloan',
'Smith', 'Soto', 'South', 'Sparks', 'Spirea', 'Squatrito',
'Stanford', 'Stanley', 'Steans', 'Stoica', 'Strong', 'Sullivan',
'Sutwell', 'Sweetwater', 'Szabo', 'Tavares', 'Tejeda', 'Veera',
'Von Massenbach', 'Wallace', 'Wang', 'Zhou', 'Zima'], dtype=object)



len(df["Employee_Name"].unique())

输出

231

.nunique() 方法

此方法返回一系列中唯一值的数量。此方法默认使用参数 dropna = True 排除缺失值。

您可以将 False 参数传递给 dropna 参数以不删除缺失值。

df["Employee_Name"].nunique()

输出

231



df["Employee_Name"].nunique(dropna=False)

输出

231

相关文章

有用资源

打印 下一节:如何使用正则表达式和数据类型选择多个 DataFrame 列 ❯❮ 上一节:用 Python 编写程序以正确的顺序查找机场?