Python机器学习小知识：Pandas去重

在使用Pandas处理数据的时候经常遇到重复数据的情况，这时候可以使用drop_duplicates（pandas.DataFrame.drop_duplicates）来进行处理。函数定义如下：
DataFrame.drop_duplicates(subset=None, keep='first', inplace=False, ignore_index=False)[source]
返回去重后的数据集（如果inplace为True则返回None）

参数：
subset：column label or sequence of labels, optional
Only consider certain columns for identifying duplicates, by default use all of the columns.
用来指定特定的列，默认所有列

keep：{‘first’, ‘last’, False}, default ‘first’
Determines which duplicates (if any) to keep. - first : Drop duplicates except for the first occurrence. - last : Drop duplicates except for the last occurrence. - False : Drop all duplicates.
确定保留哪条重复的数据，默认第一条

inplace：bool, default False
Whether to drop duplicates in place or to return a copy.
是直接在原来数据上修改还是保留一个副本

ignore_index：bool, default False
If True, the resulting axis will be labeled 0, 1, …, n - 1.
是否忽略Index，默认为True

例子：
df = pd.DataFrame({

'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],
'style': ['cup', 'cup', 'cup', 'pack', 'pack'],
'rating': [4, 4, 3.5, 15, 5]

})
df

brand style  rating

0 Yum Yum cup 4.0
1 Yum Yum cup 4.0
2 Indomie cup 3.5
3 Indomie pack 15.0
4 Indomie pack 5.0
By default, it removes duplicate rows based on all columns.

df.drop_duplicates()

brand style  rating

0 Yum Yum cup 4.0
2 Indomie cup 3.5
3 Indomie pack 15.0
4 Indomie pack 5.0
To remove duplicates on specific column(s), use subset.

具体可以参考说明：https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop_duplicates.html