1 Introduction

# Python
import pandas as pd
import re
covid_papers = pd.read_csv("data/covid_papers.csv")
covid_papers
_id pmid pmcid doi title source year_pub month_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 NaN 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 NaN 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira... MED 2020.0 NaN 2020-08-12 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N Y
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925 10.1016/j.ijsu.2020.07.032 A comparative overview of COVID-19, MERS and S... MED 2020.0 NaN 2020-07-31 <AUTHOR><FULLNAME>Liu J<FIRSTNAME>Jie<LASTNAME... {"volume": "81", "journalIssueId": 2986619, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Treatment", "SARS", "epidemiolog... 0 N Y
4 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... PPR 2020.0 NaN 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204 NaN BMT Settings, Infection and Infection Control MED 2018.0 NaN 2019-07-18 <AUTHOR><FULLNAME>Murray J<FIRSTNAME>John<LAST... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Isolation", "Fungal infection", ... 0 N Y
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335 NaN Early and Acute Complications and the Principl... MED 2018.0 NaN 2019-07-18 <AUTHOR><FULLNAME>Wallhult E<FIRSTNAME>Elisabe... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N Y
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections MED 2019.0 NaN 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... MED 2016.0 NaN 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting MED 2016.0 NaN 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y

148916 rows × 19 columns

covid_papers.sample(frac = 0.001).shape
## (149, 19)
  • Select columns
# Python

covid_papers.doi
## 0               10.1007/s12291-020-00919-0
## 1            10.1080/22221751.2020.1746199
## 2               10.1016/j.jiph.2020.07.011
## 3               10.1016/j.ijsu.2020.07.032
## 4         10.20944/preprints202008.0312.v1
##                         ...               
## 148911                                 NaN
## 148912                                 NaN
## 148913                                 NaN
## 148914                                 NaN
## 148915                                 NaN
## Name: doi, Length: 148916, dtype: str
covid_papers[['doi','title']]
doi title
0 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o...
1 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-...
2 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira...
3 10.1016/j.ijsu.2020.07.032 A comparative overview of COVID-19, MERS and S...
4 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ...
... ... ...
148911 NaN BMT Settings, Infection and Infection Control
148912 NaN Early and Acute Complications and the Principl...
148913 NaN Viral Infections
148914 NaN International Collaboration for Global Public ...
148915 NaN Resource Allocation and Priority Setting

148916 rows × 2 columns

covid_papers.drop(['doi','creation'],axis = 1)
_id pmid pmcid title source year_pub authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 The outbreak of the novel severe acute respira... MED 2020.0 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N Y
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925 A comparative overview of COVID-19, MERS and S... MED 2020.0 <AUTHOR><FULLNAME>Liu J<FIRSTNAME>Jie<LASTNAME... {"volume": "81", "journalIssueId": 2986619, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Treatment", "SARS", "epidemiolog... 0 N Y
4 5f69d6fa65a250f0edc4ee60 NaN NaN Virion Structure and Mechanism of Propagation ... PPR 2020.0 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204 BMT Settings, Infection and Infection Control MED 2018.0 <AUTHOR><FULLNAME>Murray J<FIRSTNAME>John<LAST... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Isolation", "Fungal infection", ... 0 N Y
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335 Early and Acute Complications and the Principl... MED 2018.0 <AUTHOR><FULLNAME>Wallhult E<FIRSTNAME>Elisabe... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N Y
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 Viral Infections MED 2019.0 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 International Collaboration for Global Public ... MED 2016.0 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 Resource Allocation and Priority Setting MED 2016.0 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y

148916 rows × 16 columns

  • using regular expression
# Python
import numpy as np
covid_papers = covid_papers.rename({ name:replace for name, replace in zip(list(covid_papers.columns[:4]),'V'+np.char.array(['1','2','3']))},axis = 1)
# filter columns using regex
covid_papers.filter(regex=("V[12]"))
V1 V2
0 5f69d6fa65a250f0edc4ee5c 32837036.0
1 5f69d6fa65a250f0edc4ee5d 32196410.0
2 5f69d6fa65a250f0edc4ee5e 32778421.0
3 5f69d6fa65a250f0edc4ee5f 32730205.0
4 5f69d6fa65a250f0edc4ee60 NaN
... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0
148912 5f69f29165a250f0edc7340c 31314312.0
148913 5f69f29165a250f0edc7340d 32091757.0
148914 5f69f29165a250f0edc7340e 28590697.0
148915 5f69f29165a250f0edc7340f 28590695.0

148916 rows × 2 columns

covid_papers.filter(regex=("V.*"))
V1 V2 V3
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925
4 5f69d6fa65a250f0edc4ee60 NaN NaN
... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709

148916 rows × 3 columns


covid_papers.filter(regex=("3$"))
V3
0 PMC7424135
1 PMC7170333
2 PMC7402212
3 PMC7382925
4 NaN
... ...
148911 PMC7121204
148912 PMC7123335
148913 PMC7124085
148914 PMC7193726
148915 PMC7193709

148916 rows × 1 columns


covid_papers.filter(regex=("^V"))
V1 V2 V3
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925
4 5f69d6fa65a250f0edc4ee60 NaN NaN
... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709

148916 rows × 3 columns

  • Filtering
# Python
covid_papers.loc[3:4,]
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925 10.1016/j.ijsu.2020.07.032 A comparative overview of COVID-19, MERS and S... MED 2020.0 2020-07-31 <AUTHOR><FULLNAME>Liu J<FIRSTNAME>Jie<LASTNAME... {"volume": "81", "journalIssueId": 2986619, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Treatment", "SARS", "epidemiolog... 0 N Y
4 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... PPR 2020.0 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N
covid_papers.drop(range(3,8))
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira... MED 2020.0 2020-08-12 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N Y
8 5f69d6fa65a250f0edc4ee64 32720031.0 PMC7384276 10.1007/s11255-020-02579-8 The kidney, COVID-19, and the chemokine networ... MED 2020.0 2020-07-29 <AUTHOR><FULLNAME>Taverna G<FIRSTNAME>Gianluig... {"journalIssueId": 2974652, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Kidney", "Chemokines", "Coronavi... 0 N Y
9 5f69d6fa65a250f0edc4ee65 32196430.0 PMC7172701 10.1080/22221751.2020.1745095 Laboratory diagnosis of emerging human coronav... MED 2020.0 2020-03-21 <AUTHOR><FULLNAME>Loeffelholz MJ<FIRSTNAME>Mic... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Review", "Jour... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Serology", "Real-Time PCR", "Poc... 53 Y Y
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204 NaN BMT Settings, Infection and Infection Control MED 2018.0 2019-07-18 <AUTHOR><FULLNAME>Murray J<FIRSTNAME>John<LAST... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Isolation", "Fungal infection", ... 0 N Y
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335 NaN Early and Acute Complications and the Principl... MED 2018.0 2019-07-18 <AUTHOR><FULLNAME>Wallhult E<FIRSTNAME>Elisabe... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N Y
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y

148911 rows × 18 columns


covid_papers[covid_papers.year_pub > 2018]
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira... MED 2020.0 2020-08-12 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N Y
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925 10.1016/j.ijsu.2020.07.032 A comparative overview of COVID-19, MERS and S... MED 2020.0 2020-07-31 <AUTHOR><FULLNAME>Liu J<FIRSTNAME>Jie<LASTNAME... {"volume": "81", "journalIssueId": 2986619, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Treatment", "SARS", "epidemiolog... 0 N Y
4 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... PPR 2020.0 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148905 5f69f29165a250f0edc73405 32091688.0 PMC7123178 NaN Management of Severe Malaria and Severe Dengue... MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Dondorp AM<FIRSTNAME>Arjen M... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148907 5f69f29165a250f0edc73407 32091836.0 PMC7123376 NaN Noninfectious Pulmonary Complications MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Carreras E<FIRSTNAME>Enric<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148909 5f69f29165a250f0edc73409 32091695.0 PMC7124120 NaN Infrastructure and Organization of Adult Inten... MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Papali A<FIRSTNAME>Alfred<LA... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148910 5f69f29165a250f0edc7340a 29262176.0 NaN NaN Acute Otitis Media MED 2019.0 2017-12-22 <AUTHOR><FULLNAME>Danishyar A<FIRSTNAME>Amina<... NaN eng NaN {"pubType": ["Book", "Review"]} NaN NaN 0 N Y
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y

89471 rows × 18 columns

covid_papers[covid_papers.title.isin(["Deep convolutional approaches for the analysis of Covid-19 using chest X-Ray images from portable devices",
                               "Evaluating the effect of city lock-down on controlling COVID-19 propagation through deep learning and network science models."])]
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
18790 5f69da2b65a250f0edc537c2 NaN NaN 10.1101/2020.06.18.20134593 Deep convolutional approaches for the analysis... PPR 2020.0 2020-06-24 <AUTHOR><FULLNAME>de Moura J<FIRSTNAME>Joaquim... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N Y
52612 5f69e03265a250f0edc5bbe0 32834328.0 PMC7402371 10.1016/j.cities.2020.102869 Evaluating the effect of city lock-down on con... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Zhang X<FIRSTNAME>Xiaoqi<LAS... {"volume": "107", "journalIssueId": 2999794, "... eng "Print-Electronic" {"pubType": ["research-article", "Journal Arti... NaN {"keyword": ["China", "Network Science", "Deep... 0 N Y

covid_papers[covid_papers.year_pub == 2018 & covid_papers.title.isin(["Deep convolutional approaches for the analysis of Covid-19 using chest X-Ray images from portable devices",
                               "Evaluating the effect of city lock-down on controlling COVID-19 propagation through deep learning and network science models."])]
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
                               
covid_papers = covid_papers.dropna(subset = ['title'])
covid_papers[covid_papers.title.str.contains('deep learning')][:5]
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
5203 5f69d7d965a250f0edc502af NaN NaN 10.1101/2020.01.21.914044 Host and infectivity prediction of Wuhan 2019 ... PPR 2020.0 2020-01-26 <AUTHOR><FULLNAME>Guo Q<FIRSTNAME>Qian<LASTNAM... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 3 N N
5527 5f69d7e765a250f0edc503f3 32837918.0 PMC7309744 10.1016/j.matpr.2020.06.245 Review on machine and deep learning models for... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Waleed Salehi A<FIRSTNAME>Ah... {"journalIssueId": 3000379, "dateOfPublication... eng "Print-Electronic" {"pubType": ["research-article", "Journal Arti... NaN {"keyword": ["Virus", "SVM", "Machine Learning... 0 N Y
6489 5f69d81165a250f0edc507b5 32280433.0 PMC7118541 10.1016/j.csbj.2020.03.025 Predicting commercially available antiviral dr... MED 2020.0 2020-04-14 <AUTHOR><FULLNAME>Beck BR<FIRSTNAME>Bo Ram<LAS... {"volume": "18", "journalIssueId": 2895444, "d... eng "Electronic-eCollection" {"pubType": ["brief-report", "Journal Article"]} NaN {"keyword": ["Coronavirus", "Drug Repurposing"... 19 Y Y
8336 5f69d86365a250f0edc50eec 32904395.0 PMC7453871 10.1007/s00500-020-05275-y Early diagnosis of COVID-19-affected patients ... MED 2020.0 2020-09-09 <AUTHOR><FULLNAME>Dansana D<FIRSTNAME>Debabrat... {"journalIssueId": 3009087, "dateOfPublication... eng "Print-Electronic" {"pubType": ["research-article", "Journal Arti... NaN {"keyword": ["Decision tree", "CT scan", "Cnn"... 0 N Y
10312 5f69d8bb65a250f0edc516a4 NaN NaN 10.1101/2020.02.14.20023028 A deep learning algorithm using CT images to s... PPR 2020.0 2020-02-20 <AUTHOR><FULLNAME>Wang S<FIRSTNAME>Shuai<LASTN... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 1 N N

covid_papers.drop_duplicates()
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira... MED 2020.0 2020-08-12 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N Y
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925 10.1016/j.ijsu.2020.07.032 A comparative overview of COVID-19, MERS and S... MED 2020.0 2020-07-31 <AUTHOR><FULLNAME>Liu J<FIRSTNAME>Jie<LASTNAME... {"volume": "81", "journalIssueId": 2986619, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Treatment", "SARS", "epidemiolog... 0 N Y
4 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... PPR 2020.0 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204 NaN BMT Settings, Infection and Infection Control MED 2018.0 2019-07-18 <AUTHOR><FULLNAME>Murray J<FIRSTNAME>John<LAST... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Isolation", "Fungal infection", ... 0 N Y
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335 NaN Early and Acute Complications and the Principl... MED 2018.0 2019-07-18 <AUTHOR><FULLNAME>Wallhult E<FIRSTNAME>Elisabe... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N Y
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y

148872 rows × 18 columns

covid_papers.drop_duplicates(subset=['V1', 'title'])
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira... MED 2020.0 2020-08-12 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N Y
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925 10.1016/j.ijsu.2020.07.032 A comparative overview of COVID-19, MERS and S... MED 2020.0 2020-07-31 <AUTHOR><FULLNAME>Liu J<FIRSTNAME>Jie<LASTNAME... {"volume": "81", "journalIssueId": 2986619, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Treatment", "SARS", "epidemiolog... 0 N Y
4 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... PPR 2020.0 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204 NaN BMT Settings, Infection and Infection Control MED 2018.0 2019-07-18 <AUTHOR><FULLNAME>Murray J<FIRSTNAME>John<LAST... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Isolation", "Fungal infection", ... 0 N Y
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335 NaN Early and Acute Complications and the Principl... MED 2018.0 2019-07-18 <AUTHOR><FULLNAME>Wallhult E<FIRSTNAME>Elisabe... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N Y
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y

148872 rows × 18 columns

  • Sorting
# Python
covid_papers.sort_values('year_pub')
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
148865 5f69f28f65a250f0edc733dd NaN PMC5613494 NaN The Essays, Cases, Intelligence, &c.: H—R. PMC 1820.0 2018-12-05 NaN {"volume": "44", "journalIssueId": 2754372, "d... eng "Undetermined" {"pubType": ["Index"]} NaN NaN 0 N Y
148799 5f69f28c65a250f0edc7339b NaN PMC5639501 NaN Syphilis, Pseudo-Syphilis, Mercury. PMC 1821.0 2018-06-23 NaN {"issue": "7", "volume": "2", "journalIssueId"... eng "Undetermined" {"pubType": ["review-article", "Review", "Jour... NaN NaN 0 N Y
148793 5f69f28b65a250f0edc73395 29919123.0 PMC5100527 NaN Observations on the Pathology of Venereal Affe... MED 1830.0 2018-06-20 NaN {"issue": "25", "volume": "13", "journalIssueI... eng "Print" {"pubType": ["other", "Journal Article"]} NaN NaN 0 N Y
148888 5f69f29065a250f0edc733f4 29917937.0 PMC5082205 NaN Periscope. MED 1832.0 2018-06-20 NaN {"issue": "31", "volume": "16", "journalIssueI... eng "Print" {"pubType": ["other", "Journal Article"]} NaN NaN 0 N Y
148867 5f69f28f65a250f0edc733df 29918028.0 PMC5086967 NaN Part III. Clinical Review and Hospital Reports. MED 1833.0 2018-06-20 NaN {"issue": "38", "volume": "19", "journalIssueI... eng "Print" {"pubType": ["other", "Journal Article"]} NaN NaN 0 N Y
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
138520 5f69f09565a250f0edc70b74 NaN PMC7120167 NaN Reconfiguring Species for Immunitary Hybridity. PMC NaN 2020-04-11 <AUTHOR><FULLNAME>Brown N<FIRSTNAME>Nik<LASTNA... {"journalIssueId": 2937077, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN NaN 0 N Y
139464 5f69f0c265a250f0edc70f24 NaN PMC7123719 NaN Metaphernreflexion und -analyse in sozialwisse... PMC NaN 2020-04-12 <AUTHOR><FULLNAME>Schmitt R<FIRSTNAME>Rudolf<L... {"journalIssueId": 2937822, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN NaN 0 N Y
142783 5f69f16265a250f0edc71c1b NaN PMC7121730 NaN Human Rights Issues. PMC NaN 2020-04-11 <AUTHOR><FULLNAME>Wei Y<FIRSTNAME>Yuwa<LASTNAM... {"journalIssueId": 2937244, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN NaN 0 N Y
145952 5f69f1fb65a250f0edc7287c NaN PMC7153414 NaN Entertainment Distribution Decisions. PMC NaN 2020-04-16 <AUTHOR><FULLNAME>Hennig-Thurau T<FIRSTNAME>Th... {"journalIssueId": 2940843, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN NaN 0 N Y
147551 5f69f24a65a250f0edc72ebb NaN PMC7123278 NaN Aviation and Intervention. PMC NaN 2020-04-12 <AUTHOR><FULLNAME>Abeyratne R<FIRSTNAME>Ruwant... {"journalIssueId": 2937709, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN {"keyword": ["Aircraft Noise", "Civil Aviation... 0 N Y

148872 rows × 18 columns

covid_papers.sort_values('year_pub',ascending=False)
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
118345 5f69ecb165a250f0edc6bca5 32835130.0 PMC7332907 10.1016/j.coemr.2020.06.005 Metabolic and inflammatory functions of short-... MED 2021.0 2020-08-25 <AUTHOR><FULLNAME>Bolognini D<FIRSTNAME>Daniel... {"volume": "16", "journalIssueId": 2999932, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Pancreas", "Immune cells", "Shor... 0 N Y
92675 5f69e7a965a250f0edc6585f NaN PMC7258713 NaN Musculoskeletal and Neurologic Diseases. PMC 2021.0 2020-05-31 <AUTHOR><FULLNAME>Huynh M<FIRSTNAME>Minh<LASTN... {"journalIssueId": 2960249, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN {"keyword": ["Canine distemper", "Rabies", "Se... 0 N Y
125910 5f69ee2465a250f0edc6da32 32904425.0 PMC7455151 10.1016/j.tbs.2020.08.006 Taking the bus? Barriers and facilitators for ... MED 2021.0 2020-09-09 <AUTHOR><FULLNAME>Mindell JS<FIRSTNAME>Jennife... {"volume": "22", "journalIssueId": 3000475, "d... eng "Print-Electronic" {"pubType": ["research-article", "Journal Arti... NaN {"keyword": ["New Zealand", "Adolescents", "Se... 0 N Y
50368 5f69dfcb65a250f0edc5b31c 32834290.0 PMC7396951 10.1016/j.paid.2020.110308 Collective self-esteem and perceived stress am... MED 2021.0 2020-08-25 <AUTHOR><FULLNAME>Chen H<FIRSTNAME>Haopeng<LAS... {"volume": "168", "journalIssueId": 2999784, "... eng "Print-Electronic" {"pubType": ["research-article", "Journal Arti... NaN {"keyword": ["Anxiety", "Perceived Stress", "C... 0 N Y
16091 5f69d9b565a250f0edc52d37 32921917.0 NaN 10.1016/j.resconrec.2020.105156 Concerns and strategies for wastewater treatme... MED 2021.0 2020-09-14 <AUTHOR><FULLNAME>Kataki S<FIRSTNAME>Sampriti<... {"volume": "164", "journalIssueId": 2999841, "... eng "Print-Electronic" {"pubType": ["Review", "Journal Article"]} NaN {"keyword": ["Wastewater", "Virus", "Disinfect... 0 N N
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
138520 5f69f09565a250f0edc70b74 NaN PMC7120167 NaN Reconfiguring Species for Immunitary Hybridity. PMC NaN 2020-04-11 <AUTHOR><FULLNAME>Brown N<FIRSTNAME>Nik<LASTNA... {"journalIssueId": 2937077, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN NaN 0 N Y
139464 5f69f0c265a250f0edc70f24 NaN PMC7123719 NaN Metaphernreflexion und -analyse in sozialwisse... PMC NaN 2020-04-12 <AUTHOR><FULLNAME>Schmitt R<FIRSTNAME>Rudolf<L... {"journalIssueId": 2937822, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN NaN 0 N Y
142783 5f69f16265a250f0edc71c1b NaN PMC7121730 NaN Human Rights Issues. PMC NaN 2020-04-11 <AUTHOR><FULLNAME>Wei Y<FIRSTNAME>Yuwa<LASTNAM... {"journalIssueId": 2937244, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN NaN 0 N Y
145952 5f69f1fb65a250f0edc7287c NaN PMC7153414 NaN Entertainment Distribution Decisions. PMC NaN 2020-04-16 <AUTHOR><FULLNAME>Hennig-Thurau T<FIRSTNAME>Th... {"journalIssueId": 2940843, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN NaN 0 N Y
147551 5f69f24a65a250f0edc72ebb NaN PMC7123278 NaN Aviation and Intervention. PMC NaN 2020-04-12 <AUTHOR><FULLNAME>Abeyratne R<FIRSTNAME>Ruwant... {"journalIssueId": 2937709, "dateOfPublication... eng "Undetermined" {"pubType": ["chapter-article", "Journal Artic... NaN {"keyword": ["Aircraft Noise", "Civil Aviation... 0 N Y

148872 rows × 18 columns

  • Summarise
# Python

covid_papers.agg({'cited':'sum'})
## cited    1241806
## dtype: int64

covid_papers.agg({'cited':['sum','std']})
cited
sum 1.241806e+06
std 3.883440e+01
  • Modifying
# Python
covid_papers.assign(oa = covid_papers.oa.map(lambda oa: 1 if  oa=='Y' else 0),
 log_cited = np.log(covid_papers.cited+1))
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa log_cited
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N 1 0.000000
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N 1 4.127134
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira... MED 2020.0 2020-08-12 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N 1 0.000000
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925 10.1016/j.ijsu.2020.07.032 A comparative overview of COVID-19, MERS and S... MED 2020.0 2020-07-31 <AUTHOR><FULLNAME>Liu J<FIRSTNAME>Jie<LASTNAME... {"volume": "81", "journalIssueId": 2986619, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Treatment", "SARS", "epidemiolog... 0 N 1 0.000000
4 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... PPR 2020.0 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N 0 0.000000
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204 NaN BMT Settings, Infection and Infection Control MED 2018.0 2019-07-18 <AUTHOR><FULLNAME>Murray J<FIRSTNAME>John<LAST... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Isolation", "Fungal infection", ... 0 N 1 0.000000
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335 NaN Early and Acute Complications and the Principl... MED 2018.0 2019-07-18 <AUTHOR><FULLNAME>Wallhult E<FIRSTNAME>Elisabe... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N 1 0.000000
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N 1 0.000000
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N 1 0.000000
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N 1 0.000000

148872 rows × 19 columns


np.where( covid_papers.oa=='Y',1,0) 
## array([1, 1, 1, ..., 1, 1, 1], shape=(148872,))

covid_papers.title.map(lambda x: str(x).lower())
## 0         the pathophysiology, diagnosis and treatment o...
## 1         hypothesis for potential pathogenesis of sars-...
## 2         the outbreak of the novel severe acute respira...
## 3         a comparative overview of covid-19, mers and s...
## 4         virion structure and mechanism of propagation ...
##                                 ...                        
## 148911        bmt settings, infection and infection control
## 148912    early and acute complications and the principl...
## 148913                                     viral infections
## 148914    international collaboration for global public ...
## 148915             resource allocation and priority setting
## Name: title, Length: 148872, dtype: str
  • Grouping
# Python

covid_papers.groupby('year_pub').language.nunique()
## year_pub
## 1820.0     1
## 1821.0     1
## 1830.0     1
## 1832.0     1
## 1833.0     1
##           ..
## 2017.0    12
## 2018.0     8
## 2019.0    10
## 2020.0    21
## 2021.0     2
## Name: language, Length: 137, dtype: int64
# By several groups

covid_papers.groupby('year_pub').agg({'cited':'sum'})
cited
year_pub
1820.0 0
1821.0 0
1830.0 0
1832.0 0
1833.0 0
... ...
2017.0 24850
2018.0 17139
2019.0 10151
2020.0 147528
2021.0 2

137 rows × 1 columns



# first obs for each group 

covid_papers.groupby('source').first()
V1 V2 V3 doi title year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
source
AGR 5f69dfd965a250f0edc5b44c NaN NaN 10.1016/j.fm.2011.12.009 The important role of polysaccharides from a t... 2020.0 2020-08-30 <AUTHOR><FULLNAME>Cao P<FIRSTNAME>Peng<LASTNAM... {"volume": "240", "journalIssueId": 2994915, "... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 0 N N
CBA 5f69e54965a250f0edc626c8 NaN NaN 10.1080/10020070312331344530 Construction of a DNA vaccine encoding the rec... 2006.0 2010-11-26 <AUTHOR><FULLNAME>Xie Li<LASTNAME>Xie Li <AFFI... {"issue": "8", "volume": "26", "journalIssueId... chi NaN {"pubType": ["Research article"]} NaN {"keyword": ["DNA vaccine", "Spike protein", "... 0 N N
CTX 5f69ebc365a250f0edc6a9dc NaN NaN NaN Combating the threat of zoonotic infections. 2008.0 2010-12-03 <AUTHOR><FULLNAME>Krijnse LockeR J<LASTNAME>Kr... {"issue": "s1", "volume": "58", "journalIssueI... eng NaN NaN {"meshHeading": [{"majorTopic_YN": "N", "descr... NaN 0 N N
ETH 5f69e80565a250f0edc65f8e NaN NaN NaN Bat as the animal origin of SARS-CoV and reser... 2009.0 2011-02-03 <AUTHOR><FULLNAME>Li S<LASTNAME>Li <AFFILIATIO... NaN eng NaN {"pubType": ["Dissertation"]} NaN {"keyword": ["Genetics", "Microbiology", "Vete... 0 N N
HIR 5f69e5dd65a250f0edc63360 NaN NaN NaN Hospital management of adults with Severe Acut... 2004.0 2008-12-17 NaN NaN eng NaN {"pubType": ["Practice Guideline"]} NaN {"keyword": ["Infection", "Viruses", "Safety",... 0 N N
MED 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y
PAT 5f69e34f65a250f0edc5fdbb NaN NaN NaN TRANSGENIC MOUSE LINES EXPRESSING HUMAN ACE2 A... 2008.0 2007-01-11 <AUTHOR><FULLNAME>CHAN TEH-SHENG<LASTNAME>CHAN... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N
PMC 5f69d6fa65a250f0edc4ee61 NaN PMC7480804 10.1016/s1473-3099(19)30181-1 Interaction of SARS-CoV-2 and Other Coronaviru... 2020.0 2020-09-13 <AUTHOR><FULLNAME>Davidson A<FIRSTNAME>Anne M.... {"journalIssueId": 2991812, "dateOfPublication... eng "Undetermined" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["China", "Serine", "Coronavirus",... 0 N Y
PPR 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... 2020.0 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN {"keyword": ["Neutralizing Antibodies", "Longi... 0 N N

covid_papers.groupby('source').last()
V1 V2 V3 doi title year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
source
AGR 5f69f16f65a250f0edc71d24 NaN NaN 10.4315/0362-028x-50.10.862 The organization and expression of coronavirus... 1987.0 1988-06-24 <AUTHOR><FULLNAME>Siddell S<LASTNAME>Siddell <... {"issue": "32", "journalIssueId": 1410212, "da... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 0 N N
CBA 5f69f0cf65a250f0edc7103d NaN NaN 10.1080/10020070312331344530 Isolation and purification of the virus of cor... 1994.0 2006-01-13 <AUTHOR><FULLNAME>Lu Ping<LASTNAME>Lu Ping <AF... {"issue": "1", "volume": "12", "journalIssueId... chi NaN {"pubType": ["Research article"]} NaN {"keyword": ["DNA sequencing", "Mutation", "Cl... 0 N N
CTX 5f69f00f65a250f0edc700b7 NaN NaN NaN Crystal structure of a coronavirus main protei... 2002.0 2013-12-09 <AUTHOR><FULLNAME>Anand K<LASTNAME>Anand <AFFI... {"issue": "s1", "volume": "58", "journalIssueI... eng NaN NaN {"meshHeading": [{"majorTopic_YN": "N", "descr... NaN 0 N N
ETH 5f69ee9365a250f0edc6e2d7 NaN NaN NaN IBV D-RNAs as delivery vectors for heterologou... 2002.0 2010-06-18 <AUTHOR><FULLNAME>Hackney K<LASTNAME>Hackney <... NaN eng NaN {"pubType": ["Dissertation"]} NaN {"keyword": ["Genetics", "Biochemistry", "Muta... 0 N N
HIR 5f69e5dd65a250f0edc63360 NaN NaN NaN Hospital management of adults with Severe Acut... 2004.0 2008-12-17 NaN NaN eng NaN {"pubType": ["Practice Guideline"]} NaN {"keyword": ["Infection", "Viruses", "Safety",... 0 N N
MED 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 10.1007/s13225-011-0088-y Resource Allocation and Priority Setting 2016.0 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... {"issue": "31", "volume": "16", "journalIssueI... eng "Print" {"pubType": ["chapter-article", "Review"]} {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N Y
PAT 5f69f1e565a250f0edc726c3 NaN NaN NaN VACCINE FOR NEONATAL CALF DIARRHEA 1975.0 1973-10-12 <AUTHOR><FULLNAME>MEBUS CHARLES A<LASTNAME>MEB... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N
PMC 5f69f29065a250f0edc733f1 NaN PMC2297516 10.1890/1051-0761(2000)010[0115:snzfdw]2.0.co;2 Forty-Third Annual Meeting of the British Medi... 1875.0 2009-11-19 <AUTHOR><FULLNAME>Harcourt-Brown F<FIRSTNAME>F... {"issue": "765", "volume": "2", "journalIssueI... eng "Undetermined" {"pubType": ["meeting-report", "Meeting Report"]} NaN {"keyword": ["Wastewater", "Pathogen", "Treatm... 0 N N
PPR 5f69eaec65a250f0edc698de NaN NaN 10.7287/peerj.preprints.2351v1 The evidence of porcine hemagglutinating encep... 2016.0 2018-07-10 <AUTHOR><FULLNAME>Li Z<FIRSTNAME>Zi<LASTNAME>L... NaN NaN NaN {"pubType": ["Preprint"]} NaN {"keyword": ["Depression", "Anxiety", "Chronic... 0 N N

# Add a column with number of observations for each group

covid_papers['grp_count'] = covid_papers.groupby('source')['source'].transform('count')

1.1 More complex use of by

  • Select specific row by group
# Python

covid_papers.groupby('source').apply(lambda df: df.iloc[:3])
V1 V2 V3 doi title year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa grp_count
source
AGR 50672 5f69dfd965a250f0edc5b44c NaN NaN NaN The important role of polysaccharides from a t... 2020.0 2020-08-30 <AUTHOR><FULLNAME>Cao P<FIRSTNAME>Peng<LASTNAM... {"volume": "240", "journalIssueId": 2994915, "... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 0 N N 35
54418 5f69e08865a250f0edc5c2ee NaN NaN NaN The Middle East Respiratory Syndrome Coronavir... 2015.0 2016-05-10 <AUTHOR><FULLNAME>Józefiak A<FIRSTNAME>Agata<... {"issue": "4", "volume": "15", "journalIssueId... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 0 N N 35
81491 5f69e58d65a250f0edc62caf NaN NaN 10.1016/j.fm.2011.12.009 Stability of bovine coronavirus on lettuce sur... 2012.0 2012-09-23 <AUTHOR><FULLNAME>Mullis L<FIRSTNAME>Lisa<LAST... {"issue": "1", "volume": "30", "journalIssueId... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 2 N N 35
CBA 79980 5f69e54965a250f0edc626c8 NaN NaN NaN Construction of a DNA vaccine encoding the rec... NaN 2010-11-26 <AUTHOR><FULLNAME>Xie Li<LASTNAME>Xie Li <AFFI... NaN chi NaN {"pubType": ["Research article"]} NaN {"keyword": ["DNA vaccine", "Spike protein", "... 0 N N 282
81102 5f69e57b65a250f0edc62b2a NaN NaN NaN Serological relationships of the human coronav... 2006.0 2008-07-29 <AUTHOR><FULLNAME>Wang Yadi<LASTNAME>Wang Yadi... {"issue": "8", "volume": "26", "journalIssueId... chi NaN {"pubType": ["Research article"]} NaN {"keyword": ["Serology", "Corona virus", "Coro... 0 N N 282
81738 5f69e59965a250f0edc62da6 NaN NaN NaN Comparative Analysis and Evolution Studies on ... 2003.0 2006-04-20 <AUTHOR><FULLNAME>Ma Xiangru<LASTNAME>Ma Xiang... {"issue": "6", "volume": "22", "journalIssueId... chi NaN {"pubType": ["Research article"]} NaN {"keyword": ["Evolution", "Comparative genomic... 0 N N 282
CTX 113536 5f69ebc365a250f0edc6a9dc NaN NaN NaN Combating the threat of zoonotic infections. 2008.0 2010-12-03 NaN NaN eng NaN NaN {"meshHeading": [{"majorTopic_YN": "N", "descr... NaN 0 N N 3
114908 5f69ec0865a250f0edc6af38 NaN NaN NaN A cell-biological analysis of the corona virus... 1994.0 2012-07-19 <AUTHOR><FULLNAME>Krijnse LockeR J<LASTNAME>Kr... NaN eng NaN NaN NaN NaN 0 N N 3
135771 5f69f00f65a250f0edc700b7 NaN NaN NaN Crystal structure of a coronavirus main protei... 2002.0 2013-12-09 <AUTHOR><FULLNAME>Anand K<LASTNAME>Anand <AFFI... {"issue": "s1", "volume": "58", "journalIssueI... eng NaN NaN NaN NaN 0 N N 3
ETH 94514 5f69e80565a250f0edc65f8e NaN NaN NaN Bat as the animal origin of SARS-CoV and reser... 2009.0 2011-02-03 <AUTHOR><FULLNAME>Li S<LASTNAME>Li <AFFILIATIO... NaN eng NaN {"pubType": ["Dissertation"]} NaN NaN 0 N N 24
96909 5f69e87965a250f0edc668e9 NaN NaN NaN Functional study of the spike protein of sever... 2007.0 2011-02-03 <AUTHOR><FULLNAME>Du L<LASTNAME>Du <AFFILIATIO... NaN eng NaN {"pubType": ["Dissertation"]} NaN NaN 0 N N 24
98499 5f69e8c665a250f0edc66f1f NaN NaN NaN Cell biology and regulation of angiotensin-con... 2008.0 2010-07-07 <AUTHOR><FULLNAME>Rushworth CA<LASTNAME>Rushwo... NaN eng NaN {"pubType": ["Dissertation"]} NaN NaN 0 N N 24
HIR 83204 5f69e5dd65a250f0edc63360 NaN NaN NaN Hospital management of adults with Severe Acut... 2004.0 2008-12-17 NaN NaN eng NaN {"pubType": ["Practice Guideline"]} NaN {"keyword": ["Infection", "Viruses", "Safety",... 0 N N 1
MED 0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y 113919
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... 2020.0 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y 113919
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira... 2020.0 2020-08-12 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N Y 113919
PAT 69471 5f69e34f65a250f0edc5fdbb NaN NaN NaN TRANSGENIC MOUSE LINES EXPRESSING HUMAN ACE2 A... 2008.0 2007-01-11 <AUTHOR><FULLNAME>CHAN TEH-SHENG<LASTNAME>CHAN... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N 1373
69526 5f69e35265a250f0edc5fdf2 NaN NaN NaN Transgenic Mouse Lines Expressing Human Ace2 a... 2009.0 2007-01-11 <AUTHOR><FULLNAME>CHAN TEH-SHENG<LASTNAME>CHAN... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N 1373
80787 5f69e56d65a250f0edc629ef NaN NaN NaN Purified trimeric S protein as vaccine against... 2007.0 2006-06-27 <AUTHOR><FULLNAME>KAM YIU W<LASTNAME>KAM YIU W... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N 1373
PMC 5 5f69d6fa65a250f0edc4ee61 NaN PMC7480804 NaN Interaction of SARS-CoV-2 and Other Coronaviru... 2020.0 2020-09-13 <AUTHOR><FULLNAME>Davidson A<FIRSTNAME>Anne M.... {"journalIssueId": 2991812, "dateOfPublication... eng "Undetermined" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["China", "Serine", "Coronavirus",... 0 N Y 19428
12 5f69d6fa65a250f0edc4ee68 NaN PMC7457103 NaN Liver in the limelight in the corona (COVID-19... 2020.0 2020-09-10 NaN {"issue": "16", "volume": "8", "journalIssueId... eng "Undetermined" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Liver", "Coronavirus", "Gastroin... 0 N Y 19428
19 5f69d6fa65a250f0edc4ee6f NaN PMC7471763 NaN Defensin 5 for prevention of SARS-CoV-2 invasi... 2020.0 2020-09-06 NaN {"volume": "143", "journalIssueId": 3009534, "... eng "Undetermined" {"pubType": ["Letter"]} NaN {"keyword": ["ACE2", "Paneth Cell", "Defensin ... 0 N Y 19428
PPR 4 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... 2020.0 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N 13807
246 5f69d70465a250f0edc4ef52 NaN NaN 10.1101/2020.08.28.271601 Sequence analysis of Indian SARS-CoV-2 isolate... 2020.0 2020-08-30 <AUTHOR><FULLNAME>Dash P<FIRSTNAME>Pujarini<LA... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N 13807
329 5f69d70765a250f0edc4efa5 NaN NaN 10.1101/2020.08.17.253682 Transcriptomic profiling of human corona virus... 2020.0 2020-08-19 <AUTHOR><FULLNAME>Friedman N<FIRSTNAME>Nehemya... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N 13807

covid_papers.groupby('source').apply(lambda df: df.iloc[-5:])
V1 V2 V3 doi title year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa grp_count
source
AGR 139938 5f69f0d865a250f0edc710fe NaN NaN NaN Feline coronavirus in the multicat household. 1994.0 1994-08-18 <AUTHOR><FULLNAME>Addie DD<LASTNAME>Addie <AFF... {"issue": "3", "volume": "22", "journalIssueId... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 0 N N 35
140353 5f69f0ea65a250f0edc7129d NaN NaN NaN Feline coronavirus is not a major caused of ne... 1993.0 1994-01-07 <AUTHOR><FULLNAME>Addie DD<LASTNAME>Addie <AFF... {"issue": "5", "volume": "21", "journalIssueId... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 3 N N 35
140440 5f69f0f065a250f0edc712f4 NaN NaN NaN The feline coronavirus infections: feline infe... 1991.0 1991-05-10 <AUTHOR><FULLNAME>Grahn BH<LASTNAME>Grahn <AFF... {"issue": "4", "volume": "86", "journalIssueId... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 1 N N 35
142874 5f69f16665a250f0edc71c76 NaN NaN 10.4315/0362-028x-50.10.862 Antiviral substances in raw bovine milk active... 1987.0 1987-11-13 <AUTHOR><FULLNAME>Panon G<LASTNAME>Panon <AFFI... {"issue": "10", "volume": "50", "journalIssueI... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 1 N N 35
143048 5f69f16f65a250f0edc71d24 NaN NaN NaN The organization and expression of coronavirus... 1987.0 1988-06-24 <AUTHOR><FULLNAME>Siddell S<LASTNAME>Siddell <... {"issue": "32", "journalIssueId": 1410212, "da... eng "Undetermined" {"pubType": ["Journal Article"]} NaN NaN 0 N N 35
CBA 137119 5f69f05265a250f0edc705fb NaN NaN NaN Establishment and rudimentary application of r... 2001.0 2006-03-23 <AUTHOR><FULLNAME>Qiao Jun<LASTNAME>Qiao Jun <... {"issue": "1", "volume": "10", "journalIssueId... chi NaN NaN NaN {"keyword": ["RT-PCR ", "Canine distemper viru... 0 N N 282
137397 5f69f06065a250f0edc70711 NaN NaN NaN Indirect ELISA for detecting serum antibodies ... 1999.0 2006-01-13 <AUTHOR><FULLNAME>Fan Zhiqiang<LASTNAME>Fan Zh... {"issue": "2", "volume": "19", "journalIssueId... chi NaN NaN NaN NaN 0 N N 282
138229 5f69f08765a250f0edc70a51 NaN NaN NaN Genetic variation of mRNA1 ORF1 of avian infec... 2004.0 2006-04-20 <AUTHOR><FULLNAME>Liu Shengwang<LASTNAME>Liu S... {"issue": "4", "volume": "20", "journalIssueId... chi NaN {"pubType": ["Research article"]} NaN {"keyword": ["DNA sequencing", "Mutation", "Cl... 0 N N 282
139281 5f69f0b965a250f0edc70e6d NaN NaN NaN Application of electron microscopy to the diag... 1997.0 2006-01-13 <AUTHOR><FULLNAME>Chen Dehui<LASTNAME>Chen Deh... {"issue": "5", "volume": "16", "journalIssueId... chi NaN NaN NaN NaN 0 N N 282
139745 5f69f0cf65a250f0edc7103d NaN NaN NaN Isolation and purification of the virus of cor... 1994.0 2006-01-13 <AUTHOR><FULLNAME>Lu Ping<LASTNAME>Lu Ping <AF... {"issue": "1", "volume": "12", "journalIssueId... chi NaN NaN NaN NaN 0 N N 282
CTX 113536 5f69ebc365a250f0edc6a9dc NaN NaN NaN Combating the threat of zoonotic infections. 2008.0 2010-12-03 NaN NaN eng NaN NaN {"meshHeading": [{"majorTopic_YN": "N", "descr... NaN 0 N N 3
114908 5f69ec0865a250f0edc6af38 NaN NaN NaN A cell-biological analysis of the corona virus... 1994.0 2012-07-19 <AUTHOR><FULLNAME>Krijnse LockeR J<LASTNAME>Kr... NaN eng NaN NaN NaN NaN 0 N N 3
135771 5f69f00f65a250f0edc700b7 NaN NaN NaN Crystal structure of a coronavirus main protei... 2002.0 2013-12-09 <AUTHOR><FULLNAME>Anand K<LASTNAME>Anand <AFFI... {"issue": "s1", "volume": "58", "journalIssueI... eng NaN NaN NaN NaN 0 N N 3
ETH 125320 5f69ee0765a250f0edc6d7e4 NaN NaN NaN Interaction of the coronavirus nucleocapsid pr... 2004.0 2010-06-18 <AUTHOR><FULLNAME>Wurm T<LASTNAME>Wurm <AFFILI... NaN eng NaN {"pubType": ["Dissertation"]} NaN NaN 0 N N 24
125487 5f69ee0f65a250f0edc6d88b NaN NaN NaN Studies of the infections of wild bird populat... 2008.0 2011-04-14 <AUTHOR><FULLNAME>Hughes LA<LASTNAME>Hughes <A... NaN eng NaN {"pubType": ["Dissertation"]} NaN NaN 0 N N 24
125645 5f69ee1765a250f0edc6d929 NaN NaN NaN Delineation of trafficking signals in the coro... 2006.0 2010-04-12 <AUTHOR><FULLNAME>Reed ML<LASTNAME>Reed <AFFIL... NaN eng NaN {"pubType": ["Dissertation"]} NaN NaN 0 N N 24
127851 5f69ee8465a250f0edc6e1c7 NaN NaN NaN Acute severe asthma : viruses and eosinophilic... 1999.0 2010-07-21 <AUTHOR><FULLNAME>Chanarin N<LASTNAME>Chanarin... NaN eng NaN {"pubType": ["Dissertation"]} NaN {"keyword": ["Microbiology", "Medicine", "Pcr;... 0 N N 24
128123 5f69ee9365a250f0edc6e2d7 NaN NaN NaN IBV D-RNAs as delivery vectors for heterologou... 2002.0 2010-06-18 <AUTHOR><FULLNAME>Hackney K<LASTNAME>Hackney <... NaN eng NaN {"pubType": ["Dissertation"]} NaN {"keyword": ["Genetics", "Biochemistry", "Muta... 0 N N 24
HIR 83204 5f69e5dd65a250f0edc63360 NaN NaN NaN Hospital management of adults with Severe Acut... 2004.0 2008-12-17 NaN NaN eng NaN {"pubType": ["Practice Guideline"]} NaN {"keyword": ["Infection", "Viruses", "Safety",... 0 N N 1
MED 148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204 NaN BMT Settings, Infection and Infection Control 2018.0 2019-07-18 <AUTHOR><FULLNAME>Murray J<FIRSTNAME>John<LAST... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Isolation", "Fungal infection", ... 0 N Y 113919
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335 NaN Early and Acute Complications and the Principl... 2018.0 2019-07-18 <AUTHOR><FULLNAME>Wallhult E<FIRSTNAME>Elisabe... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N Y 113919
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections 2019.0 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y 113919
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... 2016.0 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y 113919
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting 2016.0 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y 113919
PAT 143307 5f69f17b65a250f0edc71e27 NaN NaN NaN Non-pathogenic viral agent for the prevention ... 1988.0 1986-07-09 NaN NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N 1373
145353 5f69f1dc65a250f0edc72625 NaN NaN NaN Vaccine for neonatal calf diarrhea 1975.0 1974-11-29 <AUTHOR><FULLNAME>MEBUS CHARLES A<LASTNAME>MEB... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N 1373
145378 5f69f1de65a250f0edc7263e NaN NaN NaN Vaccine for neonatal calf diarrhea 1975.0 1974-11-29 <AUTHOR><FULLNAME>MEBUS CHARLES A<LASTNAME>MEB... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N 1373
145407 5f69f1df65a250f0edc7265b NaN NaN NaN Vaccine for neonatal calf diarrhea 1975.0 1974-11-29 <AUTHOR><FULLNAME>MEBUS CHARLES A<LASTNAME>MEB... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N 1373
145511 5f69f1e565a250f0edc726c3 NaN NaN NaN VACCINE FOR NEONATAL CALF DIARRHEA 1975.0 1973-10-12 <AUTHOR><FULLNAME>MEBUS CHARLES A<LASTNAME>MEB... NaN eng NaN {"pubType": ["Patent"]} NaN NaN 0 N N 1373
PMC 148876 5f69f28f65a250f0edc733e8 NaN PMC534772 NaN BIOLOGICAL SCIENCES. 1984.0 2009-11-19 NaN {"issue": "Suppl", "volume": "81", "journalIss... eng "Undetermined" {"pubType": ["Index"]} NaN NaN 0 Y N 19428
148879 5f69f29065a250f0edc733eb NaN PMC2308293 NaN The British Medical Journal. 1884.0 2009-11-19 NaN {"issue": "1252", "volume": "2", "journalIssue... eng "Undetermined" {"pubType": ["research-article", "Journal Arti... NaN NaN 0 N N 19428
148880 5f69f29065a250f0edc733ec NaN PMC2322303 NaN SEVENTY-THIRD ANNUAL MEETING OF THE British Me... 1905.0 2009-11-19 NaN {"issue": "2337", "volume": "2", "journalIssue... eng "Undetermined" {"pubType": ["meeting-report", "Meeting Report"]} NaN NaN 0 N N 19428
148884 5f69f29065a250f0edc733f0 NaN PMC2332244 NaN SEVENTY-NINTH ANNUAL MEETING OF THE British Me... 1911.0 2009-11-19 NaN {"issue": "2654", "volume": "2", "journalIssue... eng "Undetermined" {"pubType": ["meeting-report", "Meeting Report"]} NaN NaN 0 N N 19428
148885 5f69f29065a250f0edc733f1 NaN PMC2297516 NaN Forty-Third Annual Meeting of the British Medi... 1875.0 2009-11-19 NaN {"issue": "765", "volume": "2", "journalIssueI... eng "Undetermined" {"pubType": ["meeting-report", "Meeting Report"]} NaN NaN 0 N N 19428
PPR 107435 5f69ea7565a250f0edc69207 NaN NaN 10.1101/090936 Lung epithelial cells have virus-specific and ... 2016.0 2018-07-10 <AUTHOR><FULLNAME>VanLeuven JT<FIRSTNAME>James... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N 13807
107490 5f69ea7865a250f0edc6923e NaN NaN 10.1101/102798 Metagenomic Sequencing Detects Respiratory Pat... 2017.0 2018-07-10 <AUTHOR><FULLNAME>Langelier C<LASTNAME>Langeli... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 1 N N 13807
107593 5f69ea7d65a250f0edc692a5 NaN NaN 10.1101/041558 Productive infection of field strains of avian... 2016.0 2018-07-10 <AUTHOR><FULLNAME>Zhang Y<FIRSTNAME>Yueting<LA... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N 13807
108500 5f69eac865a250f0edc69630 NaN NaN 10.1101/099069 Epidemiological and ecological modelling revea... 2017.0 2018-07-10 <AUTHOR><FULLNAME>Coughtrie AL<FIRSTNAME>Abiga... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N 13807
109186 5f69eaec65a250f0edc698de NaN NaN 10.7287/peerj.preprints.2351v1 The evidence of porcine hemagglutinating encep... 2016.0 2018-07-10 <AUTHOR><FULLNAME>Li Z<FIRSTNAME>Zi<LASTNAME>L... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N 13807
  • Group counter
# Python

covid_papers.groupby(['source','year_pub']).ngroups
## 301
  • Get row number of observation by group
# Python

covid_papers.groupby(['source','year_pub']).size()
## source  year_pub
## AGR     1987.0          2
##         1991.0          1
##         1993.0          1
##         1994.0          1
##         1995.0         10
##                     ...  
## PPR     2016.0          6
##         2017.0         15
##         2018.0         24
##         2019.0         34
##         2020.0      13726
## Length: 301, dtype: int64
  • Grouping sets (multiple by at once)
# Python


covid_papers.groupby(['source','year_pub']).agg({'cited':np.mean,'V1':'count'})
cited V1
source year_pub
AGR 1987.0 0.500000 2
1991.0 1.000000 1
1993.0 3.000000 1
1994.0 0.000000 1
1995.0 11.600000 10
... ... ... ...
PPR 2016.0 0.000000 6
2017.0 0.533333 15
2018.0 0.541667 24
2019.0 0.147059 34
2020.0 0.120866 13726

301 rows × 2 columns

1.2 More Complex manipulation

  • Using multiple Variables
# Python
# Summarise several columns

covid_papers.agg({var:func for var, func in zip(['cited','year_pub'],['mean']*2)})
## cited          8.341434
## year_pub    2015.348434
## dtype: float64

# Summarise several columns by group
covid_papers.groupby(['source','year_pub']).agg({var:func for var, func in zip(['cited','year_pub'],['mean']*2)})
cited year_pub
source year_pub
AGR 1987.0 0.500000 1987.0
1991.0 1.000000 1991.0
1993.0 3.000000 1993.0
1994.0 0.000000 1994.0
1995.0 11.600000 1995.0
... ... ... ...
PPR 2016.0 0.000000 2016.0
2017.0 0.533333 2017.0
2018.0 0.541667 2018.0
2019.0 0.147059 2019.0
2020.0 0.120866 2020.0

301 rows × 2 columns


## using patterns (regex)

cols = list(filter(lambda x: x.startswith("V"),covid_papers.columns.tolist()))


covid_papers.groupby(['source','year_pub']).agg({var:func for var, func in zip(cols,['count']*len(cols))})
V1 V2 V3
source year_pub
AGR 1987.0 2 0 0
1991.0 1 0 0
1993.0 1 0 0
1994.0 1 0 0
1995.0 10 0 0
... ... ... ... ...
PPR 2016.0 6 0 0
2017.0 15 0 0
2018.0 24 0 0
2019.0 34 0 0
2020.0 13726 0 0

301 rows × 3 columns


# Summarise with more than one function by group

covid_papers.groupby(['source']).agg({'year_pub': 'mean','cited': 'median'})
year_pub cited
source
AGR 2003.371429 2.0
CBA 2004.014870 0.0
CTX 2001.333333 0.0
ETH 2004.666667 0.0
HIR 2004.000000 0.0
MED 2015.179733 0.0
PAT 2005.511289 0.0
PMC 2013.610694 0.0
PPR 2019.988194 0.0
  • Modify Multiples Variables

def tobinary(x):
    if str(x) == 'Y':
        return 1
    else :
        return 0
        
# Modify several columns (dropping the others)

covid_papers.agg(dict.fromkeys(['oa','has_data'],tobinary))
## oa          0
## has_data    0
## dtype: int64

# Modify several columns (keeping the others)
for var in ['oa','has_data']:
    covid_papers = covid_papers.assign(var = tobinary)
  • More complex expression
# Python

grp_size = covid_papers.groupby('year_pub').size().to_frame()
grp_size.columns = ['share_en']

covid_papers.assign(is_en = np.where(covid_papers.language == 'eng',1,0),
oa = tobinary).groupby('year_pub').agg(share_en= ('is_en',sum)).div(grp_size).sort_values('share_en')
share_en
year_pub
1956.0 0.666667
1964.0 0.750000
1966.0 0.750000
1967.0 0.785714
2020.0 0.796331
... ...
1935.0 1.000000
1968.0 1.000000
1936.0 1.000000
1965.0 1.000000
1820.0 1.000000

137 rows × 1 columns

2 Reshape data

Imagine you have a variable with multiple categories that are store in one cell as the variables ‘pubtype’ in our dataset. You may want to create different columns for each of this variable. The opposit is also possible, one can have several columns with value that we want to transforme in two columns : one with the factor and the other with the values.

First let’s clean this variable and separate the string in each cells in order to have one row for each article’s pubtype.

# Python
covid_papers.pubtype.loc[1]
## '{"pubType": ["review-article", "Journal Article"]}'

def clean_pubtype(str_):
  return re.sub('[{]|[}]|pubType|: |[[]|[]]','',str_)


clean_pubtype(covid_papers.pubtype.loc[1])
## <string>:3: FutureWarning: Possible nested set at position 20
## '"""review-article", "Journal Article"'

covid_papers['pubtype'] = covid_papers.pubtype.map(lambda pt: clean_pubtype(str(pt)))
pubtype_cp = covid_papers.assign(pubtype = covid_papers.pubtype.str.split('", "')).explode('pubtype')
pubtype_cp['pubtype'] = pubtype_cp.pubtype.map(lambda pubtype: re.sub('"','',(pubtype)))
pubtype_cp
V1 V2 V3 doi title source year_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa grp_count var
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" review-article NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y 113919 0
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" Review NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y 113919 0
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" Journal Article NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y 113919 0
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" review-article {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y 113919 0
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" Journal Article {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y 113919 0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections MED 2019.0 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN Review NaN NaN 0 N Y 113919 0
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN chapter-article NaN NaN 0 N Y 113919 0
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN Review NaN NaN 0 N Y 113919 0
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN chapter-article NaN NaN 0 N Y 113919 0
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting MED 2016.0 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN Review NaN NaN 0 N Y 113919 0

308408 rows × 20 columns


pubtype_cp =  pubtype_cp[['V2','pubtype']].groupby(['V2','pubtype']).size().to_frame('n').reset_index()
pubtype_cp_i = pubtype_cp[['V2','pubtype']].groupby(['V2','pubtype']).size().to_frame('n')

pubtype_cp_i
n
V2 pubtype
2142.0 Journal Article 1
research-article 1
2145.0 Journal Article 1
research-article 1
12235.0 Journal Article 1
... ... ...
32931118.0 Letter 1
32931136.0 Letter 1
32931146.0 Journal Article 1
32931161.0 Journal Article 1
32931162.0 Journal Article 1

258374 rows × 1 columns

2.1 Cast data (from long to wide)

# Python
pubtype_cp_i.unstack().fillna(0)
n
pubtype AIM Adaptive Clinical Trial Address Addresses Autobiography Bibliography Biography Book Case Reports Classical Article Clinical Conference Clinical Study Clinical Trial Clinical Trial Protocol Clinical Trial, Phase I Clinical Trial, Phase II Clinical Trial, Phase III Clinical Trial, Phase IV Clinical Trial, Veterinary Comment Comparative Study Congress Congresses Consensus Development Conference Consensus Development Conference, NIH Controlled Clinical Trial Corrected and Republished Article D Dataset Directory Editorial English Abstract Equivalence Trial Evaluation Studies Evaluation Study Expression of Concern Festschrift Guideline H Historical Article ... Technical Report Twin Study Validation Studies Validation Study Video-Audio Media Webcast abstract addendum announcement article-commentary book-review brief-report calendar case-report chapter-article correction data-paper discussion editorial expression-of-concern filler in-brief index introduction letter meeting-report methods-article news obituary oration other product-review protocol rapid-communication reply report research-article retraction review-article systematic-review
V2
2142.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
2145.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
12235.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
30881.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
36849.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
32931118.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32931136.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32931146.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32931161.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32931162.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

113919 rows × 118 columns

casted_pubtype_cp = pubtype_cp.pivot(index="V2", columns="pubtype", values="n").fillna(0)

casted_pubtype_cp
pubtype AIM Adaptive Clinical Trial Address Addresses Autobiography Bibliography Biography Book Case Reports Classical Article Clinical Conference Clinical Study Clinical Trial Clinical Trial Protocol Clinical Trial, Phase I Clinical Trial, Phase II Clinical Trial, Phase III Clinical Trial, Phase IV Clinical Trial, Veterinary Comment Comparative Study Congress Congresses Consensus Development Conference Consensus Development Conference, NIH Controlled Clinical Trial Corrected and Republished Article D Dataset Directory Editorial English Abstract Equivalence Trial Evaluation Studies Evaluation Study Expression of Concern Festschrift Guideline H Historical Article ... Technical Report Twin Study Validation Studies Validation Study Video-Audio Media Webcast abstract addendum announcement article-commentary book-review brief-report calendar case-report chapter-article correction data-paper discussion editorial expression-of-concern filler in-brief index introduction letter meeting-report methods-article news obituary oration other product-review protocol rapid-communication reply report research-article retraction review-article systematic-review
V2
2142.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
2145.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
12235.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
30881.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
36849.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
32931118.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32931136.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32931146.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32931161.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
32931162.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

113919 rows × 118 columns

2.2 Melt data (from wide to long)

# Python

casted_pubtype_cp.stack()
## V2          pubtype                
## 2142.0      AIM                        0.0
##             Adaptive Clinical Trial    0.0
##             Address                    0.0
##             Addresses                  0.0
##             Autobiography              0.0
##                                       ... 
## 32931162.0  report                     0.0
##             research-article           0.0
##             retraction                 0.0
##             review-article             0.0
##             systematic-review          0.0
## Length: 13442442, dtype: float64


casted_pubtype_cp.stack().to_frame('n').reset_index()
V2 pubtype n
0 2142.0 AIM 0.0
1 2142.0 Adaptive Clinical Trial 0.0
2 2142.0 Address 0.0
3 2142.0 Addresses 0.0
4 2142.0 Autobiography 0.0
... ... ... ...
13442437 32931162.0 report 0.0
13442438 32931162.0 research-article 0.0
13442439 32931162.0 retraction 0.0
13442440 32931162.0 review-article 0.0
13442441 32931162.0 systematic-review 0.0

13442442 rows × 3 columns

2.3 Split

# Python

list(covid_papers.groupby('oa'))

2.4 Split and transpose a vector/column

# Python
covid_papers[['doi1','doi2']] = covid_papers.doi.str.split("/",expand=True).loc[:,:1]
covid_papers[['doi','doi1','doi2']]
doi doi1 doi2
0 10.1007/s12291-020-00919-0 10.1007 s12291-020-00919-0
1 10.1080/22221751.2020.1746199 10.1080 22221751.2020.1746199
2 10.1016/j.jiph.2020.07.011 10.1016 j.jiph.2020.07.011
3 10.1016/j.ijsu.2020.07.032 10.1016 j.ijsu.2020.07.032
4 10.20944/preprints202008.0312.v1 10.20944 preprints202008.0312.v1
... ... ... ...
148911 NaN NaN NaN
148912 NaN NaN NaN
148913 NaN NaN NaN
148914 NaN NaN NaN
148915 NaN NaN NaN

148872 rows × 3 columns


covid_papers.doi.str.split('/').explode()
## 0                       10.1007
## 0            s12291-020-00919-0
## 1                       10.1080
## 1         22221751.2020.1746199
## 2                       10.1016
##                   ...          
## 148911                      NaN
## 148912                      NaN
## 148913                      NaN
## 148914                      NaN
## 148915                      NaN
## Name: doi, Length: 279799, dtype: str

3 Index and Keys

  • df.loc are for labels/ names
  • df.iloc are for position numbers
# Python
covid_papers = covid_papers.set_index('V2')

covid_papers.loc[32837036,'doi']
## '10.1007/s12291-020-00919-0'

covid_papers.iloc[2,5]
## np.float64(2020.0)

4 Join/Bind data sets

4.1 Join

# Python

df1 = pd.DataFrame({'Id': ['A','B','C','C'],
                    'X1': [1,3,5,7],
                    'X2': ['x2','x4','x6','x8']}).set_index('Id')

df2 = pd.DataFrame({'Id': ['A','B','B','D'],
                    'Y1': [1,3,5,7],
                    'Y2': ['y1','y3','y5','y7']}).set_index('Id')
  • Join matching rows from y to x
# Python

df1.join(df2)
X1 X2 Y1 Y2
Id
A 1 x2 1.0 y1
B 3 x4 3.0 y3
B 3 x4 5.0 y5
C 5 x6 NaN NaN
C 7 x8 NaN NaN

pd.merge(df1, df2, how= 'left', on = 'Id')
X1 X2 Y1 Y2
Id
A 1 x2 1.0 y1
B 3 x4 3.0 y3
B 3 x4 5.0 y5
C 5 x6 NaN NaN
C 7 x8 NaN NaN
  • Join matching rows from x to y
# Python

df2.join(df1)
Y1 Y2 X1 X2
Id
A 1 y1 1.0 x2
B 3 y3 3.0 x4
B 5 y5 3.0 x4
D 7 y7 NaN NaN


pd.merge(df1, df2, how= 'right', on = 'Id')
X1 X2 Y1 Y2
Id
A 1.0 x2 1 y1
B 3.0 x4 3 y3
B 3.0 x4 5 y5
D NaN NaN 7 y7
  • Join matching rows from both x and y
# Python

pd.merge(df1, df2, how= 'inner', on = 'Id')
X1 X2 Y1 Y2
Id
A 1 x2 1 y1
B 3 x4 3 y3
B 3 x4 5 y5
  • Join keeping all the rows
# Python

pd.merge(df1, df2, how= 'outer', on = 'Id')
X1 X2 Y1 Y2
Id
A 1.0 x2 1.0 y1
B 3.0 x4 3.0 y3
B 3.0 x4 5.0 y5
C 5.0 x6 NaN NaN
C 7.0 x8 NaN NaN
D NaN NaN 7.0 y7
  • Return rows from x matching y
# Python

df1[df1.index == df2.index]
X1 X2
Id
A 1 x2
B 3 x4
  • Return rows from x not matching y
# Python
df1[df1.index != df2.index]
X1 X2
Id
C 5 x6
C 7 x8

4.2 Bind

  • Bind rows
# Python

df1 = pd.DataFrame({'Id': ['A','B','C'],
                    'X1': [1,3,5],
                    'X2': ['x2','x4','x6']}).set_index('Id')

df2 = pd.DataFrame({'Id': ['A','B','C'],
                    'Y1': [1,3,5],
                    'Y2': ['y1','y3','y5']}).set_index('Id')

pd.concat([df1,df2])
X1 X2 Y1 Y2
Id
A 1.0 x2 NaN NaN
B 3.0 x4 NaN NaN
C 5.0 x6 NaN NaN
A NaN NaN 1.0 y1
B NaN NaN 3.0 y3
C NaN NaN 5.0 y5
  • Bind columns
# Python

df3 = pd.concat([df1,df2],axis =1)
df3
X1 X2 Y1 Y2
Id
A 1 x2 1 y1
B 3 x4 3 y3
C 5 x6 5 y5

5 Other

  • Lead/Lag
# Python


df3.shift(periods = 1)
X1 X2 Y1 Y2
Id
A NaN NaN NaN NaN
B 1.0 x2 1.0 y1
C 3.0 x4 3.0 y3

df3.shift(periods = 1, axis= 1)
X1 X2 Y1 Y2
Id
A NaN 1 x2 1
B NaN 3 x4 3
C NaN 5 x6 5

6 Read/Write data

6.1 Read

  • read csv file with pandas
# Python

covid_papers = pd.read_csv("data/covid_papers.csv")

covid_papers
_id pmid pmcid doi title source year_pub month_pub creation authors_gender journal language pubmodel pubtype mesh keywords cited has_data oa
0 5f69d6fa65a250f0edc4ee5c 32837036.0 PMC7424135 10.1007/s12291-020-00919-0 The Pathophysiology, Diagnosis and Treatment o... MED 2020.0 NaN 2020-08-25 <AUTHOR><FULLNAME>Das SK<FIRSTNAME>Subir Kumar... {"journalIssueId": 2994713, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Angiotensin", "RNA polymerase", ... 0 N Y
1 5f69d6fa65a250f0edc4ee5d 32196410.0 PMC7170333 10.1080/22221751.2020.1746199 Hypothesis for potential pathogenesis of SARS-... MED 2020.0 NaN 2020-03-21 <AUTHOR><FULLNAME>Lin L<FIRSTNAME>Ling<LASTNAM... {"issue": "1", "volume": "9", "journalIssueId"... eng "Print" {"pubType": ["review-article", "Journal Articl... {"meshHeading": [{"majorTopic_YN": "N", "descr... {"keyword": ["Pathogenesis", "Anticoagulation"... 61 N Y
2 5f69d6fa65a250f0edc4ee5e 32778421.0 PMC7402212 10.1016/j.jiph.2020.07.011 The outbreak of the novel severe acute respira... MED 2020.0 NaN 2020-08-12 <AUTHOR><FULLNAME>Bchetnia M<FIRSTNAME>Mbarka<... {"journalIssueId": 2988080, "dateOfPublication... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Transmission", "Prevention", "Tr... 0 N Y
3 5f69d6fa65a250f0edc4ee5f 32730205.0 PMC7382925 10.1016/j.ijsu.2020.07.032 A comparative overview of COVID-19, MERS and S... MED 2020.0 NaN 2020-07-31 <AUTHOR><FULLNAME>Liu J<FIRSTNAME>Jie<LASTNAME... {"volume": "81", "journalIssueId": 2986619, "d... eng "Print-Electronic" {"pubType": ["review-article", "Review", "Jour... NaN {"keyword": ["Treatment", "SARS", "epidemiolog... 0 N Y
4 5f69d6fa65a250f0edc4ee60 NaN NaN 10.20944/preprints202008.0312.v1 Virion Structure and Mechanism of Propagation ... PPR 2020.0 NaN 2020-08-16 <AUTHOR><FULLNAME>Ghosh S<FIRSTNAME>Swapan<LAS... NaN NaN NaN {"pubType": ["Preprint"]} NaN NaN 0 N N
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
148911 5f69f29165a250f0edc7340b 31314321.0 PMC7121204 NaN BMT Settings, Infection and Infection Control MED 2018.0 NaN 2019-07-18 <AUTHOR><FULLNAME>Murray J<FIRSTNAME>John<LAST... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Isolation", "Fungal infection", ... 0 N Y
148912 5f69f29165a250f0edc7340c 31314312.0 PMC7123335 NaN Early and Acute Complications and the Principl... MED 2018.0 NaN 2019-07-18 <AUTHOR><FULLNAME>Wallhult E<FIRSTNAME>Elisabe... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN {"keyword": ["Sepsis", "Mucositis", "Haemorrha... 0 N Y
148913 5f69f29165a250f0edc7340d 32091757.0 PMC7124085 NaN Viral Infections MED 2019.0 NaN 2020-02-25 <AUTHOR><FULLNAME>Ljungman P<FIRSTNAME>Per<LAS... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148914 5f69f29165a250f0edc7340e 28590697.0 PMC7193726 NaN International Collaboration for Global Public ... MED 2016.0 NaN 2017-06-08 <AUTHOR><FULLNAME>Meslin EM<FIRSTNAME>Eric M.<... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y
148915 5f69f29165a250f0edc7340f 28590695.0 PMC7193709 NaN Resource Allocation and Priority Setting MED 2016.0 NaN 2017-06-08 <AUTHOR><FULLNAME>Daniels N<FIRSTNAME>Norman<L... NaN eng NaN {"pubType": ["chapter-article", "Review"]} NaN NaN 0 N Y

148916 rows × 19 columns

6.2 Write

  • write data to a csv file
# Python

covid_papers.to_csv("data/covid_papers.csv")

7 Exercise

Using first dplyr and the tidyverse, Use table_1.csv, table_2.csv and loc_cities.csv, we will need information from the three datasets.

  • Import datasets

  • Remove articles with no pmid and no DOI, and all articles before 1975

  • Merge the two datasets, pmid is unique for each paper

  • Create a new variable with the number of authors for each paper

  • plot distribution for the log(number of authors +1)

  • How many papers contains ‘deep learning’ or ‘machine learning’ and ‘neural network’ (also with a ‘s’ for neural networks) in their title ? Create a binary variable to save this information. What is the mean of authors for ML papers and non#ML papers ?

  • Transform has_data and oa into binary variable also, what is the share of ML paper that are oa

  • Clean up pubtype, for simplicity just get the first type

  • What is the pub type with the highest mean/sd of citation for each type of publication ? (use cited and the cleaned pub_type)

  • What are the most representative country by year ? In another tibble, keep only pmid and authors, and get the country for each author from the loc_cities.csv. You may want to separate rows for each authors to get all countries involved in the paper. If an author have multiple affiliations, take the first one.

  • Select the top 25 of countries involved in coronavirus research since 2001, plot the evolution on a bar chart with plot_ly

solution with pandas (python)