Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact John Barnett Scholarly Communications Librarian Jennifer Chan Assistant Scholarly Communications Librarian Office of Scholarly Communication and Publishing University Library System University of Pittsburgh Here Comes the Sunburst: Measuring and Visualizing Scholarly Impact University of Pittsburgh Pittsburgh campus + regional campuses in Bradford, Greensburg, Johnstown, and Titusville 16 undergraduate, graduate, and professional schools 456+ degree programs 2012: conferred 8,949 degrees University of Pittsburgh Top 10 American higher ed. in federal funding (NSF) Top 5 in annual research support (NIH) 5,369 faculty; 4,470 full-time faculty Research conducted: more than 300 centers, institutes, laboratories, clinics University Library System ARL 22nd largest academic library system in North America 25 libraries; 6.6 million volumes 279,000 current serials Sum of the Parts Office of Scholarly Communication and Publishing ULS Department of Information Technology Liaison Librarians Why Pitt? Strategic goal: Innovation in scholarly communication Providing services that scholars understand, need, and value Putting ourselves in faculty “spaces” Re-envisioning our librarian liaison program Deepening our understanding of scholarly communications issues Why PlumX? Making research “more assessable and accessible” – Gathering information in one place – Making it intelligible and useful Measuring and visualizing research impact Correlating metrics from traditional and new forms of scholarly communication Allowing researchers, labs, departments, institutions to track real-time scholarly impact Promoting research, comparing with peers, connecting with new research Altmetrics Project Timeline Spring 2012: Fall 2012 • First meeting with Plum Analytics • Gathered data from pilot participants Summer 2012: • Announcement of Pitt as Plum Analytics’ first partner Spring 2013 • Faculty surveyed; enhancements made Winter 2013 Fall 2013 • PlumX pilot system made public • IR widget launched; rollout preparations Pilot project aims Develop a tool for measuring and visualizing research impact Gathering information in one place Intelligible and useful Impact in social media and other scholarly communication methods Traditional measures counted as well See where to disseminate works to increase impact Traditional vs. new • Traditional measures are also counted • Findings are complementary to conventional methods of measuring research impact (e.g., H-Index) • Not intended to replace them New measures More comprehensive: Altmetrics = ALL METRICS – – – – – Citations Usage Captures Mentions Social Media Covers impact of online behavior – Because scholars increasingly work online Measures impact immediately – Because citation counts take years to appear in literature Pilot Process Created Altmetrics Task Force CV receipt and records creation Engaged Liaison Librarians Built Sharepoint Site Pilot Selection Pilot Project Participants • 32 researchers, various disciplines • 9 schools • 18 departments • 1 complete research group • Others joined as they learned about the project Pilot Project Participants discipline school/department Selected faculty participants, diversified by: online behavior level of career advancement Technologies Internal – – – – IR built on Eprints Platform Sharepoint Microsoft Office Suite PMID/DOI data import tool External – PlumX – DOIs – PMID Data collection for pilot project • Created records in [email protected], our institutional repository • Focused on articles, books, book chapters, proceedings • Scholarly output with standard identifiers • DOI, ISBN, PubMed ID, official URL, etc. • Scholarship produced since 2000 Other Library work • Developed guidelines to standardize record creation • Data entry from faculty CVs into IR (2 to 3 student workers with QA by librarians) • Librarian liaisons and other staff trained in record creation • SharePoint site used to track work completed • Coordination with pilot faculty • Gathered feedback and administered online survey Sharepoint Altmetrics Meetings Minutes Faculty CVs Excel spreadsheets Word docs External Data Sources Metadata sources Faculty CVs . . . But verify metadata! Books: PittCat, WorldCat, Books in Print, publisher sites, online retailers Journals: Serials Solutions list, journal websites, JournalSeek, UlrichsWeb, DOAJ, PubMed Conference presentations: Websites, PittCat, indexes, WorldCat PMID Import Tool Custom build by SysAdmin for Eprints Platform Utilizing PMIDs from PubMed, able to import records that prepopulate metadata fields – Item Type, Title, Abstract, Creators, Publication Title, ISSN, Volume/Issue, Page ranges, Date and Date type, DOI, MeSH Headings, Grant Information, Keywords, etc. Data Ingestion Full-text sources DOAJ ERIC PLOS SSRN* Other repositories* Federal government websites* Conference websites* * Use with caution Plum Analytics processing activities Harvest records from Pitt IR for each participant Build profile for each researcher in PlumX Harvest additional online artifacts NOT in Pitt IR Use data mining to harvest publically available metrics from hundreds of sites on the Web Create visualizations to display metrics on PlumX interface Key features Faculty profiles Online ‘artifacts’ – – – – – Article Book Book chapter Video Etc. Impact graph Sunburst Faculty profile Online ‘artifact’ display Impact graph Sunburst Feedback • Solicited via email and online survey • Generally positive in most cases • Data corrections • Errors in profiles • Links to wrong data • Quickly corrected by Plum staff • Requests for results from additional online sources (Google Scholar, SlideShare, Reddit, etc.) • PlumX collects data from these but did not gather information in advance for profiles The survey says Surveyed pilot project faculty in spring 2013 @ 1/3rd responded to the survey Meaning 13 out of 32 participants responded Accurate and useful data 90 76.92 80 70 60 Strongly disagree Disagree Agree Strongly agree 50 40 30 20 10 15.38 7.69 0 0 Summary profile data page The bar graph 70 58.33 60 50 40 30 25 20 16.67 10 0 0 Usefulness of interactive bar graph Strongly disagree Disagree Agree Strongly agree The sunburst 60 50 50 41.67 40 Strongly disagree Disagree Agree Strongly agree 30 20 8.33 10 0 0 Usefulness of sunburst Traditional & new measures 60 54.55 50 40 36.36 Strongly disagree Disagree Agree Strongly agree 30 20 9.09 10 0 0 Conveying traditional and new measures Usefulness of altmetrics 60 54.55 50 40 Strongly disagree Disagree Agree Strongly agree 30 20 18.18 18.18 9.09 10 0 The value of altmetrics Learning something new 40 36.36 35 30 27.27 27.27 25 Series 1 Series 2 Series 3 Series 4 20 15 10 9.09 5 0 Learning something new about my own research Comments Affiliations/bio inaccurate or has missing information “Mentions” by whom & when? Publications misclassified – Books vs. conference proceedings Data not collected – Google Scholar – Slideshare Comments Filter out unwanted information Data are wrong—and not useful Overabundance of information in sunburst “I only care what a select group of scholars thinks of my work” “I did not find this useful for my discipline” Observations Lacked information about faculty practices Are the results useful to all faculty, all disciplines? May appeal more to faculty who are early in their careers or whose work is more contemporary Will the data be used against faculty or programs? Labor-intensive strategy When it comes down to it . . . Does anyone care? Embeddable widgets (in development) For researchers, to add to: • their own Web pages • department directories • IR researcher profile page For individual artifacts, to build article level metrics for imbedding in: • IR document abstract page • Article abstract page for journals we publish Roll-out challenges Who creates profiles? Who edits? What information should be included in profiles? Who can view them? Separate data gathering from D-Scholarship deposits? Who promotes the service? Who trains? Timing . . . Future plans Data checking Additional data gathering Record merging/deduping Ability to edit user profiles and artifact records locally Open API To allow integration with other online systems More exhaustive scholarly practices survey for all faculty Rollout to all Pitt Researchers Will use automatic feed from Pitt IR to PlumX Discussion How would you “sell” PlumX to additional faculty?