Tag Archives: Librarianship and Legal Technology
For those of us who are not engineers or programmers, magical results appear when we run searches in legal databases. However, we have little understanding of the machinations behind the ever-present e-wall. What kind of confidence can we have when the underlying structure of legal databases are hardwired with human biases? We must ask ourselves the question posed to then-Senator Obama and Senator McCain at a Town Hall Debate in 2008, “What don’t you know and how will you learn it?”
When I teach legal research, my students compare the same searches in different databases. One goal is to demonstrate that there are different results. But a more nuanced goal is to examine the results closely enough to provide insights into which databases might be more useful for updating, for case searching, for browsing statutes, and other research tasks. Susan Nevelow Mart’s study will elevate these discussions because of her focus on human-engineered algorithms and the inherent biases in the databases used for legal research. This study will also guide researchers to think more about search strategy and will help set more realistic expectations about search results.
Mart studied the impact of human judgment and bias at every step of the database search process. Her study explains how bias is hardwired into the human-engineered algorithm of each database. Add additional layers of human judgment and bias to the choice of database, to the date and time of the search, to the search terms, to the vendor’s classification scheme, and to the fact that searchers typically only browse the first 10 sometimes-relevant results. Mart introduces us to the concept of algorithmic accountability or “the term for disclosing prioritization, classification, association, and filtering.” Mart contends that algorithmic accountability, or understanding a bit more about the secret sauce in the inputs, will help researchers produce more accurate search results.
Mart’s research sought to test hypotheses about search algorithms by examining the results of the same searches in the same jurisdiction across six databases: Casetext, Fastcase, Google Scholar, Lexis Advance, Ravel, and Westlaw. When examining the relevance of the top 10 results, it is unsurprising that Lexis Advance and Westlaw lead in the relevancy rankings because they have the longest standing in the market. However, it is surprising that the top 10 results for those two vendors were relevant only 57% and 67% of the time, respectively.
Mart found that each of the six databases average 40% unique cases in the top 10 results. Mart also explores how many of the unique results are relevant in each database’s results. Again, it is unsurprising that Westlaw (at 33%) and Lexis Advance (at about 20%) lead in these two categories. It is surprising, however, that there are so many relevant cases that are unique results when the same search was performed in each database. And because we don’t know what is in the secret sauce, it is difficult to improve these outcomes.
There are a number of takeaways from Mart’s study. First, algorithmic variations lead to variations in the unique, and in the relevant, results returned from each database. Second, database vendors want us to have confidence in their products but it is still necessary to run the same search in more than one database to improve the chances of yielding the most comprehensive, relevant results. Third, while some of the newer legal databases yield less unique and less relevant results, they can bring advantages depending on the research topic, the time period, and other contextual details.
This well-researched and well-written article is required reading for every attorney who performs research on behalf of a client and for every professor who teaches legal research or uses legal databases. Because we often don’t know what we don’t know, Mart’s work pushes us to think more deeply about our search products and processes. Mart’s results provide an opportunity to narrow the gap in knowledge by learning a bit about what we don’t know. Learning from this scholarly yet accessible article brings the reader closer to understanding how to derive the optimal output even without knowing the ingredients in the secret sauce.
Cite as: Elizabeth Adelman, What Don’t You Know and How Will You Learn It?
(February 19, 2018) (reviewing Susan Nevelow Mart, The Algorithm as a Human Artifact: Implications for Legal [Re]Search
, 109 Law Libr. J.
387 (2017)), https://lex.jotwell.com/dont-know-will-learn/
Experiential learning is currently one of the buzz words of legal education. Recent changes to the ABA Standards and Rules of Procedure for Approval of Law Schools have focused greater attention on learning outcomes and assessment and increasing opportunities for learning and practicing skills that students will use as attorneys. In fact, ABA Standard 303(a)(3) requires a minimum of 6 credit hours of experiential course work.
Traditionally, experiential learning was widely thought to be the domain of law school clinics and externships, or field placements. However, the increased credit hour requirement for experiential learning has caused law schools to review their curriculum and determine whether sufficient experiential learning opportunities exist to meet the minimum requirement. Accordingly, there is a push to design new courses, or redesign existing courses, to meet a third type of experiential learning termed simulation courses, as described in ABA Standard 304. In order to qualify as a simulation course under the standard, a course should provide an experience “reasonably similar” to client representation although the student is not working with a real client.
Professor Alyson M. Drake’s article calls for the creation or retooling of stand-alone research classes that will meet the requirements to be designated as experiential classes. An increase in the number of research classes categorized as experiential will provide two benefits. First, and most importantly, it can serve to provide additional legal research instruction beyond the first year of law school. It will also support the mission of law schools to expand course offerings that meet the experiential standard.
Although classes in legal research have traditionally been sparse and under-valued in law schools, this approach does not match the necessity for proficiency in legal research needed as a practicing attorney, or the amount of an attorney’s time that is devoted to legal research. Professor Drake notes that newer attorneys spend approximately 35% of their time on legal research. More experienced attorneys spend approximately 18% of their time doing research. In contrast, Professor Drake notes that hiring attorneys often assess new attorney research skills as needing improvement. Providing more legal research instruction, in a format that closely mimics the work of a practicing attorney, will produce students who are better prepared to transition to practice.
Professor Drake’s article provides a very useful overview of the history of this shift in the ABA Standards and legal education in general. Similarly, the article also provides a breakdown of the Standards and how they can be interpreted in regards to experiential education and, in particular, simulation courses. This background and analysis of the standards is quite useful for those seeking to understand the context of the shift toward increased experiential learning as well as those seeking to create experiential learning opportunities in legal research courses.
Finally, Professor Drake discusses several current legal research teaching methods and how they might be retooled to satisfy the requirements of Standard 304. She breaks down the components of different pedagogical structures, highlighting which components are likely to already meet the requirements, which aspects might not, and recommending ways in which those components might be restructured to meet the standard.
Professor Drake begins with the “traditional legal research course,” described as “those where the lecture takes place during class time.” (P. 529.) Some classroom practice is usually included, but the balance between lecture and practicing skills can vary significantly. A traditional class of this sort may be the most challenging to convert to an experiential model. However, Professor Drake offers several changes to a traditional class that could support its meeting the ABA requirements for an experiential class. These include ensuring the balance between lecture and practice weighs most heavily toward practice, using assignments that closely approximate problems likely to occur in practice, and limiting class size so that the professor can provide “direct supervision.”
Professor Drake goes on to discuss key components of flipped, online, and specialized legal research courses and outlines several suggestions for how these courses can be retooled to meet the experiential requirements. Her article is a call to create experiential legal research courses, but it goes farther by providing a roadmap that can be used to design new classes or restructure existing classes. Legal research is a critical lawyering skill; however, it is also an area where hiring attorneys believe new attorney skills are lacking. This gap between the necessity of efficient and effective research and ability at law school graduation creates an opportunity for the growth of legal research courses in the curriculum.
Learning the substantive law has always been the foundation of a legal education. As job prospects for attorneys tightened, a focus on practitioner skills began trending in legal education. There is an expectation that law schools will produce practice-ready attorneys. Despite this expectation, why are Johnny and Jane unable to research?
Professor Caroline L. Osborne’s research findings have confirmed what many legal educators surmise about the state of legal research education. Her findings demonstrate that legal research education is undervalued in law schools. “For those involved in legal education, the goal is to provide students with the tools they need to succeed . . . . ” (P. 407.) In a carpenter’s arena, the value of the hammer is universally understood. The value of legal research as an essential tool of the legal trade, on the other hand, is not well understood in legal education. This lack of understanding persists, despite the MacCrate report and its ilk, codified ethical obligations of attorneys, and promulgated research competency standards. With this in mind, Professor Osborne presents each contributing factor to the devaluation of legal research education so that the reader is equipped to ponder solutions.
The MacCrate Report was published in 1992, and served as one of the first comprehensive assessments of necessary skills for attorneys and of the legal profession. It declares that legal research skills are fundamental: “It can hardly be doubted that the ability to do legal research is one of the skills that any competent legal practitioner must possess.” (P. 163.) Similarly, Best Practices for Legal Education acknowledges legal research as “essential.” (P. 58.)
The ethical obligations of attorneys further obligate Johnny and Jane. Model Rules of Professional Conduct, Rule 1.1, which has been adopted in whole or in part by all fifty states, holds the lawyer to a certain level of competence:
A lawyer shall provide competent representation to a client. Competent representation requires the legal knowledge, skill, thoroughness and preparation reasonably necessary for the representation.
Comment 2 to Model Rule 1.1 points out that competent representation requires a lawyer to possess “important legal skills” that are typical in all legal problems. Comment 8 further articulates the need to maintain competence, in part, by requiring technology competency, including those technology skills which are essential for performing research:
To maintain the requisite knowledge and skill, a lawyer should keep abreast of changes in the law and its practice, including the benefits and risks associated with relevant technology, . . .
Technology is a major factor in the devaluation of legal research education. It has enabled the creation of and easy access to massive amounts of information. It has also transformed how we perform research. While the ubiquity of information has made research appear easier, it has actually become a more sophisticated task. To further complicate the picture, Osborne points out that today’s students seem to lack the ability to evaluate information and to think critically.
The ethical standards outlined above are bolstered by Legal Research Competencies and Standards for Law Student Information Literacy, developed and approved by the American Association of Law Libraries in 2012. These standards were promulgated to inform best practices in legal education curricular design and to provide a baseline competency for attorneys in practice settings.
Notwithstanding the existence of these industry standards, Professor Osborne discusses contributing factors for the devaluation of legal research education in law school curricula. Her survey results show, for example, that only 16% of responding institutions have a stand-alone research classes as opposed to being integrated into the legal writing curriculum. She also points out that law school writing programs have been strengthened while the emphasis on research has lessened. Combined, these factors have had an unfortunate impact on legal research education. “[T]he cost of graduating fluent writers should not be the legal research curriculum.” (P. 404.)
Despite what the industry demands, Osborne opines that the current state of legal education—including pass/fail grading, fewer credits awarded compared with other 1L courses, integration with and diminished presence in the writing curriculum—sends the message to law students that legal research education is unimportant. Her thesis is backed up by the findings of a recent BARBRI State of the Legal Field survey: “Faculty placed very little importance on research, with just 4 percent citing it as the most important skill for recent law school graduates . . . .” In our role as legal educators, “we fail to signal the importance of legal research in the practice of law.” (P. 409.)
In this article, Professor Osborne conveys a very important message to legal educators: we should consider ourselves on notice that our actions and our curricula demonstrate a decreased importance of legal research education to our students. As a result, Johnny and Jane are leaving law school without this fundamental tool despite the expectation that they graduate with both the substantive knowledge and the skills to be practice ready and to maintain or exceed industry standards.
Brianna L. Schofield & Jennifer M. Urban, Berkeley Digital Library Copyright Project Report: Takedown and Today’s Academic Digital Library
, U.C. Berkeley Pub. L. Research Pape
r No. 2694731 (November 2015), available at SSRN
A recent push to provide increased access to research, scholarship, and archival materials, as well as a desire to provide greater visibility to faculty and institutional work, have driven more and more academic libraries to create online repositories. These repositories have successfully generated greater visibility for scholarly work and archival collections and greatly enhanced access to these materials for researchers. Greater visibility and access, however, also bring greater potential for requests that libraries takedown materials either because of intellectual property rights claims or other claims, such as privacy.
Schofield and Urban studied the experience of academic libraries hosting open access repositories and their experience with notice and takedown requests, both under section 512(c) of the Digital Millennium Copyright Act (“DMCA”) and otherwise. They used a survey and targeted interviews to investigate how often takedown requests are received, for what type of content, the basis of the concern, and how the library responded to the takedown request. Schofield and Urban go on to provide recommendations on how libraries should respond to these takedown requests. Their findings have been published in Berkeley Digital Library Copyright Project Report: Takedown and Today’s Academic Digital Library. (available at SSRN) and will be presented at The Future of Libraries in the Digital Age conference.
Establishment of accessible repositories has been on the rise as the issue of access to research, particularly publicly-funded research, has gained attention in academia and the press. Traditionally the academic publishing model has been one where the author(s) sign over rights to their work in exchange for publication. Academic libraries then pay substantial sums of money to gain access to journals, and other publications, in which the research is published. The open access movement gained momentum particularly in the hard sciences and there are now federal, and sometimes state, restrictions requiring that certain types of publicly funded scientific research are made openly accessible for no cost. In addition, some academic institutions have begun encouraging, or even requiring, faculty to publish research in an open access format. These trends have fueled, in part, the rise in academic digital repositories.
Section 512(c) of DMCA (17 U.S.C. § 512(c) (2014)) provides protection for an “online service provider” (“OSP”) for copyright infringement by a user of the OSP. Schofield and Urban note that libraries developing and maintaining online, publicly accessible repositories may meet the definition of an OSP under the DMCA and become subject to both its requirements and protections. However, the authors also point out that the safe harbor provisions are only available for content loaded by a third party, such as a faculty member or student. Libraries who manage academic digital repositories often load the content on behalf of the author. As Schofield and Urban emphasize, that step, when performed by the library, eliminates the protections of § 512(c) since the library, through a librarian or staff member, did the actual loading of content.
The study revealed DMCA takedown requests are currently infrequent, although their incidence could rise as repositories become more prevalent. More common were non-DMCA takedown requests. While some of these did arise from copyright claims, the most frequent reasons given were privacy, embarrassment, and defamation concerns. Schofield and Urban found that many of these non-DMCA takedown requests were handled on a case-by-case basis, depending upon the cause for complaint.
Although the respondent pool for the study was small, see the article for why, the findings are intriguing and indicate that librarian managers of these repositories should be developing best practices for handling takedown requests as they are likely to grow in number and frequency. Some of the recommendations from the authors, for both DMCA and non-DMCA takedown requests, include author education about preserving rights, publication agreement transparency, and development of best practices within the academic library community. As the Schofield and Urban report highlights, authors, publishers, and academic institutions are likely to find the incidence of takedown requests on the rise. Academic libraries, as the developers, managers, and curators of digital repositories, should be prepared to respond.
Cite as: Kristina Niedringhaus, Responding to Takedown Requests for Digital Library Repositories
(March 21, 2016) (reviewing Brianna L. Schofield & Jennifer M. Urban, Berkeley Digital Library Copyright Project Report: Takedown and Today’s Academic Digital Library
, U.C. Berkeley Pub. L. Research Pape
r No. 2694731 (November 2015), available at SSRN), https://lex.jotwell.com/responding-to-takedown-requests-for-digital-library-repositories/
James M. Donovan, Carol A. Watson & Caroline Osborne, The Open Access Advantage for American Law Reviews
(October 7, 2014), available at SSRN
Open access (OA) scholarship is available online, without fees, and free of restrictive copyright and licensing provisions. As institutions of higher education implement a more metrics-driven paradigm, law schools are increasingly attentive to the quantification of both individual faculty and aggregate law school impact. Citation counts are one means of quantifying these impacts. Donovan, Watson, and Osborne build on their 2011 article, Citation Advantage of Open Access Legal Scholarship, which demonstrated that open access resources have a great impact on legal scholarship, (103 Law Lib. J. 553, 557). In this article, they work to develop a systematic and scientific explanation for why open access scholarship has a citation advantage in the legal education context.
The authors’ research shows that articles published simultaneously as print and open access law review articles provide at least a 50% citation advantage over their print-only law review counterparts. More specifically, they find that the aggregate cumulative OA advantage for new and retrospective works combined is about 53%; the OA advantage of newer works published during the years 2007-2012 is about 60%. Their research also indicates that OA articles are more heavily cited in the years immediately following an article’s publication and that OA articles tend to “command greater attention over the lifespan of the work” (Donovan et al, at 8).
The authors also explore the measurement of the OA advantage to a law review as it relates to the institution’s ranking in the U.S. News & World Report. They conclude that the greatest OA advantage is for a journal whose home institution is in tier 2, 3, or 4 of the U.S. News & World Report law school ranking. For those tiers, the aggregate cumulative OA advantage for new and retrospective works combined is about 51% compared to an OA advantage of new works published during the years 2007-2012 of about 89% for tiers 2 and 3, and 81% for tier 4. For journals at tier 1 schools, the OA impact decreases significantly because journals at higher ranked institutions have high levels of exposure even without OA. In this tier, the aggregate cumulative OA advantage for new and retrospective works combined is about 11% compared to an OA advantage of new works published during the years 2007-2012 of about 16%.
As the authors point out in their conclusion, this article is a sobering reminder that readily available information on the Internet will often be the first, and in some cases the only, source consulted. Consequently, OA publishing offers faculty the potential opportunity to increase their work’s exposure in the field by being readily available, and therefore, is fertile ground for the OA citation advantage. According to Donovan, Watson, and Osborne, the OA citation advantage for a law review article is threefold: an OA article gets attention sooner; about half of the citations to an OA article will be from the first six years of the publication’s existence; and OA articles receive attention for a sustained period of time that exceeds the length of attention received by its non-OA counterparts. Depositing faculty scholarship in an open access repository, whether in SSRN or in an educational institution’s repository, is a simple, tasteful way that faculty can promote their scholarship while supporting the open access movement.