A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories. A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Types of Literature Reviews

Argumentative Review
This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review
Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review
Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review
A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review
This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review
The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE: Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem:

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider:

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing
1.  Problem formulation -- which topic or field is being examined and what are its component issues?
2.  Literature search -- finding materials relevant to the subject being explored.
3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic.
4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review:

Clarify
If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions:
1.  Roughly how many sources would be appropriate to include?
2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)?
3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue?
4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem?
5.  Should I provide subheadings and other background information, such as definitions and/or a history?

Find Models
Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research.

Narrow the Topic
The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the libraries catolog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text.

Consider Whether Your Sources are Current
Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events
If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union.

By Publication
Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies.

Thematic [“conceptual categories”]
A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made.

Methodological
A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review
Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation: Information necessary to understand the current topic or focus of the literature review.
• Sources Used: Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History: The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods: Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards: Description of the way in which you present your information.
• Questions for Further Research: What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence
A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid.

Be Selective
Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings.

Use Quotes Sparingly
Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature.

Summarize and Synthesize
Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others.

Keep Your Own Voice
While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording.

Use Caution When Paraphrasing
When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective.

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings. If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work. If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.
• 
  

Citation tracking [a.k.a., citation analysis, citation tracing, or cited reference searching] refers to a deliberate, systematic method of measuring and evaluating the impact of research studies over time by counting the number of times that an author or publication has been cited in other works. Tracking citations helps identify leading scholars in a particular discipline and the impact of a particular study based upon an analysis of who has cited a particular study, how often a specific study has been cited by others over time, and by determining what disciplines are represented, either by the scope of the publication or by the author's affiliation, based on those subsequent citations.

Reasons to Track Citations as Part of Your Review of the Literature

It can be an effective way of using a frequently cited "landmark" or groundbreaking article to find more recent, related publications that cite the original work.

It can be an effective strategy of identifying leading scholars in a particular area of study either by how frequently their work has been cited or by the number of scholars who have subsequently cited the author's original work.

It can be a useful means for evaluating a study's impact within a particular discipline based upon the number of times a research study has been cited by others. A frequently cited study may indicate that the research findings are unique, groundbreaking, insightful, or in some way, seen by scholars as problematic or controversial.

It can be an efficient way to locate studies that critique or challenge groundbreaking research, which is important if you are seeking research to support your attempt to challenge long-held assumptions or to offer evidence that existing studies do not adequately address the research problem.

It can be an effective means of determining the interdisciplinary impact of a particular study because you can identify the number of subsequent citations in publications by scholars in other areas of study. Research that has been cited in a variety of disciplines is also a strong indication of the research study's overall impact throughout the social sciences [and perhaps beyond].

NOTE: A common assignment in social sciences classes is to assign each student the responsibility for summarizing and discussing a journal article from the required readings of the class syllabus. Before you lead the class discussion of the study, copy and paste the article you are discussing into Google Scholar. This will likely reveal "Cited by" references [e.g., Cited by 102] to other works that have subsequently cited the original study. In so doing, you can add during your in-class discussion of the article a description of how other scholars have used it to support their own research and, by extension, the ways in which the original research influenced future studies of the topic.

Keep in mind the following points, however, when using methods for tracking citations to expand the scope of the literature you want to review:

Authors do not always use the same name throughout their careers [e.g., Jane Anne Smith or Jane A. Smith] so be sure you work from a complete and accurate list of an author's publications. A thorough literature review of an author should reveal any variations in their name. To this point,...

The Web of Science citation database uses the APA [American Psychological Association] style for citing authors [last name and first initial only], so a J Smith could be John, Jeff, Jane, Julie, Jason, etc. Smith. Be sure to truncate the initial  by adding an asterisk [e.g., J*] to see a more complete list of authors, then locate a record on a topic you know the author writes about and click on that author to exclude articles written by other J Smith's. Fortunately, the database indexes more than the first author of a paper so, if a second or third author has an uncommon name, you can search the unusual name to locate the article and who has cited it instead of using an author's more common name.

Citation services are primarily based on existing journal literature. If the author is cited in books, organizational research reports, government documents, or non-scholarly publications, the usefulness of your citation analysis is limited. In addition, citation databases such as Web of Science have limited coverage of articles published in scholarly open-access journals [peer-reviewed journals published freely online], although this problem is slowly improving. In this case, be sure to check the "cited by" references in Google Scholar as it often includes citations found in other publications besides scholarly journals.

Google Scholar and most library research databases do an inadequate job citing to non-English language sources. English is generally considered the universal language of scholarly research, which some argue is a remnant of colonialism. As a result, this privileges English-born publications at the expense of research published in other languages, particularly if the study is written using non-Roman characters [e.g., Korean, Hindi, Russian, etc.]. This deficiency is slowly improving as private database vendors and, in particular Google Scholar, begin to include more non-English language studies in their systems. Nevertheless, always keep in mind that the number of citations to an original study may not represent its true impact among scholars in non-English speaking regions of the world, particularly if the study focuses on a non-English-speaking region or country. If needed, contact an expert for help identifying studies published in languages other than English. A thorough review of the literature should include studies published in other languages if you are able to read those languages and the research directly relates to the problem you are investigating.

Records found while searching Google Scholar and many library databases often include a "cited by" reference followed by a number [e.g., Cited by 143]. This number indicates, in general, how many times a study has been subsequently cited by other authors in other publications [I say "in general" because the total may include duplicate records, such as, an author citing their own work or citations to the same study in multiple types of publications].

However, the total number of times a journal article or other publication has been cited after it was published is more nuanced than just the number count. When reviewing "cited by" references as part of your review of the literature, consider the following interpretive principles that will help you determine if the source should be examined more carefully.

An item that is cited many times indicates impact, but it does not indicate value. Citations are value-neutral. A study may be cited frequently because it added important new knowledge to the understanding of a research problem. However, a study could also be cited frequently because subsequent studies found the original study to be poorly designed or the author employed an imprecise methodological approach or the conclusions were speculative and perhaps misleading [also see Another Citing Tip text box below]. Use the date limitation function to review a sample of recent studies that have referenced the work and assess how the original source is currently being described. Based on this, determine how the frequently cited source may be relevant to your own research. Note that if subsequent research shows the original study to be flawed, you can reference the study in your paper as an example of why further research is needed.

A source that is frequently cited is a subjective assessment of impact, not an objective measurement or formal statistical calculation. Should a study published ten years ago with 800 cited by references be considered a frequently cited source? How about a study published ten years ago that has over 2,000 cited by references? Is that source better than the other in terms of assessing whether it should be reviewed in detail? What about a source with only 120 cited by references? There are a variety of factors that contribute to whether a source has been cited frequently or not. Ultimately, the number of cited by references is relevant to you only in relation to the research problem being investigated. If the study directly informs your understanding of the topic, you should review the cited by references. If there are a manageable number, review them all; if there is a large number of references, focus on reviewing the most recently published sources [e.g. last three to five years].

Once published, a recently published study can become cited by other authors very quickly. This is particularly the case if the study was published in a core research journal read by almost all the leading scholars of the discipline, it was published in a source read by researchers from a variety of disciplines, the study was written by a well-known and preeminent scholar, or the research has been written about in national newspapers and magazines, discussed during television news programs, or discussed and shared on social media sites. A way to discern impact over time is to count the number of recent citations to a study [e.g., the last five years]. If the study has been cited only a few times, this likely indicate that the study's impact is waning and scholars have moved on.

A study may be considered foundational or groundbreaking if it continues to be cited frequently in the most recently published literature on the topic. A source published many years ago that continues to cited can indicate the research was the first to advance new knowledge and, therefore, the study either contributed significantly to understanding the research problem or the findings' ability to inform practice was particularly important or innovative. In most cases, there is an expectation that a study interpreted to be foundational or groundbreaking should be cited as the basis for supporting your overall review and assessment of prior research or to document when a research problem first emerged.

Review the cited by references of relevant resources to determine a study's impact across disciplines. Cross-disciplinary cited by references can be an important indicator of a study's overall impact if it has been referenced in a variety of different investigative contexts or domains of applied practice. Consider reviewing these sources in more detail in support of your own review of the literature. Do this by comparing the disciplinary focus of the original source in relation to the disciplinary focus of the cited by references. For example, an article originally published in a social work journal that is cited in journals from sociology, education, and psychology demonstrates impact beyond the field of social work.

Examine whether the source is referenced alone or always grouped with other sources. This act of interpreting impact can be challenging, but, in general, a study that is consistently cited with other sources, though rarely cited as a stand-alone study, may indicate that the research is not unique or distinctive in a way that stands out from the overall domain of prior research about a topic. For example, you find that Smith's original study is always referenced along with other authors doing similar work. This should not be viewed necessarily as negative because many scholars may have investigated a specific research problem. However, in interpreting the relevance of a particular source in relation to your own research, studies that have never been cited as a stand-alone source of prior research can be an additional way of determining if they would add value to your own evaluation of prior research.

Cited by references listed in the first few years after the original study was published can add important insights. For example, the earliest studies to cite an original work can highlight the initial interests and potential biases of scholars at the time. In addition, they can reflect a growing awareness of the need to critically examine the research problem, they can highlight the application of prevailing theories and ideas at the time, and, in some cases, the earliest published studies citing the original source can help illuminate key intellectual struggles among scholars as they attempt to make sense of the emerging problem. All or any of these factors can help contextualize the way in which you use prior research to write about the background and history of a topic in the introduction or literature review sections of your paper.

Searching Cited By References Without Getting Lost

Tracking cited by references can be an effective strategy for identifying relevant, more recently published literature about a specific research problem. However, citation tracking is not a linear process, but rather, an iterative process comprising of multi-layered pathways of discovery. Not only can you identify citations that directly cite the original study, but those studies may also have cited by references that can be reviewed, then those studies can have cited by references that can be searched, and so on.

While this process of traversing the literature by reviewing multiple layers of cited by references can be useful in discovering new sources that perhaps would have been hidden otherwise, one can easily get lost in the process. This is particularly true with following cited by references in Google Scholar because the content is so immense. A good strategy for not getting lost is to note a promising record that cites a source that has cited the original source so you can go back and review the cited by references of that study without worrying about losing track of where you are in the initial review process.

The "Replication Crisis" in the Social Sciences and Searching Cited By References

The replication crisis in academe refers to research showing that findings published in top journals often cannot be replicated in subsequent studies. The ability to replicate research is a core principle of the scientific method of discovery and verification. Replicating the research of prior studies increases the probability and, by extension, the confidence one has that a given finding is valid in a real world setting or is reliably applicable to professional practice. The problem, as noted by Prochazka, Fedoseeva, and Houdek (2021) is that, “…the replication crisis in the social sciences has showed that many of the published effects has been found by chance or due to the specific conditions of a study…Therefore, an effect found in a single study should be viewed with caution.”

But that’s not all. Nonreplicable studies tend to be cited more frequently than studies that have been successfully replicated. One reason for this, according to researchers, is that authors are often motivated to use an unorthodox approaches to studying a problem. This is because methodological unorthodoxy is often rewarded [intentionally or not] by institutions of higher education in the faculty promotion process and by journal publishers who want to be seen as on the cutting edge of research. Replicating another person’s research doesn’t garner the same attention towards obtaining prestigious grant funding or gaining tenure for the author as a “unique” study. As a result, this tendency can distort the total number count of citations in some studies, making it more difficult to discern a truly groundbreaking study from a “failed” nonreplicable study that has been cited simply because it’s found to be unique in some way.

To address this issue when searching cited by references, review studies that cite the original study shortly after it was published. Are authors attempting to replicate the research? If so, are they doing so successfully and building on this research or are they simply citing the work in their literature review section. In the end however, don't get too caught up in this "crisis"; always keep in mind that the purpose of citation tracking is to build a body of knowledge that supports investigating the research problem.

In general, "alert services" refer to features included with scholarly databases or made available by journal publishers that allow you to be notified by email or text message when something of interest to you has been added to a database or published in a journal. Alert services can be set up to notify you about newly published resources on a specific topic or when new articles are published in a journal.

USC Libraries subscribe to a number of databases that make electronic alert services available to users. Three different types of alert services are:

1. Table of contents alerts -- updates of the table of contents of the most current issues of the journals you specify when signing up.
2. Daily/weekly email alerts -- alerts that notify subscribers of articles matching submitted topics. Alert frequencies vary depending on the publisher's database updates.
3. Saved search alerts -- emailed notifications of recent articles matching previously submitted searches.

While conducting a literature review, content alert services can be especially useful because:

1. They can alert you to new articles in journals of particular interest or that you know are most likely to publish research on the topic you are investigating.
2. Databases that index journals from a variety of different fields of study offer you multidisciplinary coverage of articles related to your topic of interest.
3. They can alert you to new "pre-published" research [essentially final drafts of articles] before they are distributed to libraries and subscribers.

NOTE: In order to sign up for an alert service, an email address is required along with a username and password that you create. Be sure you read the privacy policy carefully before signing up so that you avoid receiving unwanted spam and solicitations from publishers.

Multidisciplinary alert services that notify you when a new issue of a journal is published.

• CiteULike -- currently has details of over 13,000 journals. You can search or browse for journal titles, and then scan recent articles in these journals.  If you know aboutt RSS feeds, you can get a CiteULike feed for each journal table of contents. Access to the full text will depend on institutional or personal subscriptions. Registration is free.
• ticTOCs -- covers over 12,500 journal table of contents from more than 430 publishers. You can search for journal titles, view the latest  table of contents for each journal, link to the full text of around 390,000 articles (where institutional or personal subscription allows), export table of content feeds to popular feedreaders, and select and save journal titles in order to view future table of contents (you need to register to ensure your ‘MyTOCs’ are permanently saved). Registration is free.

Procedures for setting up alert services from indexes and databases available from the USC Libraries.

• CQ Researcher -- log in or create a user profile. Select "Create New Alert" to customize your e-mail alerts. Use the "Start/Stop" action to start or stop receiving e-mail alerts.
• CSA Illumina Database -- click on the "Alerts" link located on the right hand side of the search page to login and create alerts.
• EBSCO databases -- conduct a search in one of the Ebsco databases. Select Search History/Alerts, then go to: Save Searches. Log in to your account (or register) to Create or Edit Saved Search Alerts. 
• Project MUSE -- allows you to set up alerts by journal name or by subject area or both. You will need to set up an alerts account with a username and password and an email address. You can make selections for journal titles individually or with a mix of subject collections.
• Web of Knowledge -- register with an email and password. For detailed instructions on how to setup alerts for journal content and citations, click on "Tutorial" link on the opening web page and follow the steps to setup alerts.

Descriptions of resources are adapted or quoted from vendor websites.

Evaluating the authority, usefulness, and reliability of resources is a crucial step in conducting a review of the literature that effectively covers pertinent research and, thereby, demonstrates to the reader that you know what you're talking about. The process of evaluating scholarly materials also enhances your general skills and ability to:

1. Seek out alternative points of view and differing perspectives,
2. Identify possible bias in the work of others,
3. Distinguish between fact, fiction, and opinion,
4. Develop and strengthen your ability to distinguish between relevant and non-relevant content,
5. Draw cogent, well thought out conclusions, and
6. Synthesize information, extracting meaning through a deliberate process of interpretation and analysis.

The act of thinking critically about the validity and reliability of a research resource generally involves asking yourself a series of questions about the quality of both the item and the content of that item.

Evaluating the Source

Inquiring about the Author
What are the author's credentials, such as, institutional affiliation [where he or she works], educational background, past writings, or experience? Is the book or article written on a topic in the author's area of expertise? Has your instructor mentioned this author? Have you seen the author's name cited in other sources or bibliographies? Is the author associated with a reputable institution or organization? What are the basic values or goals of that organization or institution?
Inquiring about the Date of Publication
When was the source published? Is the source current or out-of-date for your topic?
Inquiring about the Edition or Revision
Is this a first edition of this publication or not? Further editions usually indicate a source has been revised and updated to reflect changes in knowledge, to include prior omissions, and to better harmonize the contents with the intended needs of its readers. If you are using a web source, do the pages indicate last revision dates?
Inquiring about the Publisher
Note the publisher. If the source is published by a university press, it is likely to be scholarly. Although the fact that a publisher is reputable does not necessarily guarantee quality, it does show that the publisher has a high regard for the source being published [their reputation as an academic publisher relies on it].
Inquiring about the Title of Journal
Is this a scholarly or a popular journal? This distinction is important because it indicates different levels of complexity in conveying ideas and the intended readership.

Evaluating the Content

Intended Audience
What type of audience is the author addressing? Is the publication aimed at a specialized or a general audience? Is this source too elementary, too technical, too advanced, or just right for your needs?

Objectivity
Is the information covered considered to be fact, opinion, or propaganda? It is not always easy to separate fact from opinion. Facts can usually be verified; opinions, though they may be based on factual information, evolve from the interpretation of facts. Does the information appear to be valid and well-researched, or is it questionable and unsupported by evidence? Note errors or omissions. Are the ideas and arguments advanced more or less in line with other works you have read on the same topic?

Coverage
Does the work update or clarify prior knowledge, substantiate other materials you have read, or add new information? Does it extensively or only marginally cover your topic? Does it provide a balanced perspective? If the item in question does not meet this criteria, you should review enough sources to obtain a variety of viewpoints.

Writing Style
Is the publication organized logically? Are the main points clearly presented? Do you find the text easy to read, or is it stilted or choppy? Is the author's argument repetitive?

Evaluative Reviews
In the case of books, locate critical reviews of the work in a database such as ProQuest Multiple. Is the review positive? Is the book under review considered a valuable contribution to the field? Do reviewers agree on the value or attributes of the book or are there strong differences of opinion? Does the reviewer mention other books that might be better? If so, locate these sources for more information on your topic.

Web Content Requires Additional Methods of Evaluation

A report from the Stanford University Graduate School of Education found that students evaluating information that flows across social media channels or retrieved from online search engines like Google or Bing, have difficulty distinguishing advertisements from news articles or how to identity where the content came from. In general, the principles that guide your evaluation of print materials are the same that apply to evaluating online resources. However, unlike print materials that have certain features that help determine their scholarly integrity, the interactive and multimedia dynamics of online sources requires additional attention to the content in order to obtain confidence that what you are viewing is valid and credible.

Additional things to look for when considering using an online resource:

• Source of the content is stated -- determine whether the content is original or borrowed, quoted, or imported from elsewhere. Note that content imported from another source via RSS feed can be difficult to identify, as this material can be incorporated into other content on the page without being appropriately labeled.
• Don't be fooled by an attractive, professional-looking presentation -- just because a site looks professional doesn't mean that it is. However, poorly organized web page designs or poorly written content is easy to recognize and can be a signal that you should carefully scrutinize the site's content.
• Site is currently being maintained -- check for last posting dates or last revised dates. Most scholarly websites show a date when the content was last posted or revised. Note that, if no date is indicated, this does not mean its content is invalid. However, it may indicate that the content is out-of-date and does not reflect current information about the topic.
• Links are relevant and appropriate, and are in working order -- a site with a lot of broken links is an indication of neglect and out-of-date content.
• Clearly states authorship -- if a site is produced anonymously, you cannot verify the legitimacy of its creator. Note that an author of a site can be either be a person or an organization.
• The site includes contact information -- if you have questions about the site, contact information is an important indicator that the site is well-maintained.
• Domain location in the site address (URL) is relevant to the focus of the material [e.g., .edu for educational or research materials; .org for non-profit organizations; .gov for government sites; .com for business sites]. Note that the domain is not necessarily a primary indicator of site content. For example, some authors post their content on blog or wiki platforms hosted by companies with .com addresses. Also note that the tilde (~) in the URL usually indicates a personal page.

Detecting Bias

Bias, whether done intentionally or not, occurs when a statement reflects a partiality, preference, or prejudice for or against an object, person, place, or idea. Listed below are problems to look for when determining if the source is biased.

1. Availability Bias -- this is a tendency for people to overestimate probabilities of events related to memorable or dramatic occurrences [e.g., after 9/11, people took vacation by traveling by car rather than airplane even though, statistically, car travel is much more dangerous]. This form of bias in a research study can take the form of an example used to support author’s argument or the design a case study focused around a particular event. Unless the purpose of the study is to illuminate new understanding around a memorable or dramatic occurrence, be critical of studies that use this type of measurement to examine a research problem. A seemingly mundane or uneventful occurrence can be just as valid in developing solutions to a problem or advancing new knowledge.
2. Distortion or Stretching of the Facts -- this refers to the act of making issues, problems, or arguments appear more extreme by using misinformation or exaggerated and/or imprecise language to describe research outcomes [e.g., “Everyone agreed the policy was a complete disaster.” Who's everyone? How was data gathered to come to this conclusion? And, how does one specifically define something as a "disaster"? Is there sufficient evidence to support such a broad statement?]. Look for declarative statements that lack appropriate reference to supporting evidence or are follow up with detailed analysis.
3. Flawed Research Design -- bias can enter the narrative as a result of a poorly designed study; this may include a claim or generalization about the findings based upon too small a sample, manipulating statistics, omitting contrary conclusions from other studies, or failing to recognize negative results [results that do not support the hypothesis].
4. Lack of Citations -- it is acceptable to issue a broad declarative statement if it is clearly supported and linked to evidence from your study [e.g., "Testimony during Congressional hearings shows that the Department of Education is reluctant to act so teachers must do so"]. This problem refers to statements or information presented as fact that does not include proper citation to a source or to sources that support the researcher's position, or that are not statements explicitly framed as the author's opinion.
5. Misquoting a Source -- this is when an author rewords, paraphrases, or manipulates a statement, the information about a source is incomplete, or a quote is presented in such a way that it misleads or conveys a false impression. This is important when paraphrasing another author. If you cannot adequate summarize a specific statement, finding, or recommendation, use a direct quote to avoid any ambiguity.
6. Persuasive or Inflammatory Language -- using words and phrases intended to elicit a positive or negative response from the reader or that leads the reader to arrive at a specific conclusion [e.g., referring to one group in an armed conflict as “terrorists” and the other group as “peace-loving”].
7. Selective Facts -- taking information out of context or selectively choosing information that only supports the argument while omitting the overall context or vital supporting evidence.
8. Statistical Survey Bias -- this can take several forms so, if data is presented in a study that was gathered by the author(s), examine it critically for the following possible biases:
   • Measurement Error: this results from problems with the process by which data was gathered, such as, the use of leading questions that influence the response rate or that are biased toward what respondents believe is socially desirable because most people want to present themselves favorably. The only way to assess bias in these cases is to have access to the survey instrument used to gather data.
   • Sample Size: increasing the number of a sample, for example the number of people interviewed, does not necessarily decrease bias, but look to see if the sample used is representative of the population under study to ensure that any generalizations or conclusions from the interpretation of the data is valid.
   • Undercoverage: this refers to the method of data gathering that is a result of non-response to a survey because some subjects do not have the opportunity to participate. In looking at data, be sure to understand the percentage of non-responses to a survey or groups of people who were not included.
   • Voluntary Response: this bias occurs when respondents to a survey are self-selected, resulting in an overrepresentation of individuals who have strong opinions [e.g., data from a radio call-in show]. Be an especially critical reader of web-based surveys about controversial topics if the author(s) have not indicated how they interpreted thew data from voluntary surveys.

NOTE:  The act of determining bias in scholarly research is also an act of constant self-reflection. Everyone has biases. Therefore, it is important that you minimize the influence of your own biases by approaching the assessment of another person's research introspectively and with a degree of self-awareness.

ANOTHER NOTE:  The idea of bias often carries negative connotations, even though the meaning of the term is not defined in that context. Be sure when you are evaluating a source that you do not automatically reject it as invalid if you detect bias. One way to do this is to substitute the idea of bias with the idea of perspective. Ask yourself, what type of perspective does this source bring to the investigation of the research problem? As Lesh points out, the goal of research is to engage with multiple sources for the purpose of acquiring multiple perspectives about the topic. As long as the source is rooted in fact-based evidence, you should not reject it as being biased, but rather, consider it as a potential source of perspective about the research problem.

The CRAPP Test

This stands for Currency, Relevance, Accuracy, Authority, and Purpose. It is an apronym developed within the field of librarianship as short-hand for remembering the essential actions associated with effectively assessing the usefulness of a source in relation to the research problem you are investigating. Each word relates to a set of questions you should ask yourself when determining the validity of a source. These are:

• Currency --  relates to the timeliness of the information.
  • When was the information published or posted online?
  • Has the information been revised or updated?
  • Does your topic require current information, or will older sources also support your study?
  • Are the links functional?
• Relevance -- the importance of the information in relation to your research needs.
  • Does the information relate to your topic or address your research question(s)?
  • Who is the intended audience?
  • Is the information at an appropriate level (i.e., not too elementary or advanced for your needs)?
  • Have you looked at a variety of sources before determining this is one you will use?
  • Would you be comfortable citing this source in your research paper?
• Accuracy -- relates to identifying and verifying the source of the information.
  • Who is the author, publisher, source, or sponsor?
  • What are the author's credentials or organizational affiliations?
  • Is there evidence of the author's qualifications to write about the topic?
  • Is there contact information, such as a publisher or email address?
  • Des the URL reveal anything about the author or source [e.g., .com .edu .gov .org .net, etc.]?
• Authority -- relates to the reliability, truthfulness, and correctness of the content.
  • Where does the information come from?
  • Is the information supported by evidence?
  • Has the information been reviewed by an editor or peer reviewed?
  • Can you verify any of the information in another source or from personal knowledge?
  • Does the language or narrative tone appear objective and unbiased?
  • Are there spelling, grammar, or typographical errors?
• Purpose -- relates to the reason the information exists.
  • What is the purpose of the information, i.e., is it to inform, teach, sell, entertain, or persuade?
  • Do the author(s) explain why the information has been studied?
  • Is the information fact, opinion, or propaganda?
  • Does the point of view appear objective and impartial?
  • Are there political, ideological, cultural, religious, institutional, or personal biases?

As described above, a thorough evaluation of sources can encompass more than this basic model. Nevertheless, if you remember anything about how to evaluate a source as you conduct a literature review, remember this approach. The CCRAP Test applied to any source produced in any format [e.g., text, online, statistical, multimedia].

Primary sources are materials that were either created during the time period being studied or were created at a later date by a participant in the events being studied, such as, a childhood memoir. They are original documents [i.e., they are not about another document or account] and reflect the individual viewpoint of a participant or observer. Primary sources represent direct, uninterpreted records of the subject of your research study.

Primary sources enable you to get as close as possible to understanding the lived experiences of others and discovering what actually happened during an event. However, what constitutes a primary or secondary source depends on the context in which it is being used. For example, David McCullough’s biography of John Adams could be a secondary source for a paper about John Adams, but a primary source for a paper about how various historians have interpreted the life of John Adams. When in doubt, ask an expert at HAMNIC Solutions for assistance!

Reviewing primary source material can be of value in improving your overall research paper because they:

1. Are original materials,
2. Were created from the time period involved,
3. Have not been filtered through interpretation or evaluation by others, and
4. Represent original thinking or experiences, reporting of a discovery, or the sharing of new information.

Examples of primary documents you could review as part of your overall study include:

• Artifacts [e.g. furniture or clothing, all from the time under study]
• Audio recordings [e.g. radio programs]
• Diaries
• Internet communications on email, listservs, blogs, Twitter, Facebook, and other social media platforms
• Interviews [e.g., oral histories, telephone, e-mail]
• Newspaper articles written at the time
• Original official documents [e.g., birth certificate, will, marriage license, trial transcript]
• Patents
• Personal correspondence [e.g., letters]
• Photographs
• Proceedings of meetings, conferences and symposia
• Records of organizations, government agencies [e.g. annual report, treaty, constitution, government document]
• Speeches
• Survey Research [e.g., market surveys, public opinion polls]
• Transcripts of radio and television programs
• Video recordings
• Works of art, architecture, literature, and music [e.g., paintings, sculptures, musical scores, buildings, novels, poems]

In the social sciences, a secondary source is usually a scholar book, journal article, or digital or print document that was created by someone who did not directly experience or participate in the events or conditions under investigation. Secondary sources are not evidence per se, but rather, provide an interpretation, analysis, or commentary derived from the content of primary source materials and/or other secondary sources.

To do research, you must cite research. Primary sources do not represent research per se, but only the artifacts from which most research is derived. Therefore, the majority of sources in a literature review are secondary sources that present research findings, analysis, and the evaluation of other researcher's works.

Reviewing secondary source material can be of value in improving your overall research paper because secondary sources facilitate the communication of what is known about a topic. This literature also helps you understand the level of uncertainty about what is currently known and what additional information is needed from further research. It is important to note, however, that secondary sources are not the subject of your analysis. Instead, they represent various opinions, interpretations, and arguments about the research problem you are investigating--opinions, interpretations, and arguments with which you may either agree or disagree with as part of your own analysis of the literature.

Examples of secondary sources you could review as part of your overall study include:
    * Bibliographies [also considered tertiary]
    * Biographical works
    * Books, other than fiction and autobiography
    * Commentaries, criticisms
    * Dictionaries, Encyclopedias [also considered tertiary]
    * Histories
    * Journal articles [depending on the discipline, they can be primary]
    * Magazine and newspaper articles [this distinction varies by discipline]
    * Textbooks [also considered tertiary]
    * Web site [also considered primary]

A tertiary source consolidates and organizes primary and secondary source materials into one source in order to facilitate quick access to information. Tertiary sources are good starting points for research projects because they often extract the essential meaning or most important aspects of large amounts of information into a convenient format.

The distinctions between primary, secondary, and tertiary sources can be rather ambiguous depending upon the context in which an item is used. Some writers don't make a distinction between tertiary and secondary because both types of materials do not represent original works [i.e., primary sources]. However, for the purposes of reviewing the literature, it is important to understand how tertiary sources can contribute to the overall process of locating relevant information about a research topic.

Reviewing tertiary source materials can be of value in improving your overall research paper because they:

• Consolidate information in one place. Searching for information published in multiple sources takes time. A tertiary source, such as an directory, saves time because all the relevant information is compiled in one place.
• Facilitate comparing and contrasting information. One of the purposes of tertiary sources is to organize a lot of information in a way that facilitates easily comparing and/or contrasting the content. The information is formatted specifically to help the reader understand complex information or data.
• Publish regularly over a period of time. Some tertiary sources are published in predictable intervals, such as, almanacs, directories, or handbooks. This means they can be important sources of consistently presented information over time, providing a longitudinal perspective about the information they present.
• Distill large quantities of closely related information or data. A dictionary, specialized encyclopedia, or a statistical compendium can be used to review a large amount of information that would otherwise be difficult to synthesize and differentiate.
• Often contain references to additional sources. Many tertiary sources, in particular bibliographies, can be efficient places to go to locate key primary and secondary sources about a topic. In this way, they can be a good starting point when reviewing prior research on a topic.

Examples of tertiary sources you could review as part of your overall study include:
    * Abstracts
    * Almanacs
    * Bibliographies [also considered secondary]
    * Chronologies
    * Dictionaries and encyclopedias [also considered secondary]
    * Directories
    * Fact books
    * Handbooks
    * Indexes, databases, search engines, and bibliographies used to locate primary and secondary sources
    * Manuals
    * Statistical compendiums
    * Textbooks and course readers [may also be secondary]

Note that tertiary sources also include any type of user-contributed online resource such as Wikipedia.

There are three types of publications that may appear in the search results of most social and behavioral sciences databases. These are:

• Scholarly sources -- intended for use in support of conducting in-depth research, often containing specialized vocabulary and extensive references to sources. The content has been reviewed by other expects in the field to ensure the reliability of methods used and the validity of findings. Scholarly sources help answer the "So What?" question in academic writing and lay the foundation for discovering connections between variables, issues, events, or phenomena.
• Popular sources -- intended for a general audience of readers, they are written typically to entertain, inform, or persuade. Popular sources help you answer who, what, when, and where questions and are essential for finding information about current events or issues. Popular sources range from research-oriented [but lacking specific citations to sources] to special interest, agenda-driven publications that are intended to persuade the reader to believe a particular way.
• Trade publications -- intended to share general news, trends, and opinions among practitioners in a certain industry or profession. Although generally written by experts, they are not considered scholarly because they are not peer-reviewed and do not focus on advancing new knowledge discovery or reporting research results except in the context of improving best practices and fostering innovation. Trade journals, however, are an essential source of information about emerging trends in the field of business and specialized industries [e.g., tourism, environmental studies, agriculture, manufacturing, etc.].

Below is a chart developed by the USC Libraries instruction team that can help you distinguish between a scholarly [a.k.a., peer-reviewed or academic] journal article and a popular, general interest publication.