On the Rise

Is This Data for Real!?!?

On The Rise is a social norms campaign based on specific questions from the 2016 Rise Above Colorado Youth Survey. Teen participants reported their own use, which was then compared to their estimates of their peers’ use in the past 30 days. The social norms approach is all about closing the gap between people’s misperception and reality.1 And when it comes to substance misuse, the majority of teens frequently hold inaccurate ideas of how many of their peers are actually misusing. Peer pressure, the social norms that “everyone is doing it” or that “experimentation is just part of being a teenager” all can hold a lot of weight without anyone ever saying anything. Correcting misperceptions about substance misuse makes healthy choices easier because we come to realize that the saying “everyone is doing it” is just simply not the case.

Do misperceptions about substance misuse exist among Colorado teens?
The first step for developing On the Rise was determining if a gap exists between teens’ perceptions and reality and, if so, how common those misperceptions are. The following Rise Above Colorado Youth Survey questions were considered to determine the need and value of a social norming campaign:

  • Perception of Usage – What percentage of students at your school do you believe have used [each substance] during the past 30 days? (Shown in blue in the graphs below.)
  • Reported Usage – During the past 30 days, how many times have you used [each substance]? (Shown in pink in the graphs below.)

The following two graphs display the data for the responses to these two survey questions for middle school youth (12-14 year olds) and high school youth (15-17 year olds). To the right of the bar graphs that compares perception to reported usage (Consideration 1) is a column that depicts how many teens in the middle or high school overestimated the percent of their peers who are misusing (Consideration 2).

OK…, but where do these data come from? Who participated in this survey?

What is a survey?
Surveys are a research tool developed for the purpose of collecting information about a specific topic from a sample, or small group, of people through a series of specific questions.2 Surveys can be conducted through paper/pencil questionnaires, online forms or by an interview.

What is the RACYS? 
The Rise Above Colorado Youth Survey (RACYS) is a 119-question survey that addresses behavioral health, which is defined as social, emotional and mental health and substance misuse. The survey also examines risk and protective factors for behavioral health. Risk and protective factors are individual, familial or community characteristics that research has shown contribute to either positive or negative health outcomes.3 RACYS has traditionally been conducted by phone by a trained interviewer. However, in 2016, 40% of surveys were completed online because more teens are reachable through online platforms as opposed to phones.

I don’t remember getting surveyed…who participated?
Research panel recruitment, the process used to find teens to participate in the survey, is a business completely focused on ensuring they maintain a large database of diverse participants so that researchers can access their target population. If you haven’t been solicited by a research panel, an adult you know definitely has, especially during election season when everyone wants to make predictions about who is most likely to win.

Surveys can’t reach everyone because of time and money. Plus, it’s not necessary if you chose your sample, or subgroup, to survey correctly. Research panel recruitment businesses help to ensure the survey reaches the correct sample of people. In this case, “correct” most often means two things:

  1. The sample is large enough to conduct statistical analyses with a high degree of precision.
  2. The sample is representative of the whole population relative to certain demographic characteristics (e.g. ethnicity, sex, age).

Is the RACYS sample size large enough?
607 Colorado teens were surveyed in the 2016 RACYS. This may seem small since Colorado has about 397,000 12-17 year olds, but that’s the advantage of statistical analysis. With this sample size, we can be confident that 95% of the time, the results from this small sample can be assumed to be accurate within +/- 4 points for the whole teen population of Colorado.

Is the RACYS representative of Colorado’s teen population?
Most survey research utilizes the U.S. Census Bureau’s data, which is “the leading source of quality data about the nation’s people and economy.”4 Census data is only collected every 10 years, so 2010 data is the most recent. The RACYS sample mirrors the demographic characteristics of Colorado’s 2010 Census data for 12-17 year olds. This means that we purposefully chose participants based on these characteristics so that the final results could be generalized to the state’s teen population. Here’s the breakdown of the RACYS sample:

You’ll see that about 250 participants who participated in the online version of the survey were asked two additional questions about gender and sexual orientation. These were new questions that were tested through the new online data collection method. Of the 250 online participants, 20 (or 8%) self-identified as lesbian, gay, bisexual, transgender and/or queer (LGBTQ). Because 8% is representative of Census estimates for LGBTQ teens, the sensitivity or comprehension of these questions is not a concern and, therefore, will be included in all future RACYS data collection efforts.

Small proportions of the whole sample (607), such as the 20 who identified as LGBTQ, is too small to make any specific statements about the Colorado teen LGBTQ community. Similarly, any of the subpopulations that were less than 5% of the total sample, including the African American, Asian American, Native American and “other” subpopulations as well as the Southwest region, are too small to make any definitive statements about as well. However, this campaign addresses the state as a whole and these small proportions represent the state’s overall population for 12-17 year olds. This allows us to be sure that the statements made about the whole state are inclusive of these smaller subgroups within our state; however, they are averages. Using the average means that while this data reflects the group as a whole, there are individuals and even subgroups – like a small circle of friends, for example – whose responses may be different from the average. Despite the data not reflecting your immediate circle, state averages provide you with a broader and representative perspective of what other teens and communities across the state are experiencing.
Note: in the future, Rise Above Colorado Youth Survey will work to ensure we obtain a large enough sample size for each subpopulation to make credible statements about each.

Still have doubts?
You’re not alone. No data set is 100% accurate, including the US Census. Self-reporting always has limitations because even with the best of intentions, people remember and report on things a little different from what they actually were. For example, some people want to please the interviewer and think they should answer “correctly.” Despite the instructions defining “correct” as truthful, some people still want to please. Other times, people forget or estimate a little more or a little less. And, of course, there’s also the small few who want to “mess the data up” and provide outrageous responses. Often these respondents can be identified and removed from the data set. These limitations hold true for all survey research because we are relying on people’s memory and reporting. This factors into the margin of error, that +/- 4 points above or below the average.

Some ways to know that the RACYS is on point is to compare the data to other similar survey data. For example, the Healthy Kids Colorado Survey examines some of the same questions with a similar population. When we compare the responses, we see similar trends. This is called triangulation, using multiple data sources to confirm the validity of the data.5 Since we see similarities in both data sets, we can have more confidence that both are valid and accurate.

If you still have doubts or are just ready to geek out, here’s the link to the full survey results with the survey protocol beginning on slide 95. If you have more questions, we’d love to continue the conversation with you – please reach out to us on Facebook.

What numbers surprised you the most?

1National Social Norms Center (2016) Social Norms Approach http://socialnorms.org/social-norms-approach/
2Ponto, J. (2015). Understanding and evaluating survey research. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4601897/#A3
3O’Connell, M. E., Boat, T., & Warner, K. E.. (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press; and U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration (2009). Risk and protective factors for mental, emotional, and behavioral disorders across the life cycle. Retrieved from http://dhss.alaska.gov/