SURVEY RESEARCH PRIMER 2.0

         Conducting student, parent, staff, and community surveys will provide information that can be used for at least five - five! - purposes. Think about these applications:

1. Strategic Planning. Surveys can be used to assess an organization's effectiveness and measure important opinions and perceptions. As such they generate benchmark information that will enhance any strategic planning process
   
2. Conducting a North Central Association Self-Study . The NCA accreditation process calls for schools to conduct a self-study and to develop a school profile. Surveys generate the information that you need to complete the self- study and to develop a comprehensive school profile. Surveys can also be used for use as pre- and post-measurements.
   
3. School Improvement. As is the case with strategic planning, surveys can help you identify and assess your priorities in the context of student, parent, staff, and community opinion.
   
4. Marketing. Surveys generate information that will help assure that your communication and marketing initiatives are focused and on target.
   
5. Thinking. No one does enough thinking. The surveys will help you step back from the urgencies of the moment and reflect on what is ... and what can be. In short, they can provide the information that you need to make a difference in your school's educational program. Think about that!

        Everyone wants everyone to do a better job of listening. The reason: listening provides insight and leads to understanding. In fact, that may be reason enough to conduct a representative survey in your school or community.

2. Types of surveys

        There are three basic types of surveys - personal interviews, written surveys, and telephone surveys. Each approach has its advantages and disadvantages.
        Personal interviews are for trained professionals. Interviewers meet with people one-on-one to obtain their responses. Interviews are typically conducted in the respondent's home or place of business.
        The immediate advantage of personal interviews is that they can be quite lengthy. Trained professionals can also probe to enhance responses, and they can note things about the environment in which the interview is conducted (e.g., the condition of the furniture, the respondent's attire, the number and kinds of interruptions during the interview, what kinds of pets are in the house, etc.).
        The downside of personal interviews is that they are quite expensive, out of the price range of most organizations.
        Written surveys seem easiest to do. The tendency, however, is to write difficult questions and/or too many questions. When this happens, the response rate dives, and when the response rate dives your respondents tend not be representative of the population that you are surveying.
        Written surveys that are mailed to respondents also take time - time to mail out questionnaires, time for people to respond, time for respondents to return their completed questionnaires. This disadvantage can be negated or minimized, however, by distributing and collecting written questionnaires at a meeting or event, or by providing respondents with an on-line response option.
        Surveys can also be conducted by telephone. The advantage of a telephone survey is that you can quickly reach large numbers of people across a wide geographic area. But telephone surveys have to be short. (Except for teenagers, a minute is a long time on the telephone!) To conduct a telephone survey you'll also need to train (and probably pay) interviewers. And, you'll need a physical location from which to make the calls.
        The type of survey that you decide to use will depend on several factors, including the target audience that you're trying to reach, your timeline, and your budget.

3. The essential steps in conducting a survey

        As we indicate in our book (The ABC Complete Book of School Surveys from Scarecrow Press) there can be up to eighteen essential steps in the survey process:

1. Decide what you want to learn from the survey

2. Determine if you really need to do a survey to obtain the information you need

3. Determine the target audience for the survey

4. Determine the type of survey method

5. Determine the level of confidence

6. Develop a survey timeline

7. List the resources needed

8. Draw the sample

9. Outline the content areas and draft the questions

10. Refine the wording and design the questionnaire format

11. Pretest the questionnaire

12. Develop the final questionnaire

13. Recruit and train interviewers (personal and telephone interviews)

14. Administer the survey

15. Tabulate the data

16. Analyze the results

17. Report the findings

18. Use the results

4. What are confidence levels?

        Don't think like this: "Well, we have about 300 parents, so if I survey 100 of them that should do it."
        Let's start with an explanation of confidence levels, and then a common sense approach to surveying.
        Most everyone has read a news media account of a survey that mentions sampling error (Sampling error is usually reported at the end of the story like this: "The survey has a 5% margin of error.")
        But you need two numbers to determine the accuracy of a survey: The confidence level and the sampling error. For example, you should know that the survey has a confidence level of 95% with a sampling error of +/- 5%.
        Here's what this means: If the survey were administered again to the same sample, 95 times out of 100, at the 50:50 split, the results would be 5% either way from those reported.
        As an example, if a yes-no question results in 50% of the respondents answering yes and 50% of the respondents answering no, then you can be 95% certain that if the survey were administered again to the same population, the yes and no responses would fall between 45% and 55% (+/- 5%).
        By the way, the 50:50 split is illustrated in the above example. Here respondents split evenly (50:50) given two choices. This is the least accurate a properly conducted survey will be. If, for example, the results were 80% yes and 20% no, the sampling error would be less ... maybe three or four percent for that question. As a rule of thumb, the more distant responses are from the 50:50 split, the more confidence you can have in the survey results.

5. Sample size … and common sense

        Below is a sample size chart for 95% confidence with a +/- 5% sampling error for selected universe sizes. (95 +/-5 is often regarded as "the industry standard" for surveys of the type we're discussing.)
        If you want to survey your staff and it has 20 members (the universe), you need to survey 19 staff members to attain confidence of 95% with a +/-5% sampling error.
        If you have 300 parents (the universe), you need to sample 169 of them. And, if you have a million students (the universe), you have to sample 384 students.
        Look at the numbers. The number of people that you should survey is not based on a flat percentage (like 10% of students, parents, and staff). Consider the examples above: If your universe of staff members is twenty, you have to sample 95% of the staff to attain 95+/-5. If your universe of parents is 300, you have to sample 169 or 56% of them. And if you have a million students, you have to sample 384 or .04% of them. The number of people to survey is based on a statistical formula, not a flat percentage.

UNIVERSE   10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 110 120 130 140 150 160 170 180 190 200 210 220 230 240 250 300 400 500 600 700 800 900 1,000 2,000 3,000 4,000 5,000 50,000 1,000,000

NUMBER TO SURVEY (for 95% confidence with a +/-5% sampling error) 10 14 19 24 28 32 36 40 44 48 52 56 59 63 66 70 73 86 80 86 92 97 103 108 113 118 123 127 132 136 140 144 148 152 169 196 217 234 248 260 269 278 322 341 351 357 381 384

        Now let’s deal with political reality and some common sense. If you have 20 staff members, should you do a sampling of 19 or give the survey to everyone? Give it to everyone or you’ll spend the rest of your life explaining how you drew the sample and the process you used to eliminate one person.

         What if you have 300 parents ... should you survey them all or do a sampling? It depends. The easy answer to the question is, “Do a sampling.” But if your parent community believes that “everyone should have a chance to respond,” give a copy of the survey to every family. (Note that I changed the universe on you ... from parents to families. Do you really want a sampling of parents or a sampling of families?)

6. Increasing your response rate

        Response rates are important. After all, to have a 95% level of confidence with a +/- 5% sampling error from a universe of 300, you must have 169 completed surveys.
        So, how do you get people to respond? Students are easiest because they are "captive." You administer and collect the survey in school.
        To get higher returns from parents, publicize the importance of the survey in advance. Hand-sign the cover letter (in blue ink). Create contests. (Provide pizza for the classroom with the highest response rate!) Or distribute and collect the surveys at school events that draw a representative sampling of the population. (Careful here. The science fair, the art fair, or the musical may only draw people who are interested in science, art, or music. If that’s the case, find another event. On the other hand, some schools have 90-plus percent participation in parent conferences. If that’s the case in your school, distribute and collect the questionnaires at parent conference time.)
        Staff members sometimes hesitate to respond because they think that they will be identified or that their responses will not remain confidential. Reassure them that their responses will remain confidential (and make sure that they do remain confidential). Give them a plain envelope in which to return their survey. Put a drop box in the staff workroom instead of the office. Or give staff members a postage-paid envelope that is preaddressed to a survey processing organization.


7. Building your own survey

        You can build your own survey by addressing each step in the survey process (Refer back to Section 3 above.)
        Begin with step one in the survey process: Decide what you want to learn from the survey
        In effect, this step asks you to identify what you want to know. (It may be the most difficult step in the survey research process!)
        The next section of our primer presents categories of questions from our NCA community survey. They’re presented here to help you address Step 1 in the survey process. For example, you may want to know about demographics, perceptions of quality, and communication effectiveness.
        If that’s the case, you should begin by looking through the DEMOGRAPHICS category and selecting questions that will generate the information that you want. For example, you’ll probably want to know if respondents are parents of school-age children. If that’s the case, you should ask: Do you have school-age children?
        This question will enable you to view the responses of parents and respondents without school-age children. This question also enables you to view the responses of parents and nonparents to any (or all) of the questions in the questionnaire.
        You may also want to ask parent respondents the grade level(s) of their children. This enables you to view the responses of parents who have children at a given grade level (or the responses of all parents at each grade level).
        Once you identify the demographic questions that you want, you should follow the same process with questions in the PERCEPTIONS OF QUALITY, COMMUNICATION EFFECTIVENESS, and other categories.
        Then create a file to which you can transport the questions that you’ve selected. When this task is completed you’ll have a first (rough!) draft of your questionnaire.
        Of course, there are probably areas of inquiry that you’d like to address which aren’t covered by the questions in this document. That puts you in the position of having to create your own questions.
        Here are a couple of things to keep in mind:

1.    2.   3.   4.         5.       6.

Make sure your questions are clear and easy to understand. (That’s why pre-testing the questionnaire is important. Avoid jargon and acronyms (e.g., Do you think NCLB is a good idea or not?) Avoid combining two thoughts (or questions) in a single question. (e.g., Do you think that our reading program is effective and appeals to students? ) Watch the placement of questions. The open-end question above (If you could set one priority for our school district, what would that one priority be?) is a great question, but a bad opener. Start your questionnaire with this question and respondents will head for the shredder. Here’s a rule of thumb: Don’t lead with an open-end question. Start with a simple questions (e.g., Do you have children of school-age or younger?) End with the more difficult questions. Do not bias or “box in” the respondent. (Questionnaires from some elected officials provide good examples of what not to do: Everyone supports lower taxes. What about you … are you in favor of lower taxes or not? Another example of what not to do: Do you think that convicted felons should be locked up or left free to roam our streets and neighborhoods?) Question sequence should flow, from the general to the specific, from the present to the past (or from the present to the future).

8. Types of questions

        This section of our primer contains some sample questions from the community survey that we designed for NCA. Our purpose here is to show you some properly worded questions and some questioning techniques. You’ll find questions in the following categories:

            ■ Demographics
            ■ Perceptions of Quality
            ■ Communication Effectiveness
            ■ Importance of School Programs/Services
            ■ Agree-Disagree Scales
 

        Demographic questions are typically used to classify responses to a survey. For example, you may want to know if males and females hold the same opinions (or to what extent their opinions differ) on a given question (or all questions). Or, you may want to know if parents get their school information from different sources than nonparents. Or, you may want to know if people who have lived in your community more than ten years perceive the quality of your schools differently than newcomers. The questions below will help you develop such classifications.

        Are you …
                    [ ] male
                    [ ] female


        How many years have you lived in our school district?
                    [ ] less than one year
                    [ ] 1-3 years
                    [ ] 4-9 years
                    [ ] 10 or more years

        Do you have children of school age or younger?
                    [ ] yes
                    [ ] no

        What grades are they in? (CHECK ALL THAT APPLY)
                    [ ] preschool
                    [ ] elementary (grades Kindergarten – 5)
                    [ ] junior high/middle school (grades 6-9)
                    [ ] high school (grades 10-12)

        What is your age?
                    [ ] 18 – 24
                    [ ] 25-34
                    [ ] 35-44
                    [ ] 45-54
                    [ ] 55-64
                    [ ] 65 or older
 

        The following questions are designed to assess respondent perceptions of quality, generally and specifically.
        The first question provides a general rating (“all things considered”), and is comparable to the annual PDK/Gallup Poll of Public Attitudes Toward the Public Schools.
        Subsequent questions in this category measure specific perceptions of quality.

        Students are often given the grades of A, B, C, D, or F (FAIL) to indicate how well they are doing in school. Please use the same grades to tell us how well we’re doing. All things considered, what grade would you give our school district, an A, B, C, D, or F?

                    [ ] A
                    [ ] B
                    [ ] C
                    [ ] D
                    [ ] F
                    [ ] Can’t say


        Below are examples of how the same A, B, C, D, F scale can be used to assess perceptions in specific areas.

 

       Of course, it would be easy to get too specific and ask respondents to grade a long list of programs and services. For example, we could ask respondents to grade the lunch program, foreign language offerings, student transportation, and a host of other items. In this regard, there are two things to keep in mind: 1) what you want to know from the survey (Step 1 in the survey process!); and, 2) the more specific that you get, the more time it will take respondents to complete your survey. And, while you are measuring perceptions, respondents may “bail out” of your survey if you ask them a long series of detailed questions about which they have little knowledge.

        Here is another way to use the grading scale to assess perceptions of school programming. (Incidentally, these are the areas that you should address in parent and student surveys as part of your North Central Association accreditation process.)

to read?   to write?    to understand science?  to do basic math (adding, subtracting, multiplying, dividing)?   to solve math problems?  to understand social studies?   to be healthy and physically fit?   to learn skills needed for the world of work? to learn about jobs and careers? to use computers and other technology? to think?

Grade   _____   _____   _____ _____ _____ _____ _____ _____ _____ _____ _____

       Or you could ask respondents to identify the indicators of a good school by using a question like this:

        Here are some things that people say are indicators of a good school or a quality education. After reading the list, check the two that you think are the best indicators.

                    [ ] competent teachers
                    [ ] caring staff
                    [ ] number of students going on to postsecondary education
                    [ ] high standards and expectations
                    [ ] good test scores
                    [ ] low dropout rate
                    [ ] parent/community involvement in the educational program
                    [ ] students learn the basics (reading, writing, arithmetic)
                    [ ] comprehensive curriculum
                    [ ] buildings and facilities are attractive and well-maintained
                    [ ] quality leadership
                    [ ] other ____________________________________

        This question could be followed by a question that asks how well respondents think you are doing in the areas that they selected, for example …

        Now use the A, B, C, D, F rating scale to tell us how well we’re doing in each of the two areas that you identified above.

                                                                                                                                              Grade

                                                                The 1st area that you identified                          _____
                                                                The 2nd area that you identified                         _____

        The following questions are related to communication. They are designed to identify sources of school information, information needs, and communication effectiveness.

        Where do you get your school information? (CHECK ALL THAT APPLY) 

                   
                    [ ] district publications
                    [ ] school publications
                    [ ] communications from classroom teachers
                    [ ] newspapers
                    [ ] radio/TV
                    [ ] my children/students
                    [ ] friends / neighbors
                    [ ] other _________________________

        What types of school information interest you? (CHECK ALL THAT APPLY)

                    [ ] none
                    [ ] information about the curriculum / what students are learning
                    [ ] finance information
                    [ ] information about school teachers /staff
                    [ ] information about goals and priorities
                    [ ] test scores
                    [ ] other __________________________

        How well informed do you feel about what’s happening in our schools … would you say that you are very well information, reasonable well informed, or not too well informed?

                    [ ] very well informed
                    [ ] reasonably well informed
                    [ ] not too well informed
                    [ ] can’t say

        Agree-Disagree scales enable you to assess perceptions on a variety of topics. One benefit of this question format is that respondents can quickly respond to a relatively large number of items. The section below illustrates this question format.
        Note that the statements can focus on a single topical area (such as demographics) or a variety of areas (as is the case here).

---

_____ I feel reasonably well informed about what’s going on in our school district. _____ Our school district is responsive to citizen questions and concerns. _____ The curriculum is more challenging today than it was when I went to school. _____ The educational program is better today than when I went to school. _____ Learning expectations for students are reasonable in our school district. _____ Generally speaking, our school district is a safe and orderly place. _____ Our school district does a good job of teaching basic subjects (like reading and writing). _____ Our schools are clean and well maintained. _____ People in our community respect teachers and other school employees. _____ Our school district is a source of community pride. _____ Citizens are well informed of school goals, policies and practices. _____ The instructional program provides students with skills they need to be successful. _____ Our school district strives to meet the learning needs of all students. _____ Staff members in our school district are courteous to people in the community. _____ Teachers in our school district appear dedicated and enthusiastic.

        Below are some areas that are important to program planning (and should be attended to as part of the NCA accreditation process). Here we use a three-point “importance scale” to ask respondents the importance of each item.

        Note that with a different introduction, the questions above can be used to determine respondent perceptions of the quality of the job being done in each area. For example, we could say …

        These questions let respondents express their opinions without condition.
        We like open-ended (or free-response) questions for two basic reasons: First, they provide valuable information and insights. Second, they “open up” the survey, allowing respondents to get outside any box that you may have inadvertently created as you designed your questionnaire. For example, your questionnaire may not address busing, but “student transportation” may emerge as something people really like. Or, your strategic plan may be focused on reading while parents say “better playgrounds” are their most important priority. In both cases, you’ve learned something important … for planning, for communication, for discussion.

        Is there a school program or service that you’d like to know more about?



        Is there anything about our schools that you particularly dislike?



        In your opinion, what is the single best thing about our school district?



        If you could set one priority for our school district, what would that one priority be?



9. BB&C’s questionnaires: Saving yourself time and money


        Of course we’re biased. But here are some reasons that you should consider using Banach, Banach & Cassidy’s NCA-endorsed surveys:

1. They will save you time and money.

2. They have been professionally developed.

3. They have been pretested.

4. They are being continuously normed and allow you to make comparisons

5. They can provide comprehensive insights into opinions and perceptions.

6. They will generate information you need to conduct an NCA self-study and develop the survey portion of your school profile.

        Think for a moment about an alternative -- developing your own survey.
        Developing your own survey will probably require convening a planning committee to determine what you want to know, conducting a review of the North Central Association’s process and profile recommendations, tapping someone to draft the questionnaire (and develop subsequent redrafts), pretesting the questionnaire, and – finally – developing the final instrument. This can take a substantial amount of staff and volunteer time … and endless meetings. Then you’ll have to repeat the process for each target audience that you plan to survey.
        In the end, odds are very good that you’ll wind up with a questionnaire that lacks validity, isn’t normed, and doesn’t enable you to make comparisons between student, staff, and parent responses. You’ll also find that you’ve wasted precious time, energy, and money.

        We are continually developing surveys to address the strategic planning, accreditation, and marketing needs of schools and school districts. There are four surveys for use at the school building level and one survey – the community survey – for use at the district level (to address the requirements of NCA district accreditation).

1. Parent Survey
2. Staff Survey
3. Student Survey I
4. Student Survey II
5. Community Survey

        Student Survey I is designed for students in grades 3-5. Student Survey II is designed for students in grades 6-12. If you order a student survey, we send you both versions at the single survey price. (Some of our client schools are K-12 buildings and use both surveys.)

        The Parent, Student II, and Staff surveys are designed to be administered in written form. The Student I survey (for grades 3-5) is designed to be administered by a classroom teacher or another responsible adult. Students are provided with an answer sheet on which to record their responses.
        The Community Survey is designed to be administered as a written survey.

        The building surveys are $75 each or all four for $160. Each survey that you purchase can be used twice by your school (e.g., as a benchmark survey and then as a follow-up in two or three years ... or at the conclusion of the accreditation process).
        Each of the building surveys is registered to a school (not a school district), and may be used only by that school.
        The Community Survey is also $75. It’s designed for administration across a school district, and, like the other surveys, may be used twice by the school district.

        We charge 95 cents per questionnaire (with a $250 minimum charge) to computer process your data (including the written responses to the open-end or free response questions).
        We do not charge a minimum of $250 for each survey that you send us.  Rather, we invoice you for the grand total number of questionnaires tabulated.
        When the tabulation is complete, we provide you with a printout of the numerical and percentage responses for each question. 
        If you want cross-tabulations of the data, they are $50 each.