Short answer: They deserve weight, but how much depends on what we are claiming with each finding. Not all findings carry the same weight, and our work has not used them all in the same way.
One thing needs to be very clear. If the question is “exactly what percentage of people in Bangladesh think this?”, this survey was not designed to answer it. So when 90% of respondents give a view, we do not say, “90% of people in Bangladesh think this.” We say, “90% of our 548 participants identified this issue.” The book also states clearly that this is not a statistically representative national sample.
But if the question is “which problems keep coming up in the field of dawah, where are the important gaps and which issues demand attention?”, these findings have considerable value, especially when the same kinds of issues also emerge, alongside the survey, in field observation, workshops, stakeholder consultation and several rounds of review. The foundation of our blueprint is not the survey alone but the combination of these multiple sources.
Survey methodology, too, does not automatically treat a non-probability sample as “useless”. The main caution is that its results must not be inflated into population estimates, and the sampling method and limitations must be disclosed clearly. AAPOR’s disclosure standards emphasise exactly this transparency and limited inference.
Another important point is that a large sample is not automatically representative, and a non-representative sample does not make the information worthless. Sample design has to be matched to the research question. Our question was not national polling; it was to identify gaps, recurring concerns and possible areas of improvement in dawah.
Moreover, 72 recommendations were not announced straight from the survey. The issues that emerged from the survey were taken through workshops, field experience, input from people in various professions and fields, and successive reviews, to develop possible actions and models. So here the survey is an input, not a verdict; the recommendations are a synthesis of multiple inputs.
That is why our position is very simple:
Where the question is about exact national percentages, we acknowledge the limitations. But where the task is to identify recurring gaps, develop possible solutions and bring forward ideas that can be applied in practice, this survey of 548 people is a meaningful evidence base, especially as it has been checked against workshops, expert review and field experience.
Put most briefly, you can say from the stage:
“We did not use the findings as a final measure of national public opinion. We took them as recurring signals, then combined them with workshops, field insight and review to develop actions. So the survey is not our last word, but it is certainly an important evidence base.”
Helpful examples:
Example 1: bigger numbers are not necessarily right, and smaller numbers are not necessarily wrong
United States, 1936. The Literary Digest polled about 2.3 million people and announced that Roosevelt would lose. At the same time, Gallup polled only about fifty thousand people and said Roosevelt would win. Roosevelt won by a huge margin, and the magazine closed not long afterwards.
Two lessons:
The size of a sample is not the measure of its reliability. If someone says “548 is too few”, the question is aimed at the wrong place.
The real question is whether the type of sample matches the type of question. The Digest went wrong because it tried to make a prediction with an unsuitable sample. We are not making a prediction; we are making a diagnosis.
Example 2: there is bias, but we know which way it points
The Second World War, United States. The military saw that planes returning from combat had more bullet holes in their wings and tails, and decided to add armour to those areas. The mathematician Abraham Wald stopped them. His reasoning: this sample contains only the planes that came back; the ones that did not return are missing. That means the planes hit in the engine were the ones that went down. Armour had to go exactly where there were no holes. Following his advice saved countless lives.
The key lesson: even a biased sample is useful if you know which way the bias points.
In our case, the direction is clear. In an open survey, those who respond are usually the comparatively more active, more interested and more involved. If even these relatively active people speak of gaps in training, coordination and continuity, the situation of those less involved is likely to be worse still. In other words, our error leans towards understating the problem, not overstating it. It is a conservative error, not a dangerous one.