Introduction
Should the numbers from a clinical trial be trusted? Does the context in which knowledge from a randomised clinical trial is produced affect the results? These are relevant questions, because while statistics – and thus numbers – offer a powerful tool for presenting evidence and governing (Desrosières, 2014), the social conditions of their production get very little attention.
The aim of any clinical trial is to produce reliable data. Those data are obtained via a complex, meticulously kept medical record based on carefully completed forms. A trial participant’s medical record refers to a medical practice, one which is fed and maintained by scientific experimentation. This article will discuss the process through which data are produced and multiple ways of making them reliable in a clinical trial. The study in question was a clinical trial for a new vaccine against severe forms of rotavirus diarrhoea in children under 5; the vaccine is manufactured by the Serum Institute of India Ltd., a pharmaceutical firm based in India, the world’s largest supplier of generic drugs (Bertho-Huidal, 2012; Lefranc, 2015; Sunder Rajan, 2017). The trial was conducted in Madarounfa, Niger over a four-year period, beginning with the randomisation of children in their first weeks of life and continuing through the two or more years of follow-up needed to evaluate the safety and efficacy of a new vaccine. The new drug, at least on the surface, appeared to solve some of the environmental and structural constraints hampering health systems in sub-Saharan Africa. Described as more tolerant of ambient temperatures, the vaccine does not require a continuous cold chain, making it less burdensome logistically in terms of packaging, and it would also be less expensive.
Epicentre, an epidemiological research centre belonging to Médecins Sans Frontières (MSF) which is based in Paris, chose Niger because it has one of its two satellite offices in Africa (the second satellite office is in Uganda), with a well-equipped laboratory for conducting its research in collaboration with locally recruited primarily Nigerien staff and expatriates. Epicentre was created in 1986 to guide the organisation in certain contexts of its humanitarian action. So beyond the compassionate action that has historically characterised MSF as a humanitarian non-governmental organisation (NGO) (Fassin, 2010; Likin, 2009; Redfield, 2005), creating an in-house research institution has helped legitimate that action via evidence-based medicine. Epicentre conducted the trial from 2015 to 2017, and the vaccine which seemed to offer a solution to the logistical, environmental and financial constraints faced by the health systems in sub-Saharan Africa met MSF’s commitment to improving developing countries’ access to essential medicines (Mfuka, 2002).
Based on an ethnographic survey that used a combination of participant observations and interviews to provide real-time monitoring of the study by an anthropologist advising the study, this article falls under the rubric of Science and Technology Studies (STS), which describes ‘science in the making’ (Callon, 1986, 2003; Latour and Woolgar, 2006; Pestre, 2010). It shows – taking in consideration the difficulties inherent in real life – the problems of a clinical trial that failed to consider fully the local context and had to deal with unexpected consequences.1
This article describes and analyses the tensions created by flaws in the system used to test a treatment. It shows the consequences for a clinical trial in which a set of practices was carefully monitored and closely supervised by a medical profession-specific bureaucracy and hierarchy designed to minimise bias and to produce reliable data, but those practices were accompanied by various events that created uncertainty. The nature of that uncertainty took several different forms: there were technical flaws in the trial mechanism itself, where the staff members’ routine practices became the practical norms (Olivier de Sardan et al., 2017), and also problems due to the complicated nature of collaboration with the population. The Madarounfa clinical trial relied on the numbers. The investigators’ excessive faith in the questionnaires from which they got the numbers eclipsed the real conditions in which the data were produced, inputted, or processed. This explains the difficulties of a trial that seemed far removed from such conditions and suffered the revenge of (the local) contexts in which it was conducted (Olivier de Sardan, 2021). It also explains our anthropological approach, which documents the scientists’ actual working conditions in ethnographic form.
The Medical Record: A Medical Quantification Tool
A randomised clinical trial tests a new drug or vaccine in order to evaluate its efficacy in the human body, while taking all possible safety precautions, to make sure it is effective prior to marketing approval, which is itself based on a set of well-regulated procedures (Brives, 2012; Couderc, 2011; Lakoff, 2007; Marks, 1999; Petryna, 2009). Trial participants are tracked and followed medically via a medical record. In modern medicine the medical record is essential for producing various types of data about the body (Berg and Bowker, 1997). It documents the course of a disease and other aspects of the organisational life of medicine (Strauss, 1992). It contains medical practices and the professional development and career of practitioners. A medical record starts with the patient’s admission. It describes the diagnostic process and all the diagnostic procedures. It details the itinerary of the care received from admission to resolution (which might be cure, death or discontinuation of care). If constituting a medical record in the context of a clinical trial is a lengthy, complex and rigorous process, it is because all the actions chronicling the participant’s health and disease history must be clearly documented in accordance with specific protocols and ethical commitments.
That rigour and complexity lie at the very heart of a clinical trial, which is first and foremost a scientific activity. According to Latour, scientific activity is a process of trace production and inscription. It creates what he calls the chains of reference (Latour, 2012), which consist of working on forms superposed one onto another. In clinical trials, the process of validating participant records and encoding, refining and entering the collected data results in many medical inscriptions in a record. In the present case, that knowledge concerned administration of the vaccine being tested, its adverse effects (serious or not), doctor visits, nursing care, home visits, lab analysis of the samples collected, verbal autopsies, etc.
According to Berg and Bowker (1997), that information plays a major role in the production of a body politic. It describes the clinical work at the facility to which the patient goes for care. It can also be used to retrace and understand the history of the trial’s practices. It is thanks to those inscriptions that data on the vaccine’s efficacy can be and are used. They also help provide a variety of epidemiological statistics and information about the most pressing health needs of the trial’s catchment area. In addition, entering data and information into a medical record produces a history of the participant’s illnesses during the trial via chronological data. Also, thanks to the strict procedures for validating and cross-validating forms, the record shows how the medical bureaucracy operates. The validation process follows a medical hierarchy, with clearly detailed daily, weekly, monthly, etc. sequences. Entering data into a computer database requires approval by the investigator, who is the sole legal authority and whose decision is incontrovertible. In short, knowledge about the clinical and experimental practices of biomedicine depends on writing, and on inscriptions that represent lots of traces left behind. In a clinical trial like Epicentre’s, which evaluated a new vaccine against severe forms of diarrhoea in thousands of children under the age of five, the process of producing quantitative data was a veritable machine of data refinement, entry and storage.
The Materiality of the Madarounfa Clinical Trial
Epicentre’s randomised clinical trial was funded by MSF Switzerland. There was a parent study and a sub-study. The first evaluated the safety and efficacy of a new vaccine called Rotasiil, manufactured by the Serum Institute of India Ltd, against severe forms of gastroenteritis in children under age two years. The second evaluated the children’s immune response to the new vaccine when boosted by prenatal lipid-based supplements.
The trial data were painstakingly collected, recorded and entered. To do that, Epicentre deployed the necessary logistical and human resources in an area that was relatively inaccessible due to nearly non-existent transportation infrastructure and rural roads that are impassable during the rainy season. In addition to the people and things involved in producing the knowledge from the Madarounfa trial (the vaccine, lab samples, lab, participants, staff, cars and medications) another entity played a key role in the system and made the trial a reality – namely the forms. They represented the materiality of the trial and recorded the tangible evidence of its existence and conduct. The forms made the trial exist in the sense that all its activities were documented on them.
The size of a trial can be measured by the forms and the information they contain. The Madarounfa trial used thirty different forms for the parent study and thirty-six different forms for the sub-study, not counting numerous non-numbered data sheets and four different informed consent templates for the participants. They varied from one to eighteen pages in length, depending on the type of data sought. From randomisation to discharge from the parent study, for example, a given participant’s medical record contained at least 147 completed forms, not counting any non-protocol visits due to illness, in which case the number was even higher. In other words, the more a participant visited the health centre, the more of that person’s treatment journey was recorded. A participant’s medical record could easily reach 190 questionnaires with Form 9A (weekly home visit), a form that was filled out 104 times, at a rate of once per week over the two years of follow-up. So with 7,770 children and 3,000 pregnant women, each with their own remarkably accurately archived medical record, the logistical capacity needed to store those records required a lot of space and tons of numbered, coded files. That is why the cabinets and boxes in all the sites’ records rooms were full. All of the empty rooms at the study sites were rearranged several times to increase capacity. Sometimes, when medical records would arrive at the data-entry department by the hundreds, before being entered and refiled, they would be stacked on the floor for lack of space. A local health centre in Maradi – the major city where Epicentre had most of its medical research infrastructure – offered a room that the organisation refitted to free up space in the usual record storage areas (Figure 1).
Records room at one of the five study sites. The forms were stored in their respective record folders before being sent for data entry. Photo: Author.
Citation: Journal of Humanitarian Affairs 5, 1; 10.7227/JHA.099
The Foundation of Data Management: A Hierarchical Data-Entry Department
Several types of samples (blood, stool, urine, breast milk, etc.) were collected from the children and their mothers and transported to a testing laboratory. The samples collected from the children were logged on Form 17A before being sent to the laboratory. The results of the various tests were entered on Form 17B. For the sub-study, the children’s samples were logged on Form S17A and their results on Form S17C. Forms S17B and S17D were for the mothers’ samples and results, respectively. Each form was filled out at the laboratory in duplicate, with an original and a carbon copy. The duplicates were logged and kept at the lab, and the two originals were stapled together and sent to the data-entry department. To those forms were added others, filled out in the field.
In a different building than the research centre administration and its laboratory, the data-entry department was the ideal place to have all data from Epicentre’s epidemiological studies in Niger entered. In late 2015, there were some forty data-entry operators (‘Ops’) under contract, not counting several dozen operators with daily contracts. Unlike the lab, ‘Data’ (as the operators usually called it) was a very hierarchical department with highly specialised tasks. At the top, a data manager – aided by two assistants – led the department. They worked under the direction of an expatriate who managed all the organisation’s databases, the network and the IT infrastructure. He personally received instructions from the advisor in Paris, like all the expatriates leading the centre’s teams. The data manager and his assistant reported directly to the controllers, who acted as intermediaries with the data-entry operators at the bottom rung of the ladder. The Ops’ job was to faithfully enter the data from the forms filled out in the field, at the lab or by the monitors. To ensure the security of those data, strict standards were set out right from the time of their job interview. Those standards applied to the confidentiality of the data, which could not be taken outside, the prohibition against using USB drives or connecting to the internet on work computers, etc. In the words of the data manager,
Study data aren’t shared, because they are sensitive. They are also confidential. These are standards wherever clinical trials are done. It’s a loss for Epicentre if someone can get access and analyse them. That’s why we take strict measures. Even the USB drives can be used only for work. (Data manager, male, age 41, interviewed on 27 November 2016 at the data-entry office in Maradi)
The number of controllers varied from three to seven, depending on the amount of data the data-entry operators had to enter. Their duties might be reversed, depending on need. The controllers controlled how the case report forms (CRFs) (the forms) were received and how participant records flowed from the study sites to the data-entry office (Figure 2). They also got the records for any diarrhoea cases, which were filed and entered automatically as with a medical emergency. The controllers were also responsible for setting up the input masks used for entering data from the forms. They determined the number of forms that each Op had to enter per day. Backups were performed daily or weekly, depending on the amount of data entered in the mask. To do this, the controllers retrieved the data from each machine and consolidated them into a single data file before backing them up on a special server used only for storage. In addition, the records clerks who used scheduling software to arrange the different nursing assistant and supervisor field visits (weekly, monthly and quarterly visits) would occasionally get overwhelmed or lost in monitoring and verifying the physical records. It was the controllers’ job to help them. Every Friday, the records clerks sent the controllers a weekly activity report via a data collection sheet.
Data-entry operator arranging already-entered forms. He would first arrange the forms by order of arrival. The controller would tell him which forms to process first. Forms whose data had been entered were filed by order of exit and stored on the shelves to the right. Photo: Author.
Citation: Journal of Humanitarian Affairs 5, 1; 10.7227/JHA.099
Form Validation Procedures: Colour-Coded Monitoring
A participant’s medical record would fill up gradually. The information was managed using a colour-code system. The forms that made up a record were used in various datasheets, depending on the information and data to be collected. Everyone from the field investigator to the community health agent living at the edge of a village involved in the trial – including the doctor, lab tech, midwife, supervisor and nurse – helped collect information, depending on their status and their role in the system and in the procedures for filling out forms.
In the area where Epicentre conducted the trial, there were a total of five centres where newborns and pregnant women were enrolled. These were called randomisation or enrolment sites. Every day, the forms filled out there were taken to the monitor or investigator for signature, ensuring their accuracy and reliability. Small stickers were used to indicate the progress of each form. The colour of each sticker indicated the progress of the form and the validity of the data on it. The colour-code system reflected both the bureaucracy of the trial itself and the hierarchy of the medical profession (Freidson, 1984). Like the signal lights used to regulate transport, the coloured stickers attached to the forms indicated whether or not they were ready to move on to the next step or be entered into the database. At the bottom of the hierarchy were the community health agents, the nursing assistants and the nurses. They had no authority to validate a document. They filled out the participants’ follow-up forms and sent them to their superiors for observation, being careful to initial them.
The very first validation was done by the nurse-supervisors, who applied a yellow sticker if the document was filled out correctly. Otherwise, the document was sent back to its initial user, without a sticker, for correction. Next came the pink sticker used by the site doctor to validate, in turn, the supervisors’ validation and the forms recorded by the nursing assistant and the nurse. The final step was the monitor’s verification of the forms in each medical record. He used a green sticker, without which a form could not be entered in the database. Thus ended the record’s first physical journey.
Records filled out at the lab did not require stickers because there were only two levels of validation. The lab technicians only had to initial the forms to attest to the veracity of the information. Validation was done by the lab supervisor. By initialling he confirmed that the lab technician followed the lab test procedures and that the forms were filled out correctly. After that, the head of the team also initialled to validate the lab supervisor’s comments. Sometimes, the monitor was unable to verify the increasingly numerous forms or was delayed by the constant back-and-forth between the field and the study office. If that happened, it was done by the field investigator, who had the final say on all field activities unless decided otherwise by his advisor at MSF headquarters in Paris, where the principal investigator was headquartered.
Despite this hierarchical structure, the validation process ran into difficulties. The biggest problem had to do with the staff members’ level of training and comprehension. Second, they failed to master the study procedures. That necessitated a series of trainings and skill-building for hundreds of people deployed in the field. After that came the lack of rigour and motivation in following those procedures. That slowed down activities and negatively impacted the reliability of the data.
I worked more than six months without a break due to all the records that had to be validated. Sometimes I stayed at the office until 11 at night, including weekends and holidays. The workers act in bad faith. The doctors say they’re busy. That’s why they’re bad at validating. With the nurse-supervisors, it’s even worse. They don’t even look at what the nursing assistants wrote, they just validate, and don’t go to the field to verify follow-up. They don’t look at anything they’ve written. What’s more, they don’t even understand what they’re validating. And it’s up to us to correct their mix-ups. (Medical monitor, male, age 35, interviewed on 15 February 2016 in Maradi)
A doctor’s validation of a document did not mean it was duly filled out. The doctor may have skipped over certain parts or hurried through the corrections in order to deal with other activities. He may also have validated the form without really looking through it. After verification, the monitor was required to send it for corrections. If those were incorporated and reviewed, he applied a green sticker. Sometimes, however, the participant’s social life or just reality made data impossible to recover. For example, samples could not be collected from a pregnant woman who travelled to her parents’ home to give birth. Nor was it possible to know a child’s exact weight more than three months after a visit to the study site. Yet boxes on a trial form could not be left empty. So the staff made up their own data, inventing approximate numbers to ensure that the forms were filled out.
Moreover, a green sticker did not mean that corrections were finished. Instead, it marked the end of one stage in the form validation procedure. No matter how rigorous the monitors’ efforts to make the information reliable, some inevitably escaped their attention. But the software used to ensure that data was entered in the database made no allowance for doubtful responses on forms, unless it was poorly designed, in terms of error reporting, by the data manager. As one can see, the purpose of using a hybrid hierarchical bureaucracy combining scientific research and the medical hierarchy for validating forms was to minimise doubt.
The concept of monitoring – an expression dear to investigators – is interesting here. It makes understanding the surveillance, control and flow of data from the trial – whose structure and practices were just an engine for producing data – possible. It was also an engine for monitoring how forms were filled out and staff activities at various levels using coloured stickers. Yet did following those formal rules for control and supervision help eliminate uncertainty and doubt? That is, did that type of monitoring ensure that the data was reliable?
Reducing Uncertainty in the Production of Reliable Data
To leave no room for doubt, the data-entry department had several safeguards in place. The first was designing an input mask. An input mask is a software programme for entering and storing raw quantitative data. The data manager designed it using EpiData software, which was developed by Danish scientists to help in creating electronic versions of survey questionnaires and facilitate data entry (Lauritsen et al., 2001). EpiData allows users to define their own validation rules for each of the variables in a questionnaire.
After the data manager or controller designed a questionnaire’s input mask, it was tested by at least three different Ops and sent back for correction. In addition, each form’s content was encoded with the site, village and participant enrolment order. The five enrolment sites were coded 1 through 5. For example, the third site to begin study enrolment was coded as 3. Likewise for the villages covered by a given site. They were coded from 1 to n. For example, a medical record labelled 2/35/0968 belonged to the 968th participant from the 35th village of the second enrolment site. This coding system satisfied the confidentiality rules for all randomised, double-blind trials. It is a procedure used to avoid manipulation.
Reducing uncertainty consisted of consolidating and checking the data entered. Two different operators entered the data from each form. After double entry, the controller performed data matching to identify discordant entries.
When you do data matching, you can have 10 to 15% discordant entries. After eliminating the discordances, you can be sure that your data are 99% correct because they are strictly faithful to the spirit of the questionnaire’s data. (O.A., controller, male, age 34, interviewed on 25 November 2016 at the data-entry office in Maradi)
The controller sent the corrected data to the data manager, who verified the data matching, consolidated all the entered databases for each form, and pointed out any other inconsistences in filling out the forms. Since the operators’ entries were faithful to the data collected in the field, any such anomalies had to have come from the field. Those were called queries, requests, or data correction forms (DCFs). There were three types of queries: missing data (e.g. a child with no sex or a woman with no age); inconsistent data (e.g. an age 3 follow-up, since no participants were seen after 24 months); and outliers (e.g. an infant weighing 10 or 15 kg at birth). All queries were documented and sent to the monitor. Queries fell into one of two major categories: laboratory errors, concerning data from lab tests (forms 17C and 17D), and errors concerning data from the investigation, that is, data recorded in the field. Query resolution referred to the point at which all a form’s errors – either recording or data-entry errors – were corrected and re-entered in the database (Figure 3).
Data manager checking finalised records after query requests. Records are arranged with their stickers, depending on the hierarchical level of validation. The records, stored on shelves, were filed according to participant identifiers. Photo: Author.
Citation: Journal of Humanitarian Affairs 5, 1; 10.7227/JHA.099
Conditions of Data Production, Starting with Filling Out Forms
The size of the forms, their length, and the questions they asked depended on trial objectives. A document entitled Identification form for corrections to be made to the CRFs (the forms) was designed by the data manager and made available to the operators. Each documented all the errors he found during the test entry, which was used to get an improved version of a form. The data manager then centralised the identified errors and compiled the comments in a single file, which he sent to his advisor in Paris, where those comments were used to produce a new, dated version. This back-and-forth continued throughout the correction process up to the deadline set by the study sponsor or investigator for validating the final version. So it was not unusual to have more than six different versions of a given form, and nearly as many protocols, given the imperfections reported. The usefulness of that back-and-forth was the reduction of uncertainty. The forms were designed in the form of questionnaires, the responses to which were provided beforehand. This made perfect sense from the perspective of the investigators and clinicians, whose objective was to ensure that the participants’ routine medical follow-up at home and at the health centre was trackable and consistent. It is also a fundamental aspect of any clinical trial modelling the rules of evidence.
However, many of the questions, and their wording, were ill-suited to the population’s daily lives and living conditions. For example, in analysing Form 6A (general data at inclusion) – which was used to collect information about the socio-economic and demographic characteristics of the household – and how it was administered, one might have doubts about the accuracy of the responses. First, it was a questionnaire that asked eighty questions of a single mother who had come in to enrol her child. The greater the number of questions, the more tired the women became of answering them and the less focused they were in answering. And there was pressure on the nurses who asked them. One nurse at a site had to administer a questionnaire to all the mothers. They also had to administer the informed consent form and manage both non-protocol visits and the protocol visits to administer the candidate vaccine. Each of the five randomisation sites averaged four enrolments per day. It turned out to be difficult or impossible to conduct those activities with any focus. That was primarily a time management issue. While the sites were officially open from 8 a.m. to 5:30 p.m., they actually closed at 1 p.m. – or 2 p.m. at the latest. Even at public healthcare facilities, activities were generally conducted in the morning. Yet for Epicentre, four or five enrolments a day is not a huge amount. The staff, like the population, simply did not frequent the sites after 2 p.m. That is why, for example, the Form 6A questions were asked in cursory fashion, taking very little time. The answers were sometimes checked off without even asking the women the questions. This saved time; taking less than ten minutes to fill out the form left them time to deal with their other tasks.
The next problem was the language barrier, and the impossibility of answering questions due to the shame they provoked. Some questions were hard to translate into Hausa, and mistranslating a trivial expression could change the question’s meaning and thus skew the response. Example:
-
‘In the past four weeks, how many times did you or a member of your household not eat your preferred foods due to a lack of resources?’
Because that question touched on a purely private domain, it caused embarrassment and shame. To reveal one’s living conditions was to betray the head of household by exposing his inability to provide for his family’s daily needs.
-
‘In the past four weeks, did you have to borrow food or money because there wasn’t enough food or enough money to buy it?’
-
‘In the past four weeks, did you or a member of your household go an entire day and night without eating because there wasn’t enough food?’
Then there were questions that were considered blunt, whose answers weren’t spoken even if they were known. Those were questions having to do with family organisation and the household’s standard of living; they were considered embarrassing because they concerned a private matter. They also showed weakness on the part of the husband, who as the head of the family was supposed to provide for all of its members’ needs. For example, a nurse would check one of the following codes, depending on the responses to the questions that followed:
[Code: 0 = Not even once; 1 = Rarely (1 or 2 times); 2 = Occasionally (3−10 times); 3 = Often (>10 times)]
The most common responses were coded 0, to mean never. But knowing in advance that the women could never explicitly say how their household ate, for the reasons described above, the nurses got into the habit of checking a code without even asking the questions. Sometimes they decided to change the code to avoid any suspicion that they were automatically filling out the questionnaire.
I’m really embarrassed to ask a woman whether she had to borrow money to eat. She isn’t going to tell the truth. That’s why I always check zero. Some questions will never get truthful answers. (A.Z., nurse, female, age 40, interviewed on 10 November 2015 in Maradi)
Another question was: ‘What type of sanitation facility does your household use most often?’ The possible responses were:
[Code: 1 = Flush toilets; 2 = Improved pit latrines; 3 = Simple pit latrines with slab; 4 = Simple pit latrines without slab (open); 5 = Bucket/basin; 6 = None (bush); 7 = Other, specify]
For one thing, the nurses found it hard to translate this question – which seeks information about the population’s living conditions – into Hausa, since the expressions to which the signifiers referred could not be found. The few who could manage it explained the various types of latrines they were familiar with and asked the women to choose which they had in their house. Few chose option #6, as they were ashamed to say that they relieved themselves in the open air. Only the wealthy people in the village had enough to build simple latrines, with or without slabs.
In addition, Madarounfa is an area where several different ethnic groups co-exist. In the joking relationships2 between the various groups, the Peul – considered cousins of the Beriberi and Arawa3 – are described as most closed to modernity, and their women as the most reserved, who do not answer questions about their lifestyle. So when a Peul woman would come to the site, the nurses wouldn’t ask any questions. They simply checked off the forms, assuming they wouldn’t get answers in any case. And although it was environmental – not clinical – data, it was important, since it was some of the highest priority information being collected at the start of each randomisation. Indeed, in informal discussions with the women, they made fun of the questions they found absurd.
They [the nurses] always see us going whole days without eating when we take our children to the yan kwamuso.4 That means they’re hungry. It’s the same for us. They know we can’t afford to eat every day, but they ask us anyway.
From a distance, the expression or questions may have seemed innocuous in French. But they were problematic in this context and could easily have had a different meaning in Hausa. These questions were used to get some idea of the socio-economic conditions of the households whose women took part in the trial. They were a tool for planning and decision-making regarding their vulnerability, and for comparing the participants in terms of their standard of living. As such, these questions were not essential to the study. The responses given, whether untrue or not, had no direct impact on the efficacy of the vaccine being tested. That is probably one of the reasons why the staff or investigators paid no attention to the different wordings of such questions, how they were collected, or the form itself. Like Hamani 2013 study on essential family practices (EFPs), such as soap-and-water handwashing and exclusive breastfeeding, promoted by the World Health Organization (WHO) and the United Nations Children’s Fund (UNICEF), we see that questionnaire-based surveys can sometimes yield numbers that do not reflect reality. This also applies to the management of medical records made up of forms filled out with false or biased data.
We have huge concerns, and everyone knows it. People know what is happening in the field. A lot of our staff fake them. But some are honest. They are very few. The others never do what we ask them to do honestly. (O.A., male, age 34, interviewed in June 2016)
That perspective sums up some of the difficulties that the study investigators vastly underestimated, so great was the extent to which their confidence in the questionnaires outweighed the actual conditions of their completion. In the field, the staff – whatever their status – often faked the answers to the questionnaires, particularly the ones used for participant follow-up. Due to lack of motivation or under the pretext of being overloaded, the nursing assistants and community health agents did not do home visits for sick participants. Some did their assessments at home, out of the public eye. The others – fewer in number – who did conduct the visits lacked focus or just made up the numbers, checking empty boxes as they saw fit when work equipment like the thermometer was defective. For some, it was a way to avoid sending the forms in late. That explains many of the errors and the constant back-and-forth in the form validation process. More importantly, query resolution did not mean, for example, that questionnaires were filled out properly. In fact, the values they asked for (weight, height, mid-upper arm circumference, age, health and disease, etc.) changed over time. They could never be precisely known. One consequence of filling out questionnaires in a rush was outliers. In other words, there were a number of questionnaires to complete, and the staff wanted to do the work very quickly with less focus. As a result, some did not pay attention to what they were writing and did not re-read the completed documents to look for errors. Yet the investigators in the field worked hard to make sure that the protocol was followed. It was impossible, however, for them to monitor the activities of more than 400 workers in nearly 150 villages, spread out over a vast geographic area. Added to that was the lack of coordination of the community health agents’ activities. They were the ones who filled out Form 9A (weekly home visit), which summarised the participant’s adverse events, nutrition, clinical history, and medical care (using a pictorial calendar). Yet many filled it out without visiting the families. When a 9A was filled out completely but dishonestly, the nurse-supervisor, the doctor and the monitor would validate it without any of them noticing.
I’m sure that the ATRs5 [community health agents] filled out the 9A forms themselves. They didn’t do any visits. For example, one ATR had twenty participants. At four forms a month for each one, he had 80 forms to fill out himself. If you enter them, you’ll see the same information everywhere for the same month, without any changes. That’s impossible. At least a few participants must have had at least a cough. It was the same in the pilot phase, because I can assure you that fewer than 2% were followed. If you want to make the point, you just have to put in the worker’s initials and you’ll see everything he filled out since the start of the study. (O.A., male, age 34, interviewed in June 2016)
The only way to catch irregularities in their activities was to data-match with Form 13, in cases where the woman came to the site for consultation. Some people did their work according to Epicentre’s recommendations, others took those recommendations as mere suggestions and did something else. In such cases they adhered to other norms. Their approach and style in carrying out their activities were based on practical norms, and in some cases on the impossibility of doing otherwise. Some did not have the qualifications needed for the work.
In addition, the twenty-five nurse-supervisors responsible for overseeing the nursing assistants – in case of adverse events, in particular – were not in the field on a regular basis. Despite administration pressure to live in their assigned village, they preferred to stay in town and commute daily. They were unmotivated due to contentious relationships with the administration over salary, being forced to live on-site, their relationship with their superiors in the trial, promotions, study demands, etc. So they would come in around 10 a.m. and leave before 1 p.m. without having had time to check whether the forms were duly completed.
From the Thoroughness of Recording Practices to Production of Unreliable Data: The Example of Clinical Data Manipulation
Form completion and data collection in the field were strictly supervised and monitored. The entire trial was designed to manage uncertainty. The trial made multiple attempts to control such uncertainty. At every level, from participant randomisation to data entry, the study seemed painstakingly designed to produce reliable data. But ultimately, it was clear that it would never succeed. There would always be doubt about the reliability of the data and scientific results. The examples below attest to the many attempts at supervision and show how the system was unable to eliminate uncertainty in the real world.
Completion of Follow-Up Forms
The database management team faced some challenges regarding data entry and uncertainty management. In addition to those challenges there were inherent shortcomings in both the study design and the team itself. First, the queries (correction requests or errors) could be normal data that the programme could not identify. According to O.A., a records clerk and controller at the data management office,
There were cases where the queries were not actually data-entry errors. Rather, it was errors in the program detecting them. Imagine, yesterday they gave us a 180-page list of queries for the lab. It was impossible, and I even explained to the data manager that the problem was at their level. But he refused to admit it. The head of the lab ended up saying that the forms were filled out correctly. (O.A., male, age 34, interviewed in June 2016)
Then, contrary to the protocol requirement that all trial forms be double-entered to ensure that the entered data was reliable, from the start of the study all such documents were entered only once. The only forms that got particular attention from headquarters and were double-entered in the database were, for obvious reasons, the forms for reported diarrhoea cases; those got special handling.
For years we were told to just do single-entry. I don’t know why. Afterward we asked R.G. to initial as the person who did the double-entry, in order to lie and cover things up. He said he would never lie, unless he drafted his resignation and wrote a letter to the administration to say why he was resigning. Everyone knew what was going on, but no one said anything and things continued. (L.C., male, age 33, interviewed in June 2016)
That issue pitted the database managers against the data-entry operators for two years running. Ultimately, the data-entry operators agreed to put their first and last initials on thousands of documents, although they were not entered.
They told us that it was really an internal matter. No one could know what was going on, and we were all in it together. So we should protect them. We should initial as if we had done double-entry. It’s supposed to be a scientific study. There shouldn’t be any manipulation. There will be lies. (K.B., male, age 30, interviewed on 12 March 2017)
If the data-entry operators all refused to sign the fake double-entries at the outset, it was in part because there had been a previous incident that had a far-reaching impact on how field activities themselves were conducted.
Posthumous at-home follow-up of a participant?
On a Tuesday in April 2015, an infant died of gastroenteritis after having been referred to the Madarounfa district hospital. Notified by the village’s community health agent, the enrolment site managers offered the family their condolences. The following week, the nurse-supervisor asked the nursing assistant for the 11A form (surveillance of gastroenteritis cases), which was the proof of the obligatory daily at-home follow-up. It required putting together the questionnaires from the participant’s medical record, which would officially be closed after a verbal autopsy (Form 22). The nursing assistant was unaware at the time of the child’s death. He submitted the form with details, indicating that the infant was not only alive, but doing better – that is, that he followed-up on him on Tuesday, Wednesday, Thursday and Friday. Worse yet, he explained to his supervisor that he would be unable to visit the participant over the next few days because it was the weekend, but that he intended to go see him right away.
How was it possible to follow-up on a participant who had died four days earlier? The scandal was too serious to be covered up. The study site doctor summoned the investigators. The field medical investigator went there and asked the nursing assistant to formally acknowledge having cheated, so that the record could be filled out properly. The latter refused, under the pretext that he was not the only one who cheated, and that other staff members were also involved in various types of tinkering and schemes. According to him, other staff members did worse things than following a dead participant. One of the study site nurses challenged him in the name of religion, advising him as a Muslim to take responsibility for his mistakes and apologise to the participant’s parents. She also felt that he should explain that while the participant had died, his death might have been avoided had the child been followed in accordance with the study protocol. If he made those apologies, his friends promised to intervene with the administration to lighten the inevitable punishment.
The big sister did everything she could to get him to apologise to the child’s family and acknowledge that he hadn’t gone to see him. But he was stubborn. He said that that was what everyone did. (A. H., nurse)
To the nursing assistant in question, apologising would have meant admitting having cheated and taking sole responsibility for it, but it would also be considered a dishonour societally. During that period the 11A form disappeared from the child’s record. Attention turned to the records clerk, whom some accused of having deliberately gotten rid of the document to protect his colleague.
M.G. [the human and financial resource coordinator] called me from Niamey to ask me to do everything I could to find the 11A. They thought that I kept it. I said I didn’t have it and that I had no interest in covering it up. I couldn’t do that to someone whose son had died, because I’m a father. I have children, too. I spent three days checking all of the forms in the records that were on-site. I checked all of them, one by one. A nurse called a few days later to tell me that someone saw the form in a rubbish bin. (L. C., male, age 33, interviewed in June 2016)
The matter ended with the nursing assistant being fired. But if the episode echoed throughout the facility, it was because the participant had died. This specific case was just the straw that broke the camel’s back, because forms had been filled out automatically for a long time. And one factor in the lack of motivation was the conflict between the staff members and the administration about working conditions. The contract with the administration was of unspecified duration (no fixed term or open-ended contracts6). When the pilot phase started, the staff expected to get a fixed-term contract and another open-ended contract after three months of work. But for three years they were unsuccessful. Some chose to resign. Others stayed, but without any real motivation, pretending to live in the villages as required for an open-ended contract.
Medical and Research Standards Put to the Test by Staff Members Circumventing the Rules: The Example of a Verbal Autopsy
Sometimes, truly sick children were not sent to healthcare facilities, despite free transportation for participants’ parents to encourage them to go. Mothers did not really consider conditions like cough, conjunctivitis, nasal discharge, or even diarrhoea serious. They only considered constant vomiting and a high fever symptoms. Diarrhoea was seen as pathological when the victim was dehydrated. The others were simply treated with self-prescribed medication or traditional medicine. Only the most serious cases were sent to the health centres once other options had been exhausted.
Verbal autopsies were also an opportunity for manipulation, due to their sensitivity. If a participant died, a Form 22 was opened and the monitor always went to the site a week later to document the actual circumstances of the participant’s death. That was a situation where no one wanted to take the blame, especially if the study protocol had not been followed. So each person would try to give a suitable account.
The health workers always tried to protect one another out of solidarity, and to avoid being accused of shirking their professional duty, but most especially to avoid being punished. Hence the doctors would protect their supervisors, who would in turn protect their nursing assistants. The authentic forms were often modified, falsified or changed to meet the protocol requirements, especially in the case of a death. Practices like that became commonplace. When there were controversies like participant deaths or referrals, the staff arranged things to avoid revealing flaws in the system or breaches in professional conduct. That helped protect them from punishment. Since none of the trial monitors spoke Hausa,7 it was easy for the staff to change the information given during verbal autopsies done in their presence. Likewise, for interpersonal reasons, so as not to be nasty, the monitor might himself change an autopsy to benefit a staff member who had made a mistake. Showing leniency and humanity bolstered his legitimacy and popularity with the staff. Violence was done to the data and to sensitive information about participant follow-up on both sides of the study community. The ethical requirements and procedures for a clinical trial are highly standardised. While they can be adapted to certain situations, those procedures cannot make allowances for real-world data collection and handling. Which is as it should be, because the statistics from a trial can never see what the trial itself was not designed to see – that is, the social conditions under which the trial was conducted and the data produced. These examples show that it is impossible to manage uncertainty in a clinical trial, since it is made of interactions whose conditions are always changing (Brives et al., 2016) and which depend on working conditions. Uncertainty is itself inherent to real life. In other words, a clinical trial is not a locked box in a laboratory. It exists in the real world, with staff members that come and go and have different training, and women who agree or refuse, who have husbands, etc. Even when the system for filling out forms, validating them and entering them in the database is designed to control for such uncertainties, it never totally succeeds in doing so – life always overwhelms it.
Even in the Maradi laboratory – a facility particularly well-prepared to minimise study bias – life escaped the protocol. The technicians invented different working methods and negotiated with the context. The lab seemed like a space for negotiation. The result of scientific research is the fruit of such negotiations, a product neither totally planned for nor completely contingent. The system itself, while reducing uncertainty to produce accuracy (Callon et al., 2001), creates other uncertainties that never completely go away. From an anthropological perspective, a clinical trial is never completely reliable, since it is implemented in life as it really is.
Conclusion
Everything was happening as if the quantitative data from Epicentre’s clinical trial became totally independent of the social reality in which they were produced. Yet when science is made, the two aspects (conduct of the trial and social reality) are inextricably linked. They work together to produce knowledge (Callon, 1986). This ethnography of science ‘in the making’ (Pestre, 2010) in the Nigerien context illustrates an institutional and bureaucratic weakness detrimental to the making of rigorous science, as well as it points to a local work culture among the healthcare staff that deviated from formal standards (Olivier de Sardan, 2009, 2021; Olivier de Sardan et al., 2017).
That is one of the reasons why uncertainty never ends and why controlling uncertainty in all aspects of the trial was a perpetual quest – the quest for reliable data. In his book on the reliability of African statistics, Morten Jerven (2013) offers an ethnography of several statistical offices in Africa. Statistics are widely used as tools for governing populations, and in particular by international institutions and government for planning and decision-making. Jerven describes various types of manipulation, negligence, incompetence and the mise-en-scène associated with the fabrication of numbers. In his conclusions, he demonstrates to what extent and why data produced in Africa are poor quality. Admittedly, his ethnography does not concern data production in biomedical contexts. But it does help us rethink how data is produced in clinical trials. The production of standardised knowledge relies solely on numbers. But to produce rigorous scientific data, a multidisciplinary approach that looks at the social conditions of their production is important. Rethinking how numbers are produced in evidence-based medicine means incorporating the interactional (Brives et al., 2016) or pragmatic (Olivier de Sardan et al., 2017) context and minimising the bias in a scientific study. Far from political considerations and more or less normative approaches that outright reject the numbers derived from a scientific activity, as Jerven does, we need to look more closely at how they are produced, what they are hiding, and what they produce. Beyond epidemiological statistics, numbers can be wrong but effective. According to Vincanne Adams, while the numbers in most epidemiological statistics are admittedly incorrect, they do allow health systems in Africa to describe a reality and obtain funding (Adams, 2016). It is thanks to the numbers that funding is obtained. It is also thanks to the numbers that health programmes are evaluated. As James Scott sees it, the degree of success or failure of modern state governments depends on their ability to create a quantification system that can narrow complex social problems down to something that can be counted (Scott, 2008).
To hide real life in a clinical trial is to ignore the social conditions in which it is conducted. Despite a substantial investment in the Madarounfa trial, after more than a decade of scientific activity, we knew that the women could not answer certain questions that cause shame in their culture. We also knew of the staff members’ departures from standards and ethics, and their schemes regarding autopsies, forms, etc. Put simply, trials need numbers to prove the efficacy of the drugs they are testing.
The numbers tell and describe a reality about the world. But all numbers are produced in a certain way. We must instead question the ‘veracity’ of the numbers. Some may simply be more or less correct than others. Put differently, some may better describe a reality about the world than others. It’s all a question of the social, political and working conditions under which the data, converted into numbers, are produced. Clinical trials – whether conducted by Epicentre in Niger or by another research institution somewhere else – cannot escape that. But at bottom, that is not the issue. The real issue is whether the numbers can make health systems and medical research work, and express health policy funding needs.
The trial did not run into problems because the women didn’t know anything or were considered ignorant of the trial’s procedures. It was the system itself, the experimental method, that was flawed. That doesn’t mean that the study personnel – the sponsors and investigators, in particular – were unaware of the trial’s reality, but they chose the idealistic aim of following the protocol and the standardised operational procedures. On paper the trial seemed so well designed, using those tools, that it was hard to recognise and admit its flaws. In other words, they tended to blame the women rather than acknowledge their own difficulties, even when they were aware of them. That created a kind of complex cooperation in the clinical trial.
Ultimately, a clinical trial in Africa – and particularly in poor countries like Niger, in a context where western pharmaceutical companies are distrusted and accused of using African populations as guinea pigs (Amiel, 2011) – should not limit itself to highly standardised protocols and procedures that basically serve only to satisfy the rules of evidence. Clinical trials are only feasible when they know and incorporate the political and social dimensions, and local context, of their implementation.
Acknowledgements
This article was translated from French into English by Nina Friedman (MSF Paris) with additional checks and editing by Michaël Neuman and Bertrand Taithe.
Notes
Epicentre hired the article’s author, a doctoral student in anthropology who speaks the local language, to participate in real-time monitoring of the trial, communicating daily with everyone involved in the trial (investigators, healthcare staff, families, community health agents, etc.).
According to UNESCO: ‘Joking relationships are a social practice performed among ethnolinguistic communities, groups and individuals to promote fraternity, solidarity and conviviality. They take the form of a playful taunting between two people from two communities that represent symbolically the husband and wife cross-cousin branches of the same family. Such relationships are often based on ancestral pacts forbidding conflict or war between specific communities, and imply that the members must love one another and provide assistance where needed. The members have a duty to tell each other the truth, to joke together and to pool their respective assets, knowing that any dispute must be settled peacefully,’ https://ich.unesco.org/en/RL/practices-and-expressions-of-joking-relationships-in-niger-01009 (accessed 31 July 2023).
These are ethnic groups in Niger.
Literally, this expression means those who are hungry or those who are starving. To the population it refers to the MSF’s in-patient therapeutic feeding centre, set up to treat malnutrition in children from 0 to 5 years. In the past, the name had carried stigma, but with MSF’s presence, the expression lost its original sense. It has now become a generic term like hospital, home, etc. When the men and women speak of yan kwamuso, they are not necessarily referring to a sickly, emaciated, undernourished child. They are referring to the centre that MSF set up to treat malnourished children.
ATR: acronym for the community health agents. (Agent, Terrain, Rose / EN: Agent, Field, Pink).
The employer can end the employee’s contract without recourse or compensation.
They were all expatriates: an Ivorian, a Guinean, a Togolese, and so on.
Works Cited
Adams, V. (ed.) (2016), Metrics: What Counts in Global Health (Durham, NC: Duke University Press).
Amiel, P. (2011), Des cobayes et des hommes : Expérimentation sur l’être humain et justice (Paris: Les Belles Lettres).
Berg, M. and Bowker, G. (1997), ‘The Multiple Bodies of the Medical Record: Toward a Sociology of an Artifact’, The Sociological Quarterly, 38:3, 513–37, doi:10.1111/j.1533-8525.1997.tb00490.x.
Bertho-Huidal, M. (2012), Charity business : Le grand marché de la santé mondiale (Paris: Vendémiaire).
Brives, C. (2012), ‘L’individu dans un essai thérapeutique : Sur quelques aspects du devenir objet dans les expérimentations scientifiques’, Revue d’anthropologie des connaissances, 6:3, Art. 3, doi:10.3917/rac.017.0185.
Brives, C., Le Marcis, F. and Sanabria, E. (2016), ‘What’s in a Context? Tenses and Tensions in Evidence-Based Medicine’, Medical Anthropology, 35:5, 369–76, doi:10.1080/01459740.2016.1160089.
Callon, M. (1986), ‘Eléments pour une sociologie de la traduction : la domestication des coquilles Saint-Jacques et des marins pêcheurs dans la baie de Saint‐Brieuc’, L’Année sociologique (1940/1948), 36, 169–208.
Callon, M. (2003), ‘Science et société : Les trois traductions’, Cahiers du Mouvement universel de la responsabilité scientifique, 42, 54–69.
Callon, M., Lascoumes, P. and Barthe, Y. (2001), Agir dans un monde incertain : essai sur la démocratie technique (Paris: Seuil).
Couderc, M. (2011), ‘Enjeux et pratiques de la recherche médicale transnationale en Afrique : Analyse anthropologique d’un centre de recherche clinique sur le VIH à Dakar (Sénégal)’, Bulletin Amades, 84, doi:10.4000/amades.1327.
Desrosières, A. (2014), Prouver et gouverner : Une analyse politique des statistiques publiques (Paris: La Découverte).
Fassin, D. (2010), La Raison humanitaire : Une histoire morale du temps présent (Paris: Gallimard / Seuil).
Freidson, E. (1984), La Profession médicale (Paris: Payot).
Hamani, O. (2013), Les Pratiques Familiales Essentielles (PFE) au Niger: Socio-anthropologie d'une intervention à base communautaire (Niamey: LASDEL), https://lasdel.net/download/n104-les-pratiques-familiales-essentielles-pfe-au-niger-socio-anthropologie-dune-intervention-a-base-communautaire-par-hamani-oumarou-2013/ (accessed 31 July).
Jerven, M. (2013), Poor Numbers: How We Are Misled by African Development Statistics and What to Do about It (Ithaca, NY and London: Cornell University Press).
Lakoff, A. (2007), ‘The Right Patients for the Drug: Managing the Placebo Effect in Antidepressant Trials, BioSocieties, 2:1, 57–71, doi:10.1017/S1745855207005054.
Lauritsen, J. M., Bruus, M. and Myatt, M. (2001), EpiData Software [computer program], www.epidata.dk/.
Latour, B. (2012), Enquête sur les modes d’existence : Une anthropologie des modernes (Paris: La Découverte).
Latour, B. and Woolgar, S. (2006), La vie de laboratoire : la Production des faits scientifiques (Paris: La Découverte).
Lefranc, M.-L. (2015), L’Inde,’ pharmacie du Sud’. Son rôle en matière de santé mondiale et commerce international (Paris: L’Harmattan).
Likin, M. (2009), ‘Médecins sans frontières et l’apparition d’un consensus humanitaire’, Matériaux pour l’histoire de notre temps, 95:3, 25–9, doi:10.3917/mate.095.0004.
Marks, H. (1999), La médecine des preuves. Histoire et anthropologie des essais cliniques : 1900–1990 (Paris: Institut Synthélabo pour le progrès de la connaissance).
Mfuka, C. (2002), ‘Accords ADPIC et brevets pharmaceutiques : Le difficile accès des pays en développement aux médicaments antisida’, Revue d’économie industrielle, 99, 191–214, doi:10.3406/rei.2002.3023.
Olivier de Sardan, J.-P. (2009), ‘Les huit modes de gouvernance locale en Afrique de l’Ouest’, Afrique : pouvoir et politique, Working Paper No. 4 (London: ODI).
Olivier de Sardan, J.-P. (2021), La revanche des contextes : Des mésaventures de l’ingéniérie sociale, en Afrique et au-delà (Paris: Karthala).
Olivier de Sardan, J.-P., Diarra, A. and Moha, M. (2017), ‘Travelling Models and the Challenge of Pragmatic Contexts and Practical Norms: The Case of Maternal Health’, Health Research Policy and Systems, 15:S1, Art. 60, doi:10.1186/s12961-017-0213-9.
Pestre, D. (2010), Introduction aux Science Studies (Paris: La Découverte).
Petryna, A. (2009), When Experiments Travel: Clinical Trials and the Global Search for Human Subjects (Princeton, NJ: Princeton University Press).
Redfield, P. (2005), ‘Doctors, Borders, and Life in Crisis’, Cultural Anthropology, 20:3, 328–61, doi:10.1525/can.2005.20.3.328.
Scott, J. C. (2008), Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed [reprint] (New Haven, CT and London: Yale University Press).
Strauss, A. L. (1992), La trame de la négociation : Sociologie qualitative et interactionnisme (I. Baszanger, ed.) (Paris: L’Harmattan).
Sunder Rajan, K. (2017), Pharmocracy: Value, Politics, and Knowledge in Global Biomedicine (Durham, NC: Duke University Press).
