Department of Research & Evaluation News

After COVID-19 vaccines became available in December 2020, several smartphone-based reporting systems were developed to monitor vaccine side effects. They included V-safe from the Centers for Disease Control and Prevention. However, like most large surveillance systems, they had imperfections.

Kaiser Permanente Southern California collaborated with the CDC and others to create a new system in April 2021 that would improve vaccine safety monitoring by linking participants’ smartphone-based reports to their electronic health records. This innovation allowed researchers to collect medical and demographic information and validate clinical diagnoses.

The study, which showed the success of the program, was published November 11, 2024, in Journal of Medical Internet Research mHealth and uHealth.

Not only did Kaiser Permanente’s system provide more detailed vaccine safety information than other survey tools, it also had much higher participation rates. While the CDC’s V-safe had participation rates of around 1%, the Kaiser Permanente system had an 8% participation rate.

“Due to high participation rates and the availability of clinical records, our Kaiser Permanente Side Effect Monitor system gave researchers and physicians a more complete picture of vaccine safety among all vaccinated patients than other systems,” said lead author Debbie Malden, DPhil, MSc. Dr. Malden is an epidemiologist who was with the Kaiser Permanente Southern California Department of Research & Evaluation during the study period. “These findings emphasize the importance of using a digital reporting system through a trusted health care provider for higher participation rates and potential clinical follow-up.”

How the success was measured

To determine the Side Effect Monitor’s effectiveness, researchers assessed data from the electronic health records of 2,104,923 patients vaccinated between April 23, 2021, and July 31, 2023. Of those, 8% enrolled in Kaiser Permanente’s Side Effect Monitor system and completed at least one survey in the 35 days following their COVID-19 vaccination.

The study found that:

• The lowest participation rates were for parents of children ages 12 to 17 years (0.6% participation rate). Highest participation was among older adults ages 61 to 70 years (12.6%).
• Participation varied by race and ethnicity as well as the economic status of the neighborhood where the patients lived.
  • People who were white were twice as likely to participate compared with other races and ethnicities (13.5% vs. 3.8% to 7.4%).
  • Vaccinated patients living in poorer neighborhoods were less likely to participate than those living in wealthier neighborhoods.
  • Researchers also found lower participation among young adults and men.

“Despite achieving a higher participation rate compared with similar survey tools, we still saw lower participation among some vulnerable populations,” Dr. Malden said. “This could indicate barriers to participation among these groups. If self-reporting digital tools are to be widely used in public health, we will need to find ways to make their implementation more equitable.”

Participation rate varied by recruitment method

Researchers also found rates of participation varied by how the patients were recruited to participate.

• Passive recruitment methods such as expecting patients to scan QR codes on flyers and posters at the time of vaccination enrolled only 1.4% of the participants.
• Among people contacted through their online Kaiser Permanente member account, 17.6% joined and submitted at least 1 survey following their vaccination.
• Text message invitations resulted in a 10.8% participation rate.
• E-mail invitations achieved a 4.4% participation rate.

This study found some interesting quirks in self-reported vaccine safety systems. “Basically, people who had reactions straight after vaccination were more motivated to continue reporting to us, whether they experienced more reactions over this time or not,” Dr. Malden explained.

The system has future applications

“This study shows that it is possible to create a successful digital self-reporting system as a timely, flexible, and scalable approach that could supplement existing vaccine safety surveillance systems,” Dr. Malden said. “We feel strongly that this digital tool has many future potential benefits for patients, including enhancing vaccine safety and monitoring of new medical procedures and treatments. Right now, we are using this system to monitor side effects to RSV vaccines.”

The study was funded by the CDC but was solely conducted at Kaiser Permanente Southern California.

In addition to Dr. Malden, researchers on this study included Julianne Gee, MPH; Amelia Jazwa, MSPH; Eric S. Weintraub, MPH; and Michael M. McNeil, MD, MPH; of the CDC; and Sungching Glenn, MS; Zhuoxin Li, MS; Denison S. Ryan, MPH; Cassandra Bezi, MS; Sunhea Kim, MPH; and senior author Sara Y. Tartof, PhD, MPH; of the Department of Research & Evaluation. Dr. Tartof also is a professor with the Kaiser Permanente Bernard J. Tyson School of Medicine.