Research

What endocrinologists and diabetes teams should know about exocrine pancreatic insufficiency (EPI)

by Dana Lewis

Recently I was asked to give a talk to a group of endocrinologists and other diabetes providers about exocrine pancreatic insufficiency, and while building my presentation I was thinking through everything I wish endocrinologists knew about EPI. (We call it EPI in the US, but if you are elsewhere in the world you may see it referred to as PEI, for pancreatic exocrine insufficiency. This is the same condition.) 

This is targeted toward endocrinologists and other diabetes care providers and heavily focused on people with diabetes but this is also relevant to gastroenterologists and primary care providers as well, and includes info on prescribing enzymes that is relevant to anyone with EPI regardless of any co-condition.

1. Exocrine pancreatic insufficiency occurs – and is common – in people with diabetes. All types. 

I was diagnosed with EPI back at the end of 2023 by pure luck, because I saw a post about a dog with GI symptoms that were solved by dosing enzymes with their food. I was curious and looked up if this condition occurred in humans, too – and it does. It’s called exocrine pancreatic insufficiency (EPI or PEI) and happens when the pancreas no longer produces enough digestive enzymes to help digest the fat, protein, and carbohydrates that you eat. I had a textbook case of symptoms, yet despite ongoing investigations with a gastroenterologist for over a year, it was never brought up on the list of possibilities as a condition we should screen for. The more I read, particularly seeing that yes, it occurs in people with diabetes, and studies show around 30% prevalence in people with type 1 diabetes and people with type 2 diabetes, I was flabbergasted that my gastroenterologist never considered it.

And why had I never heard of it? I had heard of the connection between T1D and celiac, which is why I ended up getting screened and diagnosed with celiac very quickly during college when I developed GI symptoms. I hear of gastroparesis quite frequently as well in people with diabetes. But EPI is likely more common than celiac and gastroparesis combined and I never hear it talked about by endocrinologists, researchers, or people with diabetes.

I could have been saved from over a year of suffering if my gastroenterologist was more aware of this or if my endocrinologist was more aware of the prevalence of EPI in diabetes. It’s now my goal to make sure we cut this window of suffering for other people with diabetes. So the number one takeaway for endocrinologists is that EPI occurs in people with diabetes, it’s more common than celiac and gastroparesis, and even if you don’t screen for EPI first, if GI symptoms are occurring and you’ve ruled out celiac and gastroparesis, screen for EPI next! (If not first, depending on symptoms).

  1. How do you screen for EPI in people with diabetes?

Endocrinologists can absolutely order the screening test for people with diabetes.

(Yes, you can refer to a gastroenterologist…but…sadly…many gastroenterologists are not necessarily more advanced in their care of EPI, as my experience has illustrated. Gastroenterologists also underscreen, underdiagnose, and undertreat EPI across a wide variety of conditions, so unless you have a top-notch GI you know personally who is familiar and treats EPI with the actual standard of care, more on that below, you should consider ordering the test yourself and managing the EPI process for your patient with diabetes.)

The screening test most commonly used is not perfect, admittedly, or fun. The fecal elastase test requires a stool sample and getting it back to the lab in a certain time frame. But, it can be illustrative of EPI or likely EPI. The typical guidelines say <=200 ug/g indicates EPI. Historically there have been cutoffs for <100 and <=200 for different categories, but it doesn’t really matter because you don’t prescribe based on severity level or treat any differently. The 100/200 cutoffs just give you data from past research on the accuracy (sensitivity and specificity) of the elastase test.

But human elastase fluctuates, so someone could have a test with a result of 205 ug/g, for example, at the time they are tested and other times be more obviously under the 200 cutoff, e.g. 155 ug/g or some other random number.  My plea for endocrinologists is to be aware of the limitations of the elastase test in both directions, and contextualize the fact that an elastase >500 ug/g likely means not EPI, but something in the 200-500 ug/g range, though, is worth repeating a test and/or considering a trial of enzymes. (There’s an interesting study showing people with 200-500 elastase benefit from enzymes, so keep that in mind). I wrote more about elastase testing and the sensitivity and specificity here.

The TLDR I wish people knew is that 200 is not a precise, exact cutoff. I think it is incorrect to treat 201 ug/g as “you don’t have EPI!”. This is where clinical judgement comes into play, depending on if you’ve already ruled out obvious other issues and symptoms continue, and you have an elastase near or below the 200 ug/g ‘cutoff’, then you may consider enzyme trials with your patients.

Another factor to take into account is EPI-specific symptom frequency and severity. See the EPI/PEI-SS section 5 below.

And yes, EPI occurs in people who are overweight and it’s possible to have EPI and not (yet) have vitamin deficiencies. It doesn’t require unexplained weight loss or pain/pancreatitis symptoms or vitamin deficiencies in order to diagnose EPI.

It was also interesting, during my presentation, to hear from an endocrinologist, the wonderful Dr. Marina Basina from Stanford, who was sharing how she is also screening some people with diabetes who are “asymptomatic” (GI-wise) but have unexplained extreme spikes and drops on their CGM. Once they are on enzymes, they’ve been telling her how they feel like different people and have so much energy compared to before. This “asymptomatic” presentation – with some possible ways to detect based on CGM data and other atypical symptoms – is something I hope we research more in the future.

  1. Endocrinologists can prescribe pancreatic enzyme replacement therapy (PERT) and should do so according to guidelines.

Endos absolutely can and should be willing to prescribe enzymes (aka PERT) to their patients with diabetes, if they’re able and willing to help manage the prescription. But there’s a few things to know.

The first is that most guidelines recommend 40-50,000 units of lipase per meal. ( EPI in diabetes doesn’t have its own specific ‘guidelines’: because EPI is pretty similar across conditions, there’s no reason not to use guidelines developed for EPI in other conditions as a starting point for dosing. Here’s a systematic review of all dosing guidelines across conditions including the evidence and best practices for diabetes and EPI.)

Coming from units of insulin, this sounds like a lot, so you (and patients) have to wrap your brain around the dosing sizes, first. But it’s actually not that much. Pills come in a variety of sizes depending on which brand, ranging from 3,000 units of lipase per pill up to 60,000 units of lipase per pill. The pills actually contain THREE types of enzymes: lipase, for fat digestion; protease, for protein digestion; and amylase, for carbohydrate digestion. They all have different amounts per pill but fat is the most commonly (95+%) researched factor and so the pills are described solely by fat/lipase amounts. I’ll stick to that in this section, but read more about protein below.

Please, oh please, don’t prescribe 3,000 or 10,000 to “start small”. The dosing guidance is 50,000 (ish) per meal as a starting dose. Most people actually need more than that, depending on what they are eating (fat, protein, etc). Enzymes are like insulin in that underdosing can feel like a drop in the bucket. But, ‘too much’ is not like insulin. ‘Too much’ usually requires a lot more than the equivalent insulin dose to cause issues. Most people are massively underdosed and if you start with a 10,000 dose, for example, it’s common to then infer that “it’s not working” and it must not be EPI, and people stop taking it (or the prescription is ended). In reality, ‘it’s not working’ is usually “this is not enough” for the person’s individualized needs and what food they are consuming. Again, the starting dose is around 50,000 units of lipase per meal, with it being common to increase as high as 150-200,000 per meal, depending on diet. There are some studies that have some correlation to ‘severity’ of EPI, but it’s also very individual on the person’s body AND what they are eating. Someone who eats a higher fat diet – such as people with diabetes who might choose a lower carb diet with more fat and protein – will need more enzymes than someone else who eats a low fat, high carb diet. (And individual to individual it varies).

So how much should you prescribe? And how?

4. How do you prescribe enzymes to someone with EPI? 

Given the above, you can Rx a starting dose of around 40,000-50,000 units of lipase a few ways. One brand, Zenpep, has pills of 25,000 units of lipase. Let’s do the math: this means 2 pills of Zenpep 25,000 per meal. Another brand, Creon, has pills with 36,000 units of lipase. You could choose to start with 1 pill per meal (36,000 units of lipase) or two (72,000 units of lipase). Either would be suitable, BUT regardless of which brand and which dose you choose to START with, this has to come with a conversation and education to the patient about what they are taking, WHY, and WHEN.

I found that both fat and protein have to be titrated. You can give someone a low fat meal with enzymes that work and then give someone the same amount of fat with protein and get massive symptoms. Some people conclude that they can’t eat that food whereas in reality, it’s the protein that needs additional enzymes. (I have a lot of specific examples in this blog post showing how common low-fat high protein meals like chicken and veggies may need as much enzyme dosing as higher fat meals, because of the protein content.)

This means you need to teach people that they can expect to develop a ratio of lipase:fat and protease:protein similar to how they have an insulin:carb ratio. This is something they can self-titrate and figure out about how many grams of fat or protein one pill “covers”, keeping in mind that they can have different pill sizes.

You should write the Rx to be taken with food. Do NOT write this as “take three times a day” because you assume people eat three meals a day. This is faulty because 1) people don’t then understand it is WITH FOOD and if they’re not eating they shouldn’t take it and 2) some people eat snacks and they need to dose for snacks, too, plus 3) some people (especially those who also have gastroparesis or other situations) eat several small meals a day.

I would explicitly explain to people that they need to take this with food and drinks with nutrients (so water and diet soda do not need enzymes) and that you are writing the prescription for X pills a day to be taken with food. And, tell them that once they figure out what dose is working for them and how many pills a day they need to call/message in and that you will update the prescription for them. That’s what happened for me: my Rx was originally written for 2 pills per meal, so I got 6 pills per day. I ended up calling and asking them to change my Rx to 8 pills per day* so that I could have snacks some times, or to otherwise to support the flexibility of sometimes having a larger meal that requires more than 2 pills.

(* Remember the dose matters; I am not specifying my dose/pill size/brand here, but this is not to say that 8 pills is the magic number for everyone.)

The benefit of educating people this way is it helps anchor that 1) they need to figure out what dose works for them and 2) the starting dose may be not what they need.

I am over-emphasizing this because in online communities I see a huge number of people who are not told that this Rx is for consuming with food (or drinks with nutrients, e.g. milky drinks) so they take it on an empty stomach because “they’re supposed to take it”, or they don’t realize they need to adjust it in proportion to their food; or they run out and can’t get a refill because of how the original prescription was written and they for whatever reason can’t get in contact with a provider to increase the prescription. So yes, this type of education when you provide the prescription matters a lot to set people up for success! 

You can also, eventually, prescribe multiple ‘sizes’ (doses), e.g. a Creon 24,000 and Creon 36,000 or a Zenpep 25,000 and a Zenpep 60,000 based on what works for people. If someone can take 1 Zenpep 60,000 for most meals and is satisfied with that, great; they may not need that dose for small snacks though and would benefit from the additional size (e.g. Zenpep 25,000) to have on hand for snacks. Or to ‘top off’ a bigger meal without doubling up on the 60,000. (2x 60k is 120,000 whereas a 60k and a 25k becomes 85,000 units of lipase). That’s likely a follow up conversation after they’ve done some logging and titration work to figure out what works for them and what they typically eat, plus outlier meals.

Here are some common scenarios of when most people should take enzymes and some “it depends but most people don’t need enzymes”:

  • One bite of something (e.g. tasting as you are cooking dinner) – because the amount of fat, protein, and carbs in a lick or taste is pretty minimal, people could maybe get away with no enzymes.
  • Black coffee – probably doesn’t need enzymes, unless:
  • Black coffee with milk/cream/etc added – probably needs enzymes depending on the quantity added.
  • Grazing as you are at a buffet with a nibble here, a nibble there – probably needs enzymes as the grams of fat and protein go up.
  • A snack that has more than a few grams of fat/protein – probably needs enzymes
  • A piece of fruit – varies person to person. Some people can eat fruit standalone with no enzymes, others seem to need the enzymes. Why? Because of the carbohydrate digestion, even though there is little to zero fat and protein involved.
  • A salad that is low fat – probably needs enzymes especially if it has protein, even though it doesn’t have “much fat”. (Also some fat likely still needs enzymes, in and of itself.)
  • A high fat meal – needs enzymes and likely more enzymes than typical if it’s a bigger meal or a meal with a higher quantity of fat than typical. E.g. if a typical meal is around 45g of fat and someone takes two pills (of whatever size), that means their pill:fat ratio is 1 pill per ~22g of fat. If they were to then sit down to Thanksgiving dinner and have 60g of fat, they probably need at least 3 pills because the additional fat beyond 45g of fat needs to be dosed for. More examples here about figuring this out.

How many grams of fat and protein require enzymes? The great “it depends”. For me, I don’t have an absolute number but if it’s greater than 2-3 grams of fat or I am uncertain about the quantity I am eating, I take enzymes.

I do not take enzymes for “low” treats that have no fat and protein. So this means no enzymes (for me) for:

  • Juice
  • Soda
  • Candy without significant fat/protein (airheads, gummies, fruit snacks, etc.)
  • Glucose tabs

(It likely depends on the amount – probably a larger amount of carbs-only would still require enzymes. But for me since I do small or medium amounts of carbs for lows, I haven’t found a threshold where this seems to make a difference.)

I also don’t need enzymes normally if I was drinking water or diet soda.

I WOULD take enzymes for lows if I was consuming:

  • Peanut butter crackers (fat and protein need enzymes)
  • Hot chocolate made with milk (milk has fat and protein that need enzymes)
  • A snickers bar (fat and protein, needs enzymes) or other candy like this that has fat and protein alongside the carbs.

There’s probably also a threshold for which you need enzymes for an amount of carbs, but both because enzymes have a rather large dose of amylase in them in every pill; there is little (zero?) research on carb digestion needs with amylase among EPI research; and I have little anecdata to give you on exactly where this falls, I think it is less typical to run into this. Probably most of the time people get away with dosing for mostly fat and protein and carbs are mostly covered by that; but it’s also common for people to dose for fat and underdose for protein and get symptoms as a result.

  1. It gets better (EPI can be successfully treated) and we can track it with the EPI/PEI-SS

It does get better. This is an important message for people with EPI. It can be more challenging when there are multiple conditions going on at the same time, but EPI in and of itself can be managed with dosing enzymes.

And, we can track it getting better. I made the Exocrine Pancreatic Insufficiency Symptom Score (the EPI/PEI-SS) because there wasn’t a good symptom screening or tracking score for people with ALL types of EPI. (The existing ‘scores’ were developed only from people with EPI and chronic pancreatitis or cystic fibrosis and I found that the symptom list was too narrow and did not account for frequency and severity). It uses 15 symptoms and a self-rating of both frequency (0-5) and severity (0-3), and the resulting score can range from 0 to 225.

Like elastase, the ‘cutoff’ is not precise – based on an early real-world study of n=324 people, we found around 55 to be a cutoff indicating a greater likelihood of EPI. Can someone with a score of 45 have EPI? Yes. Can someone with a score of 62 not have EPI? Yes. It’s not that precise, but that doesn’t mean it’s not useful.

This symptom score can be used in a variety of ways:

  1. To increase the pre-test probability for subsequent elastase screening
  2. To help physicians and patients understand exactly what frequency and severity of symptoms the patient is dealing with.
  3. To be able to see over time with enzyme dosing how the symptom experience is changing.

For example, someone who was diagnosed with EPI had a symptom score of 72. After they went on enzymes and titrated their dosing, their score is now 7. That is a huge improvement and puts their symptom experience on par with people who don’t have EPI (or any other GI condition).

Another scenario may be someone who had more extreme symptoms – imagine an EPI//PEI-SS score of 155. They went on enzymes, and a few months later their score was 96. That’s an improvement, yay! But, it shows that they still likely have room for improvement and their enzyme dosing likely needs ongoing review.

(Note: the EPI/PEI-SS was evaluated in an online, real-world study of n=324 people, including a large number of people with diabetes. The diabetes sub-analysis is here, showing that having either type 1 or type 2 diabetes did not significantly influence the EPI/PEI-SS scores. We have an ongoing clinical study right now based at University of Florida in partnership with Dr. Brittany Bruggeman evaluating the use of the EPI/PEI-SS with elastase screening for people with diabetes. We also have an EPI/PEI-SS study running in New Zealand and Australia, with Dr. Amanda Landers, for people with pancreatic cancer. All this is to say that EPI/PEI-SS is being evaluated further for screening for EPI but we have not yet kicked off studies for using it for ongoing titration of enzymes or for capturing what ‘asymptomatic’ presentations may look like. This is an area I’d love to study, though, so if you are interested in studying its use alongside titration protocols for enzymes or asymptomatic presentations – please reach out to collaborate!)

—

A blue square with white text that says "What endocrinologists (and diabetes teams) should know about exocrine pancreatic insufficiency (EPI/PEI), a blog post by Dana M. Lewis from DIYPS.orgIf you read any or all of this – thank you for your interest in supporting people with exocrine pancreatic insufficiency. We need more of you! If I can help your work in any way – please reach out to me any time (Dana@OpenAPS.org).

TLDR summary:

  1. EPI occurs in people with diabetes and is more common than gastroparesis and celiac. Screen for EPI in people with diabetes with GI symptoms! Some patterns of unexplained low energy/fatigue and extreme glucose spikes and drops on CGM also might warrant EPI screening in PWD.
  2. Elastase is not perfect but what we can most easily access. Endocrinologists can test for it. A result around 200-ish or lower warrants an enzyme trial. Endos and diabetes teams should care about EPI because it impacts glucose levels!
  3. Give a reasonable dose of enzymes to start. Reasonable means 40-50,000 units of lipase or more. Endocrinologists could manage the enzyme prescription.
  4. Write an Rx for enzymes to take with food. Teach people to adjust their dose in relation to what they eat; the fact that it will take some time to figure out a dose; and encourage them to call back and ask for an adjusted Rx based on what they need.
  5. EPI sucks when untreated, but it does get better and we should continue pursuing better and adjusting enzyme doses until symptoms are improved.

Custom app for tracking medication trials and complicated medicine protocols (Protocol Pilot)

by Dana Lewis

It’s awesome and terrible at the same time in this day and age to be handed a medicine with a complicated protocol and given no ideal way to visualize or track the schedule and the feedback from the medicine. This can lead to “non-adherence” and “non-compliance” in clinical-speak, or “I don’t know if it’s working” or “I don’t know how and when I should take this” in patient-speak. For medication trials, it risks missing the signal of determining if it works. The awesome part is that now with AI, it’s easier than ever for someone to use AI to help them to think through what they need to track and customize it to fit their lifestyle and preferences to actually track what they are doing. But it’s terrible that we have to do that, and I recognize that not everyone will think to do this; have the time to do this, etc. 

When one of my doctors and I decided to try a new medication for me, the conversation circled around the fact that there is no data for my use case. (A recurring theme). So, using their best knowledge of basic protocols for this medication used for other conditions, they outlined the approach they recommended. Take dose A. Then take dose A twice a day. Then take dose A once and dose B the same day, separated by at least X hours. Then take dose B twice a day. You can take dose B eventually up to Z times per day. Let me know if it works for you. 

(Determining if it works for me is up to me, because it’s subjective.)

Even with all of my experience and resources, I came home feeling overwhelmed about implementing this medicine trial. Not in the least because the effect of this medicine is very time-specific, and also because the effect isn’t “do I feel better” but do certain things get better when using the medicine. I need to track those specific things. There’s the variable of taking a small dose to acclimate and minimize side effects and then ramping up; then there’s the assessing efficacy; then there’s trying to determine (if it’s working at all) what’s optimal.

It’s overwhelming to think about working my way through that. I need a spreadsheet for tracking, of course, except it’s complex enough that I wasn’t sure how to set up a spreadsheet for tracking this. There are too many variables to track (including non-med ones that influence the interpretation of efficacy)! Plus I need reminders to track outcomes at very specific time intervals. Like I built Scheduler Pilot when I needed the ability to move and reschedule treatments based on the relationship to each other, I need something more flexible than a spreadsheet, or manual phone alarms I have to set, that can track according to the relation to the time of the medication and tests I am doing.

An app it is, then. Luckily, I have enough experience recently with app development and various technical projects and applying this to niche, boutique or “custom” and “n=1” software that I know I can do this. I spent an hour with an LLM discussing all of the app specifications and building out a spec, then set the AI agent to work implementing this in an app and pushing a build to my phone for me to test and validate. It worked. I maybe spent an hour discussing; it spent 20 minutes building the first MVP; I spent 5 minutes using it and creating a list of feedback and tweaks; and the cycle continued. I now have a fully functioning medication trial app for complex medication protocols called Protocol Pilot, built and on my phone ready to go before the medication arrived at my door step. I can now track where I am progressing through the testing protocol, record time-specific feedback around different aspects of my unique scenario as well as medication side effects, and have all this data stored and reviewable not only in the app but the ability to export as json or a spreadsheet for review and analysis elsewhere. And, most importantly, there is a robust notification setup in this app. When I say I’m starting a task to test, it prompts me at the right intervals (there are multiple) to reflect and record any side effects of the med and whether the task is different with the medication peaking, plus any other time-anchored review periods I want to do. Instead of having to remember to track, it tracks for me when it’s time to track something. This is a huge benefit of doing this as an app rather than a spreadsheet.

Why “all this work” for a tracking app? Part of the reason it’s worth the “effort” is because I have learned that putting in the work up front to have a good system that works for me to track means I’ll actually track and record what matters. I dislike looking back and not being able to use tracked data because I didn’t track the right things, because it was burdensome. The whole point is building it into UIs that work for me.

The nice thing is, I don’t necessarily have to design it in a way that works for everyone else or indeed for anyone else. It lowers the threshold of action to build what works for me. Later, if someone else wants it, I can certainly share it and scale it and modify interfaces to support other use cases – but I don’t have to slow down a working prototype that works for me, when I need it, because I need to design for everyone else at the same time. And part of this is because the barriers to starting are so low, thanks to AI. (Another plug for you to try again with AI if you haven’t tried in a while, especially using agents and sub-agents where you can set goals and let it keep running and testing it’s work before it bugs you to review, so you can provide higher level direction without having to review line by line of code or fix trivial bugs.) 

Protocol-Pilot-tacking-medication-trials-and-complex-protocols-DanaMLewisThat being said, I do have in mind the ability for this app to scale and be used by other people, for example actually baking in an LLM to take an unstructured protocol (e.g. “take dose A then B then a plus b at XYZ time frames over ABC days, record JKL symptoms and watch for MNO side effects and record if those happen, when) and transform the app interface to reflect that protocol and when to record things. If this type of app sounds like something you would use for your use case – let me know and I’d love to collaborate with you to build in that feature and have you test it for your use case (whatever it is, you don’t have to tell me specifics). Email me any time – Dana+ProtocolPilot@OpenAPS.org – and let me know.

—
Update: a week later and I have already built in the ability to support different users (aka, anyone can use this: it’s not hardcoded to my use case) and a friend has already asked to try it. Plus, it supports multiple protocols/tests at once! Reach out if it’s something you’d like to see if it works for your use case.

Why I built another titration tool, this time for titrating subcutaneous IG (ScIG Pilot)

by Dana Lewis

Why is everything a titration problem; why does the traditional healthcare / research system not solve titration for so many therapies; and why do I always find myself having to do it?

Good questions that I won’t answer (because I can’t), but it is a good introduction into what I have been working on. I built a titration simulation tool for clinicians and people who want to explore different adjustments to dosing immunoglobulin, especially through subcutaneous methods. AKA, subcutaneous IG, AKA ScIG (rhymes with “rig”).

What exactly is the problem?

ScIG is studied in very few conditions – the approved FDA indications are generally for PI (primary immunodeficiency) and CIDP (chronic inflammatory demyelinating polyneuropathy). But, ScIG (and IVIG) are used to treat a lot of autoimmune conditions (often used off-label, because the few other conditions studied have a much smaller population studied and lower-quality evidence bases). Thus, there are no approval studies nor good data for these other condition use cases. Instead, there is a smattering of literature showing that occasionally it works here and there. The dosing regimens are based on the PI/CIDP studies and there are very few of those studies beyond product-specific studies. ScIG has mostly been tested at a 7 day interval (meaning once per week) and there’s one study checking to see if you do it every 14 days (every two weeks) but double the dose to maintain the total dose per month, you get about the same outcome…that is, in people with PI, which means they were measuring infections. That doesn’t answer the “does it work” question for most other use cases for ScIG, which is a variety of other autoimmune diseases. It also doesn’t appear that they tested lower doses (?), so we can’t tell if the same dose over 14 days was effective because they doubled the weekly dose for the 14 day regimen they studied.

But does it work well if you do the same dose every 9-10 days? No one can tell you that, it hasn’t been studied, so if a patient is struggling with every 7 days scheduling, they may benefit from a schedule change, but there’s no data or guidance from research for the clinician to help estimate whether that may be ok, or a problem. Or, what happens if you slightly drop your dose by 10mL? Or what happens if you split your dose across 14 days? What happens if your dosing is evenly split over time or very differently split based on the pre-filled cartridges you have available?

Clinicians don’t often seem to prescribe ScIG to very many patients, so they also don’t have a strong feedback loop of data on alternative dosing regimens. And most indications are off-label and diverse, so that further dilutes that data stream: what works for someone could be unique to their disease rather than useful to extrapolate to everyone on ScIG.

Why does this matter?

ScIG is dosed based on body size. If you have a larger body, you have a larger dose. You are limited to how much you can infuse per site which influences how many needles you use. Needles have a risk of causing bruises and lumps (resolving to small clots of tissue that can sometimes take weeks to months to go away), so in some cases fewer needles is ideal, but this then drives up the size of the fluid depot which can be painful due to skin stretching and the amount of fluid that has to be cleared. It is a tradeoff between all of these factors. And, over time, people may have less skin real estate for various reasons. But, there’s no tool that helps patients make this kind of tradeoff. Current dosing calculators can help with dose and volume and figuring out infusion times, and tracking apps can record what happened, but they usually do not show what different schedules might mean over time. For example, whether doing smaller amounts less often changes the overall amount dosed, whether this creates bigger highs and lows in IgG levels, or simply reduces the number of sites. There’s also the question of how much time it takes for these varied dose schedules given the tradeoffs where fewer needles means larger depots and slower infusion times.

I wanted a tool to make those trade-offs easier to see. I wanted to be able to compare possible schedules side by side, showing the expected pattern over time, the amount per week, the number of infusion sites, the volume per site, and also model what the impacts are for the IG itself.

ScIG Pilot

ScIG dosing decisions often involve trade-offs among dose interval, total dose intensity, infusion volume, number of sites, per-site volume, and patient tolerability, but existing product calculators don’t cover this. They generally are designed for clinicians or pharmacists and focus on narrow tasks such as converting weight-based dose to volume, estimating sites or infusion time, or supporting labeled weekly/biweekly schedules. Published PK models describe ScIG absorption, peak/trough behavior, and exposure under standard regimens, but these are not available as a practical tool for comparing based on real-world constraints.

ScIG Pilot is a literature-informed simulator that compares candidate ScIG schedules against a reference regimen (aka a standard 7-day dosing schedule), showing normalized exposure curves, predicted peak/trough patterns, weekly-equivalent dose intensity, site burden, dose volume per site, and infusion frequency. (It is also open source.)

It can help someone think through a scenario and ask these types of questions of a proposed change:

  1. If I change the schedule, how does this change the volume I am getting over time? How does this compare to the existing schedule?
  2. When might this dose change impact the body?
  3. What is the burden trade-off: sites, dose per site, infusion days, time per infusion, longest gap? How does this stack up over a month or a year?

Then, based on outcomes (does it work) and various preferences (less time; fewer sites; smaller fluid depots; etc), someone can decide what works for their scenario, or at least have visuals to help them think through a decision and understand various tradeoffs. This is also helpful when trying to estimate out to longer time frames, for example: not just what this regimen does in a month in terms of number of sites, number of infusion days etc; but also how does this translate to a year of this therapy in terms of number of infusions and sites, etc?

–

ScIG Pilot: a subcutaneous immunoglobulin titration tool, a blog post by Dana M. Lewis on DIYPS.orgIt can be really hard to titrate these types of therapies and infusions, because there is no good data. I’d love to see studies in future collect this type of data or have repositories (like I described here) that are not limited to one disease indication, so a patient or clinician could go and research across populations to better get some data to triangulate from to better inform a subsequent decision to go on a therapy or change a treatment regimen. Right now, there are no visuals, no data, and no tools. But now, for this, there is at least ScIG Pilot to help visualize and triangulate across what little data we do have.

(And it’s open source so if you have ideas to improve ScIG Pilot or suggestions to make the models more useful, please feel free to open issues or PRs or email me to discuss – Dana+ScIGPilot@OpenAPS.org)

Want signal? We need more noise (more examples to address the quiet bottleneck)

by Dana Lewis

Note: I’m assuming if you are reading this post that you’ve first read this post, summarizing the concept. The below is a companion piece with some expanded details about the concepts and more examples of how I think addressing this bottleneck will help make a difference.

—

You might hold the perspective that in the growing era of AI, there’s too much noise already. Slopocalypse. (Side note: this entire post, and my other post, was written by me, a human.) That’s not the biggest problem in healthcare, though: in both research and clinical care, so much critical data is simply not collected. We are missing sooooo much important data. Some of this is an artifact of past clinical trial design and how it was hard to collect or analyze or store the data; there were less established norms around data re-use; and some of it is a “collect the bare minimum to study the endpoints because that’s all we are supposed to do” phenomenon.

Nowadays, we should be thinking about how to make data and insights from clinical work and research available to AI, too, because AI will increasingly be used by humans to sort through what is known and where the opportunities are, plus make cross-domain connections that humans have been missing. And we need to be thinking about whether the incentives are set up correctly (spoiler: they’re not) to make sure all available data is able to be collected and shared or at least stored for humans and AI to have access to for future insight. What is studied is also ‘what is able to be funded’, which is disproportionately things that come to the commercial market eventually after regulatory approval.)

“But what’s the point?” you ask. “If no one is looking at the data from this study, why collect it?” Because the way we design studies – to answer a very limited-scope question, i.e. safety and efficacy for labeling claims and regulatory approval – is very different than the studies we need around optimizing and personalizing treatments. If you look at really large populations of people accessing a treatment –  think GLP-1 RA injectables, for example – you eventually start to see follow-on studies around optimization and different titration recommendations. But most diseases and most treatments aren’t for a population even 1/10th the size of people accessing those meds. So those studies don’t get done. Doctors don’t necessarily pay attention to this; patients may or may not report relevant data about this back to the doctor (and again even if they do: doctors don’t mentally or physically store this data, often); and the signal is missed because we don’t capture this ‘noise’.

This means people with rare diseases, undiagnosed diseases, atypical or seronegative diseases, unusual responses to treatments, multiple conditions (comorbidities), or any other scenario that results in them being part of a small population and where customizing and individualizing their care is VERY important…they have no evidence-based guidance. Clinicians do the best they can to interpret in the absence of evidence or guidelines, but no wonder patients often turn to LLMs (AI) for additional input and contextualization and discussion of the tradeoffs and pros and cons of different approaches.

We have to: the best available human (our clinicians) may not have the relevant expertise; or they may not be available at all; or they may be biased (even subconsciously) or forgetful of critical details or not up to date on the latest evidence; or they may be faced with a truly novel situation and they don’t have the skills to address it because they’re used to cookie-cutter standardized cases. This is not a dig at clinicians, but I recognize that we now have tools that can address some of the existing flaws in our human-based healthcare system. I keep talking about how we need to recognize that evaluations of AI in healthcare shouldn’t treat the status quo as the baseline to defend, because the status quo itself has problems, as I just described.

And AI has some of the same problems. (Except for the ‘not available’ part, unless you consider the lower-utility free access models to mean that the more advanced, thinking-based models are ‘not available’ because of cost access barriers). An AI may not have any training data on a rare disease… because nothing exists. It may drop information out of the context window, and we may not realize that this has happened (i.e., it ‘forgets’ something). Usually these are critiques of AI, juxtaposed against the implication that humans are better. But notice that these critiques are the same of humans, too! This happens all the time with human clinicians in healthcare. A human can’t make decisions on data that doesn’t exist in the world, either!

So…how do we “fix” AI? Or, how do we fix human healthcare? We should be asking BOTH questions. Maybe the answer is the same: increase the noise so we increase the signal. Argue about the ratio later (of signal:noise), but increase the amount of everything first. That is most important.

How might we do this? I have been thinking about this a lot, and Astera recently posted an essay contest asking for ideas that don’t fit the current infrastructure. (That’s why I’m finally writing this up, not because I think I’ll win the essay contest, but mostly because i’s an opportunity for people to consider whether/not this type of solution is a good idea as a separate analysis from “well, who is going to fund THAT?”. Let’s discuss and evaluate the idea, or riff on it, without being bogged down by the ‘how exactly it gets funded and managed over time’.)

I think we should create some kind of digitally-managed platform/ecosystem to do the following:

  1. Incentivize and facilitate written (AI-assisted allowed) output of everything. Case reports and scenarios of people and what they’re facing. All the background information that might have contributed. All the data we have access to that is related to the case, plus other passive, easy-to-collect data (that may already be collected, e.g., wearables and phone accelerometer data) even if it does not appear to be related. Patient-perspective narratives, clinical interpretations, clinical data, wearable data – all of this.

    A.) And, be willing to take variable types of data even from the same study population. For example, as a person with type 1 diabetes, I have 10+ years of CGM data. For a future study on exocrine pancreatic insufficiency, for example, not everyone will have CGM data, but because CGM data is increasingly common in people with T2D (a much larger population) and in the general population, a fraction of people will have CGM data and a fraction of people are willing to share it. We should enable this, even if it’s not for a primary endpoint analysis on the EPI study and even if only a fraction of people choose to share it – it might still be useful for subgroup analysis, determining power for a future study where it is part of the protocol, and identifying new research directions!

  2. Host storage somewhere. There can be some automated checking against identifiable information and sanity checking what’s submitted into the repository, plus consent (either self-consent or signed consent forms for this purpose.)
  3. Provide ‘rewards’ as incentives for inputting cases and data.

    A) Rewards might differ for patients self-submitting data and researchers and clinicians inputting data and cases. Maybe it’s one type of reward for patients and a batch reward of a free consult or service of some kind for researchers/clinicians (e.g. every 3 case reports submitted = 1 credit, credits can be used toward a new AI tool or token budget for LLMs or human expertise consult around study design, or all kinds of things.)

    B) Host data challenges where different types of funders incentivize different disease groups or families of conditions to be added, to round out what data is available in different areas.

  4. Enable AI and human access (including to citizen scientists/independent researchers who don’t have institutions) to these datasets, after self-credentialing and providing a documented use case for why the data is being accessed.

    A) Require anyone accessing the dataset to make their analysis code open source, or otherwise openly available for others to use, and to make the results available as well.

    B) Tag anytime an individual dataset is used by a project, that way individuals with self-donated data (or clinicians submitting a case) can revisit and see if there are any research insights or data analysis that applies to their case.

    C) Provide starter projects to show how the data can be used for novel insight generation. For example, develop sandboxes for different types of datasets with existing ‘lab notebooks’, so to speak, to onboard people to different datasets/groups of data, different types of analyses they might do, and to cut down on the environment setup barriers to getting started to analyzing this data.

    D) Facilitate outreach to institutions, disease-area patient nonprofits and advocacy groups both to solicit data inputs AND to use the data for generating outputs.

Why should we do this?

  • Clinical trials do not gather all of the possible useful data, because it’s not for their primary/secondary endpoints. Plus, they would have to clean it upon collection. Plus storage cost. Plus management decisions. Etc. There’s a lot of real barriers there, but the result is that clinical trials do not capture enough or the most possible relevant data. So we need a different path.
  • Clinicians/clinical pathways don’t have ways to review data, so they avoid doing it. There’s no path to “submit this data to my record but I don’t expect you to look at it” in medical charts (but there should be). So we need a different path.
  • Disease groups/organizations sometimes host and capture datasets, but like clinical trials, this is limited data; it often comes through clinical pathways (further limiting participation and diversity of the data); it doesn’t include all the additional data that might be useful. So we need a different path.
  • Some patients do publish or co-author in traditional journals, but it’s a limited, self-selected group. Then there are the usual journal hurdles that filter this down further. Not everyone knows about pre-prints. And, not everyone knows how useful their data can be – either as an n=1 alone or as an n=1*many for it to be worth sharing. A lot of people’s data may be in video/social media formats inaccessible to LLMs (right now) or locked in private Facebook groups or other proprietary platforms. So we need a different path.

Thus, the answer to ‘why we should do this’ is recognizing that: AI does not create magic out of nowhere. It relies on training data and extrapolating from that, plus web search. If it’s not searchable or findable, and it’s not in the training data, it’s not there. Yes, even the newer models and prompting it to extrapolate doesn’t solve all of these problems. What AI can do is shaped disproportionately by formal literature, institutional documents, and whatever scraps of publicly available content remain. If we want to shape what’s possible in the future, we need to start now by shaping and collecting the inputs to make it happen. We can’t fix the past trial designs but we can start to fill in the gaps in the data!

Here is an example of how this might work and why it matters to build and incentivize this type of data sharing.

  1. If we build and incentivize data sharing, the following individuals might self-donate their data and write-ups, or a clinician might submit them. Later, someone might analyze the data and put the pieces together.
  • Someone with type 1 diabetes (an autoimmune condition) and this new-onset muscle-related issue. Glucose levels are well-managed, and they don’t have neuropathy/sensory issues (which are common complications of decades of living with type 1 diabetes), and muscle damage and inflammation markers are normal. MRIs are normal. The person submits their data which includes their lab tests, clinical chart notes (scrubbed for anonymity), a patient write-up of what their symptoms are like, and exported data from their phone with years of motion/activity data.
  • A different person with Sjogren’s disease (also an autoimmune condition) and a similar new-onset muscle-related issue. There are known neurological manifestations or associations with Sjogren’s, but more typically those are small-fiber neuropathy or similar. The symptoms here are not sensory or neuropathy. MRIs don’t show inflammation or muscle atrophy. Their clinician is stumped, but knows they don’t see a lot of patients with Sjogren’s and wonders if there is a cohort of people with Sjogren’s facing this. The clinician asks the patient and gains consent; scrubs the chart note of identifying details, and submits the chart notes, labs, and a clinician summary describing the situation.
  • Later, a researcher (either a traditional institutional-affiliated researcher OR a citizen scientist, such as a third person with a new-onset muscle-related issue) decides to investigate a new-onset muscle-related issue. They register their hypothesis: that there may be a novel autoimmune condition that results in this unique muscle-related issue as a neuromuscular disease (that’s not myasthenia gravis or another common NMJ disease) that shows up in people with polyautoimmunity (multiple autoimmune conditions), but we don’t know which antibodies are likely correlated with it. The research question is to find people in the platform with similar muscle-related conditions and explore the available lab data to help find what might be the connecting situation and classify this disease or better understand the mechanism.
  • They are granted access to the platform, start analyzing, and come across an interesting correlation with antibody X, which is considered a standard autoimmune marker but one that doesn’t differentiate by disease, and which does highly correlate with this possibly novel muscle condition when it exists and is elevated in people with at least one existing autoimmune condition and these muscle-symptoms that are seronegative for every other autoimmune and neurologic condition. Further, by reading the clinical summary related to patient B and the self-written narrative from patient A, it becomes clear that this is likely neuromuscular – stemming from transmission failure along the nerves – and that the muscles are the ‘symptom’ but not the root cause of disease. This provides an avenue for a future research protocol to 1) allow these types of patients to be characterized into a cohort so this can be determined whether it is a novel disease or a subgroup of an adjacent condition (e.g., a seronegative subgroup);  2) track whether the antibody levels are treatment-sensitive or not or stay elevated always; and 3) cohorts of treatments that can be trialed off-label because they work in similar NMJ diseases even though the mechanism isn’t identical.
  • The mechanism for this novel disease isn’t proven, yet, but in the face of all the previous negative lab data and neurological testing patient A and patient B have experienced, it narrows it down from “muscle or neuromuscular” which is a significant improvement from their previous situations. Plus, this provides pathways for additional characterization; research; and eventual treatment options to explore versus the current dead ends both patients (and their clinical teams) are stuck in. And because there are no good clinical pathways for these types of undiagnosed cases, this type of insight development across multiple cases would not have occurred at all without this database of existing data.

2. The above is a small-n example, but consider a large dataset where there are hundreds or thousands of people with CGM data submitted. Plus meal tracking data, because people can export and provide that from whatever meal-logging apps some of them happen to use.

  • By analyzing this big dataset, an individual researcher could hypothesize that they could build a predictor to identity the onset of exocrine pancreatic insufficiency, which can occur in up to 10% of the general population (more frequently in older adults, people with any type of diabetes, as well as other pancreas-related conditions like pancreatitis, different types of cancer, etc), by comparing increases in glucose variability that correlate with a change in dietary consumption patterns, notably around decreasing meal size and eventually lowering the quantity of fat/protein consumed. (These are natural shifts people make when they notice they don’t feel good because their body is not effectively digesting what they are eating). They analyze and exclude the effect of GLP1-RA’s and other medications in this class: the effect size persists outside of medication usage patterns. This can later be validated and tested in a prospective clinical trial, but this dataset can be used to identify what level of correlation between meal consumption change and glucose variability change happen over what period of time in order to power a high-quality subsequent clinical trial. This may lead to an eventual non-invasive method to diagnose exocrine pancreatic insufficiency through wearable and meal-tracking data. (None exists today: only a messy stool test that no one wants to do and is hampered by other issues for accuracy.)

–

These are two examples and show how even small-n data or a large dataset where there are additional subgroups with additional datasets can be useful. In the past, it’s been a “we need X, Y, Z data from everyone. A, B, C would be nice to have, but it’s hard and not everyone will share it (or is willing to collect it due to burden), so we won’t enable it to be shared for the 30% of people who are willing”. Thus, we lose the gift and contributions from the people who are able and willing to share that data. Sometimes that 30% is a small n but that small n is >0 and may be the ONLY data that will eventually answer an important future research question.

signal-noise-more-examples-DanaMLewisWe are missing so much, because we don’t collect it. So, we should collect it. We need a platform to do this outside of any single disease group patient registry; we need to support clinician and patient entries into this platform; we need to support intake of a variety of types of data; and we need to have low (but sensible) barriers to access so individuals (citizen scientists, patients themselves) can leverage this data alongside traditional researchers. We need all hands on deck, and we need more data collected.

Want signal? We need more noise (looking at the quiet bottleneck)

by Dana Lewis

We need more signal, which means we want more noise. A lot of current scientific infrastructure is designed to minimize messiness: define a narrow question, collect the minimum data required to answer it, standardize the dataset, exclude complicating variables, finish the analysis, publish the result. That approach is understandable. It is also one reason we repeatedly fail the patients who most need evidence: people with unusual responses, multiple conditions, atypical phenotypes, rare diseases, or combinations of features that do not fit cleanly into existing bins. Our systems today are structurally designed to discard or never capture the kinds of heterogeneous, partial, contextual, or longitudinal data that could eventually make critical insights available to us.

My hypothesis is that one important scientific bottleneck is not a lack of intelligence, or even a lack of data in the abstract, but a lack of infrastructure for accepting, preserving, and reusing the kinds of data that fall outside formal trial endpoints and standard clinical workflows. I am proposing a solution: a shared platform that accepts optional, heterogeneous, participant- and clinician-contributed data, which will generate clinically useful subgroup hypotheses that standard trial and registry structures fail to generate.

Here are the quiet bottlenecks in the current system that I run into and that this platform would address:

  1. Atypical, comorbid, and small-population patients are systematically left without usable evidence because current research systems are optimized to suppress heterogeneous data rather than preserve it. (Clinical trials are usually designed around narrow endpoint collection, fixed inclusion and exclusion criteria, and standardized datasets that are tractable for a specific immediate question. Patients are often told there is “no evidence,” when what is really missing is a system that could have captured and preserved evidence that can help with future translation for atypical or edge-case presentations.)
  2. Useful data is routinely discarded because no existing structure is responsible for capturing it. (Clinical trials generally do not want to collect data outside primary and secondary endpoints because of cleaning, storage, analysis, and governance costs. Clinical care systems do not want to ingest large volumes of patient-generated data that clinicians are not expected to review. Disease registries are usually narrow and disease-specific. Journals are poor infrastructure for surfacing up data. There is no home for this data.)
  3. AI and humans are constrained by the same missingness problem: the knowledge base is biased and uneven because we do not capture enough of the right kinds of real-world data. (LLMs and other AI systems are often criticized for bias, holes, and nonrepresentative training data. But this overlooks that clinicians and researchers are also reasoning from the same incomplete literature, selective trial populations, under-collected real-world evidence, and publication-filtered case reports. The bottleneck is upstream of both.)AI now makes it possible to overcome these constraints, but only if the data is collected and made available in the first place.
  1. Current systems privilege uniform completeness over partial but valuable contribution, which causes preventable signal loss. (If not everyone has or is willing to provide the data, it’s not collected. Subsets of participants who are willing and able to contribute additional data, such as wearables, CGM, meal logs, phone sensor streams, or symptom data, are prevented from doing so.)

Why current structures produce these bottlenecks

No single existing institution is responsible for optional data intake, long-term stewardship, and broad downstream access. And different structures prioritize different data capture, with no harmonization between these two settings. Clinical trials are funded, regulated, and staffed to answer bounded questions. Clinical trial incentives reward endpoint completion, analyzable datasets, and publication-ready results. Anything beyond the core clinical trial protocol creates additional costs: more data cleaning, more storage, more governance, and more analytic work that may not directly serve the trial’s primary purpose. Under those conditions, narrowness is rational but it is also systematically lossy. Clinical care systems have a different but related design problem. Electronic health records are built for documentation, care coordination, and billing, not for capturing or allowing participant-generated, future-use data that a clinician does not need for real-time decision making. This illustrates the infrastructure gap: patients may have useful longitudinal data, but there is often no legitimate pathway to store it in a way that supports future analysis. Disease registries and advocacy-group datasets help in some cases, but they are typically narrow in scope, tied to a disease silo, and constrained by the same pressures toward standardization and limited datastreams. Publications are also poorly matched to this problem, because they are optimized for polished outputs, not for preserving heterogeneous data. Much of the infrastructure was shaped from prior underlying constraints (when data capture, storage, and analysis was more expensive).

Testable hypotheses that would address these challenges:

  1. Allowing partial, nonuniform data donation from participants will outperform “uniform minimum dataset only” models for early discovery in rare, atypical, and comorbid populations.
  2. Structured case narratives combined with quantitative data will enable more useful discovery than quantitative data alone for poorly characterized conditions, because the narrative context helps identify mechanistic or phenotypic patterns that are otherwise lost.
  3. Modern tools (LLMs etc) make it feasible to analyze and normalize real-world participant-contributed data at a scale and cost that was previously impractical.
  4. For some under-characterized conditions, a heterogeneous real-world repository can identify candidate biomarkers, phenotypic subgroups, or prospective-study designs and lead to a shorter timeline for protocol development and funding of eventual trials, compared to the status quo.

Here are a few example experiments we can use to validate these hypotheses:

  1. Use heterogenous real-world data to test whether a shared repository can generate an earlier detection hypothesis that existing structures are poorly positioned to generate. For example, exocrine pancreatic insufficiency. EPI is common but underdiagnosed and its current diagnostic pathway is unpleasant (a stool test) and imperfect. As digestion becomes less effective, people often change what and how they eat before anyone labels it as a problem: meal sizes may shrink, fat and protein consumption may drift downward, symptom patterns develop, and glucose variability may shift in parallel, especially in people with diabetes or others who happen to have CGM data. A conventional clinical trial would rarely collect all of these data together (GI symptoms, meal logs, and CGM data). We have shown that a patient-developed symptom survey can effectively distinguish between EPI and non-EPI gastrointestinal symptom patterns in the general population. But this relies on people to know they have a problem and fill out the symptom survey to assess their symptoms. By analyzing CGM and meal logging data, we may be able to create a diagnostic signal from CGM data that may provide an earlier detection of EPI or other digestive problems, noninvasively and much earlier. If successful, the output would be a concrete, testable hypothesis for a future prospective study: for example, that a specific combination of changing meal patterns and glucose variability could serve as an early noninvasive screening signal for EPI.
  2. A second, smaller experiment would test the same infrastructure in a very different setting: under-characterized autoimmune or neuromuscular overlap syndromes. Here the problem is not underdiagnosis at scale, but invisibility in small-n edge cases. Patients with unusual muscle-related symptoms, normal or ambiguous standard workups, and backgrounds that include autoimmune disease often remain isolated within separate clinics and disease silos. One person may carry a type 1 diabetes diagnosis, another Sjögren’s, another a different autoimmune history, and yet their new symptoms may share an overlooked mechanism. In current systems, these cases rarely become legible as a group because the relevant data are scattered across chart notes, patient stories, normal imaging, lab panels, and passive longitudinal data that no one is collecting or comparing systematically. This tests the same core hypothesis under tougher conditions: smaller numbers, less uniform data, less obvious endpoints, and a heavier dependence on narrative context. If the EPI case shows that optional heterogeneous data can support tractable hypothesis generation in an underdiagnosed condition, the neuromuscular case shows why the same infrastructure could matter even more in sparse-data, high-ambiguity situations where it is even harder to capture data to generate evidence in support of and funding for subsequent trials.

What we might learn if these fail

We will learn what the barrier is, whether it is still a problem of infrastructure; a lack of the right people (or AI) leveraging the data; whether partial nonuniform data donation is operationally feasible; and whether limiting factors are data availability, harmonization, governance, or analytic quality. Plus, we can determine whether certain diseases or use cases (e.g. developing novel diagnostics versus assessing medication responses) are better suited than others for this type of platform.

Why now?

Passive and participant-generated data collection is easier. Wearables, phone sensing, CGM data, meal logging apps, symptom trackers, and similar are now significantly more common. Technology makes it easier than ever to create custom apps to track n=1 data or study-specific data. Storage is cheaper. Technology improvements, most notably AI tools, have made it more tractable to collect data; and have made it more tractable for researchers to analyze this data. The remaining bottleneck is capturing, storing, and making the data available. It is less of a technological bottleneck and instead a bottleneck of funding/governance/etc. This is addressable now that the capture/analysis barriers have been lowered!

–

signal-noise-DanaMLewisI don’t think the question we should be asking is whether every piece of heterogenous data will be useful, but whether we can afford to keep doing what we are doing (throwing away data and still expecting discovery for the populations our current systems and infrastructure already quietly fail).  

–

Note: I am submitting this post to the Astera essay contest, which you can read about here. You should write up your ideas about the bottlenecks you see and submit as well! I also wrote an additional piece with more details and examples, which you can read here. 

A new symptom score for people with exocrine pancreatic insufficiency (the EPI/PEI-SS)

by Dana Lewis

One of the frequent complaints in the literature about exocrine pancreatic insufficiency (known as EPI in some parts of the world, or PEI for pancreatic exocrine insufficiency elsewhere) is that the symptoms are not specific and they can overlap with other conditions. Diarrhea, for example, can happen from a lot of conditions and a lot of medications. Not everyone with EPI has diarrhea, though. Another problem is that there are other symptoms that occur in EPI other than diarrhea and weight loss, but there’s not been any data on which groups of people experience which types of symptoms with EPI, or how common the other symptoms are, so they often aren’t listed. This leads to a cycle of lack of awareness, lack of screening, lack of diagnosis, and lack of treatment.

There’s been little effort to date to solve this problem, and I found myself wondering if we as patients, who experience the symptoms directly, could find a way to address this. Between my systematic review papers (where I’ve read hundreds of papers about the symptoms & diagnostic approaches to EPI) and personal experience with EPI, I made a list of 15 symptoms. But it’s not just about which symptoms people have: that’s where the overlap problem comes in. With EPI, many people have a lot of symptoms, a lot of the time, and they are VERY annoying. So the frequency and severity of the symptoms are a hallmark as well. I put together a way to quantify the frequency and severity (using plain language)of symptoms, and the EPI/PEI-SS (Exocrine Pancreatic Insufficiency Symptom Score) was born.

With help from more than a dozen people, some with EPI and some who didn’t have EPI, I ran a pilot test with the symptom score to see if the people with EPI would generate scores, the way I did, and whether people without EPI (and with either everyday gastrointestinal symptoms, or other conditions that sometimes cause GI symptoms) would have scores to match. They did not: it was a stark difference, and there wasn’t any overlap. The EPI symptom burden was quantifiably much higher than everyday GI symptoms for someone without a condition, and also higher compared to people with other conditions with GI symptoms (think food intolerances, IBS, other non-EPI GI conditions).

So I launched a bigger study that many of you participated in (thank you!), with the goal of exploring whether this score would be useful in the general population to help distinguish EPI from other conditions and whether it might possibly aid in screening for EPI.

And now, the results are published! (You can read the full open access paper here: https://doi.org/10.3390/epidemiologia6030048).

Here’s what we learned:

There were 324 participants at the time I cut off data collection for the analysis (after three weeks). This included 155 people who identified as having EPI, and 169 people without EPI. Everyone answered whether or not they had any of the 15 symptoms (falling into three groups: abdominal, toilet-related, and food-related symptoms) and indicated frequency and severity. Multiplying frequency (0–5) x severity (0–3) by each of the 15 symptoms, the EPI/PEI-SS score range is 0–225. (See Table 1 for a list of the symptoms and rating description).

The key finding: people with EPI had higher scores than people without EPI.

In this real-world study, the mean total score of those with EPI was 98.11 (min 1, max 213), in contrast to a mean total score of 38.86 for those without EPI (min 0, max 163). The difference is practically as well as statistically (p<.001) significant.

Figure 1 from the paper showing the sub-scores and total scores broken out by EPI and non-EPI groups, respectively.

Even when I separated the people without EPI into two groups, those with other gastrointestinal conditions and those without, the scores were still distinct and statistically significant from the people with EPI. I also did a sub-analysis of each individual condition and none had a significant impact on the overall score. (Because there are so many people with diabetes in my network who participated in the study, I also ran a separate sub-analysis to deeply analyse the contributions of type 1 diabetes and type 2 diabetes – and made a separate paper on this analysis, which is also open access and available to read here.) Also in the bucket of “things that did not affect the score” was age. However, females in the study reported higher scores compared to males (this matches other studies showing a higher gastrointestinal burden, so this isn’t necessarily unique to EPI).

In addition to the overall score, you can see the difference by looking at the number of symptoms people reported and the difference in frequency and severity:

  • EPI group: 12.39 symptoms, average frequency 3.02, average severity 1.73
  • Non-EPI group: 8.15 symptoms, with nearly half the frequency (1.55) and severity (0.91)
Figure 3 from the paper, showing each of the 15 symptoms and the range of scores for the EPI group (purple) and non-EPI group (blue), respectively.

Nerdy notes (you can read more in the full paper): Cohen’s d (1.475) indicated a large effect size; all comparisons overall and across sub-groups and across symptom categories were statistically significant (p<0.001). Cronbach’s alpha for sub-score categories was “good” (0.88 abdominal, 0.83 toilet, 0.88 food), indicating high internal consistency and good construct validity. Using an EPI/PEI-SS cutoff of 59 (out of possible 225), area under the curve was 0.85, sensitivity was 0.81, and specificity was 0.75.

Were there limitations for this study? You bet. It was online and based on people who happened to fill it out, so follow up studies will help confirm these results in different populations to confirm if it is representative of the average EPI experience. (Note though that this study population did have a lot more diversity of people with EPI, though, compared to most other EPI-related symptom assessment studies, which are often limited to chronic pancreatitis and cystic fibrosis, and/or pancreatic surgery/cancer.) There was a large number of people with diabetes who participated, in part because of my network and where I recruited participants from – however, as seen in this sub-study, presence of diabetes (any type, or split in type 1 and type 2) did not influence scores (analysis here). This study was also exploratory, meaning it was not powered for a specified outcome. We’ve now been able to use this data to power follow up studies, now that we know what to expect score-wise in people with and without EPI!

What should you take away from this study?

If you are a person with some kind of gastrointestinal symptoms, you can use the EPI/PEI-SS to explore your symptoms and quantify them based on frequency and severity. If your score is near or above the cutoff, you may want to consider discussing your symptoms with your doctor and exploring whether testing for EPI (often fecal elastase testing) is warranted. This tool hasn’t been validated as a diagnostic method, but this data can help the shared decision making process and hopefully also aid you in a better conversation with your doctor as you explore pathways to solutions.

The EPI/PEI-SS is available online, for free, and you can use it right now: https://danamlewis.github.io/EPI-PEI-SS/

If you are a person with EPI and you are still struggling with symptoms of EPI (PEI), you may find it handy to take the EPI/PEI-SS to document your symptom burden. Then, as you adjust your enzyme dosing, you can periodically take the EPI/PEI-SS again (every few weeks or months) and use it to help you track whether things are improving. You can use the web version, or if you want to also track your enzyme (PERT) dosing, you can use the EPI/PEI-SS in both the iOS (https://bit.ly/PERT-Pilot-iOS) and Android (https://bit.ly/PERT-Pilot-Android) versions of “PERT Pilot”. Then you can see your scores and view them over time in the same place.

Note that the scores of people with EPI in the study don’t mean that ‘this is as good as it gets’ when you go on enzymes. Many people with EPI indicate that they feel they are not dosing enough enzymes (see this study); the scores on the EPI/PEI-SS reflect this. It is possible for people with EPI to get scores in the non-EPI range, once enzymes are regularly dosed to match what you’re eating. (For example, my score went from well above the cutoff to well below the average non-EPI score once I started enzymes.)

If you are a doctor, take a look at the EPI/PEI-SS (see links, or Table 1 in the paper) so you know what some of the symptoms of EPI are. Notably, be aware that diarrhea and weight loss are not the only symptoms of EPI. In the diabetes sub-study, for example, we found food-related behaviors to be a key variable, as many people intuitively adjust what or how they are eating to try to eliminate symptoms on their own. Pain is not prominent in all corners of the EPI community (it’s more common among people with pancreatitis). Feel free to have patients use the EPI/PEI-SS any time and use it as part of your shared decision making process.

A new symptom score for exocrine pancreatic insufficiency: new research on the EPI/PEI-SS (a blog by Dana M. Lewis on DIYPS.org)If you have any feedback (for example, if it’s been helpful or not), you can email me any time (Dana+EPI-PEI-SS@OpenAPS.org). I’d also love to collaborate, if you’re interested in partnering on any research studies. We have some ongoing studies in different countries (US, Ireland, New Zealand, Australia) in different populations (general population; people with diabetes; people with pancreatic cancer; etc) and I’m looking forward to partnering with other researchers on additional validation studies and exploring if and how the EPI/PEI-SS can help us address some of the gaps of real-world clinical practice and life with EPI.

If you’re a researcher with shared interest in EPI…ditto the above!

—

Read the research referenced in this blog post: https://doi.org/10.3390/epidemiologia6030048

—

Cite it: Lewis DM, Landers A. Development of Novel Symptom Score to Assist in Screening for Exocrine Pancreatic Insufficiency. Epidemiologia. 2025; 6(3):48. https://doi.org/10.3390/epidemiologia6030048

—

Questions? Please comment below!

If you have EPI-specific questions, you might also like this blog post with 25 questions and answers about EPI (PEI) ranging from symptoms and diagnosis to treatment and dosing titration.

Facing Uncertainty with AI and Rethinking What If You Could?

by Dana Lewis

If you’re feeling overwhelmed by the rapid development of AI, you’re not alone. It’s moving fast, and for many people the uncertainty of the future (for any number of reasons) can feel scary. One reaction is to ignore it, dismiss it, or assume you don’t need it. Some people try it once, usually on something they’re already good at, and when AI doesn’t perform better than they do, they conclude it’s useless or overhyped, and possibly feel justified in going back to ignoring or rejecting it.

But that approach misses the point.

AI isn’t about replacing what you already do well. It’s about augmenting what you struggle with, unlocking new possibilities, and challenging yourself to think differently, all in the pursuit of enabling YOU to do more than you could yesterday.

One of the ways to navigate the uncertainty around AI is to shift your mindset. Instead of thinking, “That’s hard, and I can’t do that,” ask yourself, “What if I could do that? How could I do that?”

Sometimes I get a head start by asking an LLM just that: “How would I do X? Layout a plan or outline an approach to doing X.” I don’t always immediately jump to doing that thing, but I think about it, and probably 2 out of 3 times, laying out a possible approach means I do come back to that project or task and attempt it at a later time.

Even if you ultimately decide not to pursue something because of time constraints or competing priorities, at least you’ve explored it and possibly learned something even in the initial exploration about it. But, I want to point out that there’s a big difference between legitimately not being able to do something and choosing not to. Increasingly, the latter is what happens, where you may choose not to tackle a task or take on a project: this is very different from not being able to do so.

Finding the Right Use Cases for AI

Instead of testing AI on things you’re already an expert in, try applying it to areas where you’re blocked, stuck, overwhelmed, or burdened by the task. Think about a skill you’ve always wanted to learn but assumed was out of reach. Maybe you’ve never coded before, but you’re curious about writing a small script to automate a task. Maybe you’ve wanted to design a 3D-printed tool to solve a real-world problem but didn’t know where to start. AI can be a guide, an assistant, and sometimes even a collaborator in making these things possible.

For example, I once thought data science was beyond my skill set. For the longest time, I couldn’t even get Jupyter Notebooks to run! Even with expert help, I was clearly doing something silly and wrong, but it took a long time and finally LLM assistance to get step by step and deeper into sub-steps to figure out the step that was never in the documentation or instructions that I was missing – and I finally figured it out! From there, I learned enough to do a lot of the data science work on my own projects. You can see that represented in several recent projects. The same thing happened with iOS development, which I initially felt imposter syndrome about. And this year, after FOUR failed attempts (even 3 using LLMs), I finally got a working app for Android!

Each time, the challenge felt enormous. But by shifting from “I can’t” to “What if I could?” I found ways to break through. And each time AI became a more capable assistant, I revisited previous roadblocks and made even more progress, even when it was a project (like an Android version of PERT Pilot) I had previously failed at, and in that case, multiple times.

Revisiting Past Challenges

AI is evolving rapidly, and what wasn’t possible yesterday might be feasible today. Literally. (A great example is that I wrote a blog post about how medical literature seems like a game of telephone and was opining on AI-assisted tools to aid with tracking changes to the literature over time. The day I put that blog post in the queue, OpenAI announced their Deep Research tool, which I think can in part address some of the challenges I talked about currently being unsolved!)

One thing I have started to do that I recommend is keeping track of problems or projects that feel out of reach. Write them down. Revisit them every few months, and explore them with the latest LLM and AI tools. You might be surprised at how much has changed, and what is now possible.

Moving Forward with AI

You don’t even have to use AI for everything. (I don’t.) But if you’re not yet in the habit of using AI for certain types of tasks, I challenge you to find a way to use an LLM for *something* that you are working on.

A good place to insert this into your work/projects is to start noting when you find yourself saying or thinking “this is the way we/I do/did things”.

When you catch yourself thinking this, stop and ask:

  • Does it have to be done that way? Why do we think so?
  • What are we trying to achieve with this task/project?
  • Are there other ways we can achieve this?
  • If not, can we automate some or more steps of this process? Can some steps be eliminated?

You can ask yourself these questions, but you can also ask these questions to an LLM. And play around with what and how you ask (the prompt, or what you ask it, makes a difference).

One example for me has been working on a systematic review and meta analysis of a medical topic. I need to extract details about criteria I am analyzing across hundreds of papers. Oooph, big task, very slow. The LLM tools aren’t yet good about extracting non-obvious data from research papers, especially PDFs where the data I am interested may be tucked into tables, figure captions, or images themselves rather than explicitly stated as part of the results section. So for now, that still has to be manually done, but it’s on my list to revisit periodically with new LLMs.

However, I recognized that the way I was writing down (well, typing into a spreadsheet) the extracted data was burdensome and slow, and I wondered if I could make a quick simple HTML page to guide me through the extraction, with an output of the data in CSV that I could open in spreadsheet form when I’m ready to analyze. The goal is easier input of the data with the same output format (CSV for a spreadsheet). And so I used an LLM to help me quickly build that HTML page, set up a local server, and run it so I can use it for data extraction. This is one of those projects where I felt intimidated – I never quite understood spinning up servers and in fact didn’t quite understand fundamentally that for free I can “run” “a server” locally on my computer in order to do what I wanted to do. So in the process of working on a task I really understood (make an HTML page to capture data input), I was able to learn about spinning up and using local servers! Success, in terms of completing the task and learning something I can take forward into future projects.

Another smaller recent example is when I wanted to put together a simple case report for my doctor, summarizing symptoms etc, and then also adding in PDF pages of studies I was referencing so she had access to them. I knew from the past that I could copy and paste the thumbnails from Preview into the PDF, but it got challenging to be pasting 15+ pages in as thumbnails and they were inserting and breaking up previous sections, so the order of the pages was wrong and hard to fix. I decided to ask my LLM of choice if it was possible to automate compiling 4 PDF documents via a command line script, and it said yes. It told me what library to install (and I checked this is an existing tool and not a made up or malicious one first), and what command to run. I ran it, it appended the PDFs together into one file the way I wanted, and it didn’t require the tedious hand commands to copy and paste everything together and rearrange when the order was messed up.

The more I practice, the easier I find myself switching into the habit of saying “would it be possible to do X” or “Is there a way to do Y more simply/more efficiently/automate it?”. That then leads to portions which I can decide to implement, or not. But it feels a lot better to have those on hand, even if I choose not to take a project on, rather than to feel overwhelmed and out of control and uncertain about what AI can do (or not).

Facing uncertainty with AI and rethinking "What if you could?", a blog post by Dana M. Lewis on DIYPS.orgIf you can shift your mindset from fear and avoidance to curiosity and experimentation, you might discover new skills, solve problems you once thought were impossible, and open up entirely new opportunities.

So, the next time you think, “That’s too hard, I can’t do that,” stop and ask:

“What if I could?”

—

If you appreciated this post, you might like some of my other posts about AI if you haven’t read them.

How Medical Research Literature Evolves Over Time Like A Game of Telephone

by Dana Lewis

Have you ever searched for or through medical research on a specific topic, only to find different studies saying seemingly contradictory things? Or you find something that doesn’t seem to make sense?

You may experience this, whether you’re a doctor, a researcher, or a patient.

I have found it helpful to consider that medical literature is like a game of telephone, where a fact or statement is passed from one research paper to another, which means that sometimes it is slowly (or quickly!) changing along the way. Sometimes this means an error has been introduced, or replicated.

A Game of Telephone in Research Citations

Imagine a research study from 2016 that makes a statement based on the best available data at the time. Over the next few years, other papers cite that original study, repeating the statement. Some authors might slightly rephrase it, adding their own interpretations. By 2019, newer research has emerged that contradicts the original statement. Some researchers start citing this new, corrected information, while others continue citing the outdated statement because they either haven’t updated their knowledge or are relying on older sources, especially because they see other papers pointing to these older sources and find it easiest to point to them, too. It’s not necessarily made clear that this outdated statement is now known to be incorrect. Sometimes that becomes obvious in the literature and field of study, and sometimes it’s not made explicit that the prior statement is ‘incorrect’. (And if it is incorrect, it doesn’t become known as incorrect until later – at the time it’s made, it’s considered to be correct.) 

By 2022, both the correct and incorrect statements appear in the literature. Eventually, a majority of researchers transition to citing the updated, accurate information—but the outdated statement never fully disappears. A handful of papers continue to reference the original incorrect fact, whether due to oversight, habit (of using older sources and repeating citations for simple statements), or a reluctance to accept new findings.

The gif below illustrates this concept, showing how incorrect and correct statements coexist over time. It also highlights how researchers may rely on citations from previous papers without always checking whether the original information was correct in the first place.

Animated gif illustrating how citations branch off and even if new statements are introduced to the literature, the previous statement can continue to appear over time.

This is not necessarily a criticism of researchers/authors of research publications (of which I am one!), but an acknowledgement of the situation that results from these processes. Once you’ve written a paper and cited a basic fact (let’s imagine you wrote this paper in 2017 and cite the 2016 paper and fact), it’s easy to keep using this citation over time. Imagine it’s 2023 and you’re writing a paper on the same topic area, it’s very easy to drop the same citation from 2016  in for the same basic fact, and you may not think to consider updating the citation or check if the fact is still the fact.

Why This Matters

Over time, a once-accepted “fact” may be corrected or revised, but older statements can still linger in the literature, continuing to influence new research. Understanding how this process works can help you critically evaluate medical research and recognize when a widely accepted statement might actually be outdated—or even incorrect.

If you’re looking into a medical topic, it’s important to pay attention not just to what different studies say, but also when they were published and how their key claims have evolved over time. If you notice a shift in the literature—where newer papers cite a different fact than older ones—it may indicate that scientific understanding has changed.

One useful strategy is to notice how frequently a particular statement appears in the literature over time.

Whenever I have a new diagnosis or a new topic to research on one of my chronic diseases, I find myself doing this.

I go and read a lot of abstracts and research papers about the topic; I generally observe patterns in terms of key things that everyone says, which establishes what the generally understood “facts” are, and also notice what is missing. (Usually, the question I’m asking is not addressed in the literature! But that’s another topic…)

I pay attention to the dates, observing when something is said in papers in the 1990s and whether it’s still being repeated in the 2020s era papers, or if/how it’s changed. In my head, I’m updating “this is what is generally known” and “this doesn’t seem to be answered in the literature (yet)” and “this is something that has changed over time” lists.

Re-Evaluating the Original ‘Fact’

In some cases, it turns out the original statement was never correct to begin with. This can happen when early research is based on small sample sizes, incomplete data, or incorrect assumptions. Sometimes that statement was correct, in context, but taken out of context immediately and this out of context use was never corrected. 

For example, a widely cited statement in medical literature once claimed that chronic pancreatitis is the most common cause of exocrine pancreatic insufficiency (EPI). This claim was repeated across numerous papers, reinforcing it as accepted knowledge. However, a closer examination of population data shows that while chronic pancreatitis is a known co-condition of EPI, it is far less common than diabetes—a condition that affects a much larger population and is also strongly associated with EPI. Despite this, many papers still repeat the outdated claim without checking the original data behind it.

(For a deeper dive into this example, you can read my previous post here. But TL;DR: even 80% of .03% is a smaller number than 10% of 10% of the overall population…so it is not plausible that CP is the biggest cause of EPI/PEI.)

Stay Curious

This realization can be really frustrating, because if you’re trying to do primary research to help you understand a topic or question, how do you know what the truth is? This is peer-reviewed research, but what this shows us is that the process of peer-review and publishing in a journal is not infallible. There can be errors. The process for updating errors can be messy, and it can be hard to clean up the literature over time. This makes it hard for us humans – whether in the role of patient or researcher or clinician – to sort things out.

But beyond a ‘woe is me, this is hard’ moment of frustration, I do find that this perspective of literature as a process of telephone makes me a better reader of the literature and forces me to think more critically about what I’m reading, and take papers in context of the broader landscape of literature and evolving knowledge base. It helps remove the strength I would otherwise be prone to assigning any one paper (and any one ‘fact’ or finding from a single paper), and encourages me to calibrate this against the broader knowledge base and the timeline of this knowledge base.

That can also be hard to deal with personally as a researcher/author, especially someone who tends to work in the gaps, establishing new findings and facts and introducing them to the literature. Some of my work also involves correcting errors in the literature, which I find from my outsider/patient perspective to be obvious because I’ve been able to use fresh eyes and evaluate at a systematic review level/high level view, without being as much in the weeds. That means my work, to disseminate new or corrected knowledge, is even more challenging. It’s also challenging personally as a patient, when I “just” want answers and for everything to already be studied, vetted, published, and widely known by everyone (including me and my clinician team).

But it’s usually not, and that’s just something I – and we – have to deal with. I’m curious as to whether we will eventually develop tools with AI to address this. Perhaps a mini systematic review tool that scrapes the literature and includes an analysis of how things have changed over time. This is done in systematic review or narrative reviews of the literature, when you read those types of papers, but those papers are based on researcher interests (and time and funding), and I often have so many questions that don’t have systematic reviews/narrative reviews covering them. Some I turn into papers myself (such as my paper on systematically reviewing the dosing guidelines and research on pancreatic enzyme replacement therapy for people with exocrine pancreatic insufficiency, known as EPI or PEI, or a systematic review on the prevalence of EPI in the general population or a systematic review on the prevalence of EPI in people with diabetes (Type 1 and Type 2)), but sometimes it’s just a personal question and it would be great to have a tool to help facilitate the process of seeing how information has changed over time. Maybe someone will eventually build that tool, or it’ll go on my list of things I might want to build, and I’ll build it myself like I have done with other types of research tools in the past, both without and with AI assistance. We’ll see!

—

TL;DR: be cognizant of the fact that medical literature changes over time, and keep this in mind when reading a single paper. Sometimes there are competing “facts” or beliefs or statements in the literature, and sometimes you can identify how it evolves over time, so that you can better assess the accuracy of research findings and avoid relying on outdated or incorrect information.

Whether you’re a researcher, a clinician, or a patient doing research for yourself, this awareness can help you better navigate the scientific literature.

A screenshot from the animated gif showing how citation strings happen in the literature, branching off over time but often still resulting in a repetition of a fact that is later considered to be incorrect, thus both the correct and incorrect fact occur in the literature at the same time.

The prompt matters when using Large Language Models (LLMs) and AI in healthcare

by Dana Lewis

I see more and more research papers coming out these days about different uses of large language models (LLMs, a type of AI) in healthcare. There are papers evaluating it for supporting clinicians in decision-making, aiding in note-taking and improving clinical documentation, and enhancing patient education. But I see a wide-sweeping trend in the titles and conclusions of these papers, exacerbated by media headlines, making sweeping claims about the performance of one model versus another. I challenge everyone to pause and consider a critical fact that is less obvious: the prompt matters just as much as the model.

As an example of this, I will link to a recent pre-print of a research article I worked on with Liz Salmi (published article here pre-print here).

Liz nerd-sniped me about an idea of a study to have a patient and a neuro-oncologist evaluate LLM responses related to patient-generated queries about a chart note (or visit note or open note or clinical note, however you want to call it). I say nerd-sniped because I got very interested in designing the methods of the study, including making sure we used the APIs to model these ‘chat’ sessions so that the prompts were not influenced by custom instructions, ‘memory’ features within the account or chat sessions, etc. I also wanted to test something I’ve observed anecdotally from personal LLM use across other topics, which is that with 2024-era models the prompt matters a lot for what type of output you get. So that’s the study we designed, and wrote with Jennifer Clarke, Zhiyong Dong, Rudy Fischmann, Emily McIntosh, Chethan Sarabu, and Catherine (Cait) DesRoches, and I encourage you to check out the article here pre-print and enjoy the methods section, which is critical for understanding the point I’m trying to make here. 

In this study, the data showed that when LLM outputs were evaluated for a healthcare task, the results varied significantly depending not just on the model but also on how the task was presented (the prompt). Specifically, persona-based prompts—designed to reflect the perspectives of different end users like clinicians and patients—yielded better results, as independently graded by both an oncologist and a patient.

The Myth of the “Best Model for the Job”

Many research papers conclude with simplified takeaways: Model A is better than Model B for healthcare tasks. While performance benchmarking is important, this approach often oversimplifies reality. Healthcare tasks are rarely monolithic. There’s a difference between summarizing patient education materials, drafting clinical notes, or assisting with complex differential diagnosis tasks.

But even within a single task, the way you frame the prompt makes a profound difference.

Consider these three prompts for the same task:

  • “Explain the treatment options for early-stage breast cancer.”
  • “You’re an oncologist. Explain the treatment options for early-stage breast cancer.”
  • “You’re an oncologist. Explain the treatment options for early-stage breast cancer as you would to a newly diagnosed patient with no medical background.”

The second and third prompt likely result in a more accessible and tailored response. If a study only tests general prompts (e.g. prompt one), it may fail to capture how much more effective an LLM can be with task-specific guidance.

Why Prompting Matters in Healthcare Tasks

Prompting shapes how the model interprets the task and generates its output. Here’s why it matters:

  • Precision and Clarity: A vague prompt may yield vague results. A precise prompt clarifies the goal and the speaker (e.g. in prompt 2), and also often the audience (e.g. in prompt 3).
  • Task Alignment: Complex medical topics often require different approaches depending on the user—whether it’s a clinician, a patient, or a researcher.
  • Bias and Quality Control: Poorly constructed prompts can inadvertently introduce biases

Selecting a Model for a Task? Test Multiple Prompts

When evaluating LLMs for healthcare tasks—or applying insights from a research paper—consider these principles:

  1. Prompt Variation Matters: If an LLM fails on a task, it may not be the model’s fault. Try adjusting your prompts before concluding the model is ineffective, and avoid broad sweeping claims about a field or topic that aren’t supported by the test you are running.
  2. Multiple Dimensions of Performance: Look beyond binary “good” vs. “bad” evaluations. Consider dimensions like readability, clinical accuracy, and alignment with user needs, as an example when thinking about performance in healthcare. In our paper, we saw some cases where a patient and provider overlapped in ratings, and other places where the ratings were different.
  3. Reproducibility and Transparency: If a study doesn’t disclose how prompts were designed or varied, its conclusions may lack context. Reproducibility in AI studies depends not just on the model, but on the interaction between the task, model, and prompt design. You should be looking for these kinds of details when reading or peer reviewing papers. Take results and conclusions with a grain of salt if these methods are not detailed in the paper.
  4. Involve Stakeholders in Evaluation: As shown in the preprint mentioned earlier, involving both clinical experts and patients in evaluating LLM outputs adds critical perspectives often missing in standard evaluations, especially as we evolve to focus research on supporting patient needs and not simply focusing on clinician and healthcare system usage of AI.

What This Means for Healthcare Providers, Researchers, and Patients

  • For healthcare providers, understand that the way you frame a question can improve the usefulness of AI tools in practice. A carefully constructed prompt, adding a persona or requesting information for a specific audience, can change the output.
  • For researchers, especially those developing or evaluating AI models, it’s essential to test prompts across different task types and end-user needs. Transparent reporting on prompt strategies strengthens the reliability of your findings.
  • For patients, recognizing that AI-generated health information is shaped by both the model and the prompt. This can support critical thinking when interpreting AI-driven health advice. Remember that LLMs can be biased, but so too can be humans in healthcare. The same approach for assessing bias and evaluating experiences in healthcare should be used for LLM output as well as human output. Everyone (humans) and everything (LLMs) are capable of bias or errors in healthcare.

Prompts matter, so consider model type as well as the prompt as a factor in assessing LLMs in healthcare. Blog by Dana M. LewisTLDR: Instead of asking “Which model is best?”, a better question might be:

“How do we design and evaluate prompts that lead to the most reliable, useful results for this specific task and audience?”

I’ve observed, and this study adds evidence, that prompt interaction with the model matters.

Best practices in communication related to writing a journal article and sharing it with co-authors

by Dana Lewis

I’ve been a single author, a lead author, a co-author, a corresponding author, AND a last author. Basically, I have written a lot of journal articles myself, solo / single, and with other people. One area in this process that I observe frequently gets overlooked is what happens during and after the submission process, as it relates to communicating about the article itself.

I’m not talking about disseminating the article to your target audience or the public, either (although that is important as well). I’m talking about making sure all authors know the article has been accepted; when it is live; have access to a copy of the article (!); etc.

Most people don’t know that by default, not all journals give all authors access to their own articles for free.

Here are some tips about the process of submitting and saving published articles that will help all authors – even solo authors – in the future.

Basically, help you help your future self! (As well as help your co-authors).

Journals typically only notify the lead/corresponding/submitting author about where the manuscript is in terms of revision, acceptance, and publication. That puts the responsibility on the lead/corresponding/submitting author to notify the full team of authors of where the article is in the process. Similarly, some journals will send a PDF/final copy of the proofed, final, version of record article to the lead author (not always, but usually), but that often does not go out to the full author team by default.

This means that it is the lead author’s responsibility to forward the copy of the final, PDF, proofed article to the entire authorship team so everyone has a copy.

(No, most of the time authors do not have free access to the journal they are submitting to. No, most authors do not have budget to make articles open access and free to all, which means unless they manage to snag and save this PDF article when it is sent to them at the time of publication, in the future, they may not have access to their very own article! Just because you, as the lead/corresponding author do have access, this does not mean everyone on your article team will.

I’m a good example of someone who authors frequently but is not at an institution and has zero access to any paywalled journals. If I’m not given a copy of my articles at the time of publication, I have to phone-a-friend (thanks, Liz Salmi, for being the go-to for me here) to help pull articles. There are things like S c i H u b, but they more often than not do not have super recent, fresh off the press articles. So yes, people like me exist on your authorship teams.)

Best practices for authors include:

  • Once you submit a manuscript, mark your file name (somehow) with “Submitted”. This way you know this is the version that was submitted. This is a useful step related to the below, we’ll come back to why we may want to use only the ‘submitted’ version.

    Example: “JournalAcronym-Article-Blah-Blah-SUBMITTED.docx”.

    Even as the non-lead author, when co-writing articles, as any type of author I prefer to have access to this submitted version. This way, I can see all incorporated edits and the ‘final’ version we submitted. There’s also cases where, see below, I need this for sharing it with other people.

  • Usually, the article goes through peer review and you get comments, so you make revisions and re-submit your article. Again, once submitted, make sure you’ve marked this as ‘revision’ somehow (usually people do) and that is was submitted.

    Example: “JournalAcronym-Article-Blah-Blah-SUBMITTED-R1.docx”.

    Again, best practice would be to send out this re-submitted revision version to all authors so everyone has it.

  • You may end up with multiple rounds of revisions and peer review (moving to R2, etc), or you may get an acceptance notice. Your article will then move to copyediting stage and you get proofs. It’s useful to save these for your own purpose, such as making sure that the edits you make are actually executed in the final article. This is less important for dissemination, though, although I do recommend giving all co-authors the ability to edit/review/proof and request changes.
  • Accepted, proofed, published! THIS is the step that I see most people miss, so pay attention.If you are the lead or solo author, you will probably get an email saying your article is now online, either online first or published. You may get an attachment PDF of your article. If not, you should be able to click on your access link and go to access the article online.

    IMPORTANT STEP HERE: go ahead and download the PDF of the article then. Right then, go ahead and save it.

    Example: “JournalAcronym-Article-Blah-Blah-Year.PDF”.

    (Why do you care about this if you are a solo author? Because the link may expire and you may lose access to this article. More on sharing your article below.)

  • Email your entire author team (if you’re not a solo author). Tell them the article was published; provide a link and/or the DOI link; and attach the PDF to the email so everyone on the team has a copy of the final article. Not all of your co-authors will work at an institution that has unlimited library access; if they do, that might change in the future. Give everyone a copy of the article to save for themselves.You can also remind everyone what the sharing permissions (or limitations) are for the article.

    For example, some articles are paywalled but authors have permission to store the final copy (PDF of the final version) on their own repository or not-for-profit website. For an example, see my research page of DIYPS.org/research – you’ll notice how sometimes I link to an “author copy” PDF, which is what this is – the final article PDF like you would get by accessing the paywalled journal.

    Other times, though, you are specifically not permitted to share the final/proofed/formatted copy. Instead, you’ll be allowed to share the “submitted” manuscript (usually prior to the revision stage). Remember how step 1 that I told you was to save a SUBMITTED copy? This is why! You can PDF this up; add a note to the top that references the final version of record (usually, journals give you recommended language for this) and a link/DOI link to it, and share away on your own site. Again, look at DIYPS.org/research and you’ll notice some of my “author copy” versions are these submitted versions rather than the final versions.

    You’ll also notice that sometimes I link to articles that are open access and then also have a link to a PDF author copy. This is in case something changes in the future with open access links breaking, the journal changing, etc. I have actually had free non-paywalled articles get turned into paywalled journal articles years later, which is why I do point to both places (the open access version and a back up author copy).

    Regardless of what the permissions are for sharing on your own website/repository/institutional repository: you as the author always have permission to give this PDF out when you are asked directly. For example, someone emails you and asks for a copy: you can email back and attach the PDF! This is true even if the permissioning for your own website is the submitted version (not the final version), you can still hand out the final, formatted, pretty PDF version when asked directly.

    As a related tip, this is a great way to disseminate your research and build relationships, so if someone does email you and ask for an author copy…please reply and send them a copy. (Saying this as someone without access to articles who sends requests to many authors to get access to their research, and I only get responses from 50% of authors. Sad panda.) Again, this is why it is helpful to get in the habit of saving your articles as you submit and have them published; it makes it easy to jump into the “Published copy” folder (or however you name it) and attach the PDF to the email and send it.

To recap, as a best practice, you should disseminate various versions of articles to your entire co-author team at the following points in time:

  • Original submission.

    Suggestion: Write an email, say you’ve successfully submitted, remind everyone which journal this was submitted to, and attached a copy of your “JournalAcronym-Article-Blah-Blah-SUBMITTED.docx”(If you end up getting a desk rejection, and you are re-submitting elsewhere, it is also nice to email co-authors and tell them so. You don’t necessarily need to send out a newly retitled version, unless there’s new changes to the submission, such as if you did go through a partial round of peer review before getting rejected and you are submitting the revised version to the new target journal.)

  • Revision submission.

    Suggestion: Write an email, say you’ve successfully submitted the revisions, remind everyone which journal this was submitted to, and attached a copy of your “JournalAcronym-Article-Blah-Blah-SUBMITTED-R1.docx” and the reviewer response document so everyone can see how edits/feedback were incorporated (or not).

  • Acceptance.

    Suggestion:

    A) Forward the email if it has the PDF attached to your full author team. Say congratulations; the article was accepted; and point out the article is attached as PDF.

    B) If you don’t have a PDF attachment in your email already, go to the online access link the journal gave you and save a copy of the PDF. Then, email the author team with the FYI that the article is live; provide the link to the online version; and attach the PDF directly to that email so everyone has a final version.

    Regardless of A or B, remind everyone what the permissions are for sharing to their own/institution repository (eg final PDF or use the submitted version, which you previously shared or could also re-share here).

Bonus tip:

Depending on the content of your article, you may also want to think about sending copies of the final PDF article to certain people who are not co-authors with you.

For example, if you are heavily citing someone’s work or talking about their work in a constructive way – you could email them and give them a heads up and provide a copy of the article. It’s a great way to contribute to your relationship (if you have an existing relationship) and/or foster a relationship. Remember that many people will have Google Scholar Alerts or similar with their name and/or citation alerts from various services, so people are likely to see when you talk about them or their work or are heavily citing their work. Again, some of those people may not have access to your article and may reach out to ask for an article; you can (and should) send them a copy! (And again, consider thinking about it as a relationship building opportunity rather than a transactional thing related to this single article.)

I would particularly flag this as something to pay attention to and do if you are someone working in the space of patient engagement in healthcare. For example, if you write an article and mention them or their body of work by name, it would be courteous to email them, let them know about the article, and send them a PDF.

Otherwise, I can speak from the experience of being talked about as a patient like I’m an ant under the microscope where someone cites an article where my work is mentioned; talks about me by name and references my perspective; and I get a notification about this article….but I can’t access it because it’s in a paywalled journal. Awkward, and a little weird in some cases when the very subject of the article(s) are about patient engagement and involving patients in research. Remember, research involvement should include all stages from design, planning, doing the research, and then disseminating the research. So this meta point is that if there is scholarly literature of any kind (whether original research articles or reviews, commentaries, letters in response to other articles, etc) talking about specific patients and their bodies of work – best practice should be to email them and send a copy of the article. Again, think less transactional and more about relationships – it will likely give you benefit in the long run! Plus, less awkward, a short-term benefit.

—-

best practices for communicating with co-authors about published articles, by Dana M. Lewis from DIYPS.orgAs an example for how I like to disseminate my articles personally, every time a journal article is published and I have access to it, I updated DIYPS.org/research with the title, journal, a DOI link (to help people find it online and/or cite it), and a link to the open access version if available and if not, an author copy PDF of the final or submitted version. So, if you’re ever looking for any of my articles, you can head there (DIYPS.org/research) first and grab copies any time!

If you are looking for a particular article and can’t find it or it’s not listed there yet (e.g. likely because it just came out and I haven’t been sent my own copy by my co-authors yet…), you can always email me directly (Dana@OpenAPS.org) and I’m more than happy to send you a copy of whatever version I have available and/or the final PDF once I have access to it.