6. Toby Wise: Risk perception about COVID-19, natural experiments, and open science

Toby Wise is a postdoc at UCL and Caltech. He uses computational modelling and neuroimaging to study the mechanisms underlying anxiety and depression. I first encountered Toby when he and I published separate preprints on PsyArXiv on the same topic (risk perception for COVID-19) within a few hours of each other.

In this conversation, we talk about doing research about COVID-19: why we decided to do it, practical considerations, and differences and similarities between our studies. We also talk about open science practices.

BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith. New conversations every other Friday. You can find the podcast on all podcasting platforms (Apple/Google Podcasts, Spotify, etc.).

Timestamps
0:00:11: The origin of Toby's research project on risk perception about COVID-19
0:13:18: What Toby would do differently if he could go back in time
0:20:45: Criticism of COVID-19 research
0:29:17: How to do good science during natural experiments
0:44:09: Open Code, (Jupyter/RMarkdown) Notebooks, and Python
1:07:43: Comparing COVID responses across and within countries
1:27:36: Practicalities of doing research on COVID-19
1:34:19: External validity of psychological research
1:48:30: Toby's acute awareness of how unimportant his research is
2:06:32: Simulations to ensure your study actually does what you want it to do
2:14:34: Comparing Toby and Ben's COVID studies

Toby's links

Podcast links

Ben's links


References/papers mentioned
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T. H., Huber, J., Johannesson, M., ... & Altmejd, A. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour.
Levitt, S. D., & List, J. A. (2007). What do laboratory experiments measuring social preferences reveal about the real world?. Journal of Economic perspectives.
Korn, C. W., Sharot, T., Walter, H., Heekeren, H. R., & Dolan, R. J. (2014). Depression is related to an absence of optimistically biased belief updating about future life events. Psychological medicine.
Kunz, L., Schröder, T. N., Lee, H., Montag, C., Lachmann, B., Sariyska, R., ... & Fell, J. (2015). Reduced grid-cell–like representations in adults at genetic risk for Alzheimer’s disease. Science.
Kuper-Smith, B. J., Doppelhofer, L. M., Oganian, Y., Rosenblau, G., & Korn, C. (2020). Optimistic beliefs about the personal impact of COVID-19. PsyArXiv.
Shah, A. K., Mullainathan, S., & Shafir, E. (2012). Some consequences of having too little. Science.
Shah, A. K., Mullainathan, S., & Shafir, E. (2019). An exercise in self-replication: Replicating Shah, Mullainathan, and Shafir (2012). Journal of Economic Psychology.
Wise, T., Zbozinek, T. D., Michelini, G., Hagan, C. C., & Mobbs, D. (2020). Changes in risk perception and self-reported protective behaviour during the first week of the COVID-19 pandemic in the United States. Royal Society Open Science.

  • [This is an automated transcript that contains many errors]

    Toby Wise: [00:00:00] So, yeah, I'm ready. 

    Benjamin James Kuper-Smith: You're ready? Okay. Um, actually, maybe, maybe this will be the start thing. We'll see. Um, but I was curious, how did you, I mean, how did you decide to do a research project on COVID Ovid 19? I mean, in some sense it seems like this is very obvious reason for how you got exposed to the topic, uh, but you didn't, you know, you're not an epidemiologist or someone who studies anything related to this.

    So what was your, uh, like you and in the lab, how did that whole thing start? 

    Toby Wise: Yeah. Um, so I guess the main point of interest for me is, you know, because I do research on anxiety and fear, um, and essentially the entire population has been exposed to what we consider quite an unpleasant experience, which is likely to lead to anxiety and fear.

    Um, so that was, that was why I was interested in doing it. Um, uh, because a, it might tell us some things about why we become [00:01:00] anxious, you know, um, and, and b there may become mental health effects. It's not the primary thing we are looking at, but we may be able to get some insight into how, um, the pandemic has affected people's mental health.

    Um. So, uh, so the story of like, how we did this, um, basically I'm working with Dean Mobs at Caltech on this, and, um, it was in, was it March? Um, like the first, second week of March when th we kind of realized that things were getting pretty serious. Um, and, uh, I kind of was chatting to my partner who's also, uh, an academic and, um, we kind of got talking about this and thought, you know, actually maybe this is something we should be, we should be studying.

    Um, you know, it's, it's a, a, a rare thing that occurs to humanity, a pandemic on the scale. Um, and we thought it might be important to study it. So I, uh, sent Dean an email quickly. He was like, yeah, sure, let's chat tomorrow. Went into the office on Monday. [00:02:00] We'd decided, yeah, okay, let's go for it. We managed to get ethical approval on the Tuesday, um, thanks to Caltex, very, uh, efficient, uh, IRB.

    Um, and on Wednesday we were able to collect our first lot of data. Um, so yeah, it was a very rapid process going from this initial idea of wanting to look at anxiety in response to the pandemic and actually doing the thing. 

    Benjamin James Kuper-Smith: Was the first thing you recorded was, uh, the data, the first data you collected, was that what's in your Royal Society Open science paper or, um, what is, what did you record there?

    Toby Wise: Yeah, so that's the, that is the data in there. So we did, um, so on that week, I think it was, I can't remember what the first date was. I think the first day may have been the 11th of March. Um, I've 

    Benjamin James Kuper-Smith: got the paper right here. 

    Toby Wise: Oh, 

    Benjamin James Kuper-Smith: you gimme a 

    Toby Wise: second. I I should really have it up as well. Um, uh, but what was it, 11th or the 30th?

    Benjamin James Kuper-Smith: 11th, yes. 

    Toby Wise: 11th. Yeah. Okay. So that was, that was the Wednesday, I think. Um, so yeah, we got that first lot of [00:03:00] data in, I think 500 subjects, um, thereabouts. Um, and then we thought, you know, we may as well kinda keep collecting data for the next week or so, which we did. So we did, I think, uh, we did Wednesday, Thursday, Friday, Saturday.

    Different lots of people. Then we went back and, and followed the first 500 we did on the Wednesday on, I think the following Monday or Tuesday. Um, but that whole week of data basically is in that paper. So the idea is we, you know, we wanted to kind of show how people's kind of perception of risk and things like that changed over that, that very first week, because that was where it was on, on that Wednesday.

    That, that the who. They declared it a pandemic, um, a global pandemic, 

    Benjamin James Kuper-Smith: the, the 11th, or which 

    Toby Wise: I think it was on the 11th, that they, that the World Health Organization actually said, you know, this is a global pandemic. Um, and during that week, you know, politicians all around the world, including in the United States, started to make some kind of movement towards, uh, wanting to control it a bit more.

    So that was, [00:04:00] uh, it was basically the first kind of week where people thought, okay, this is, this is serious, at least in the United States. 

    Benjamin James Kuper-Smith: Was the motivation then Im from the start to, I mean, was it primarily to study, like, was was the prime thing to study the pandemic or was it more to use the pandemic as a, I mean, the phrase I think is natural experiment or what people might call it where you have these changes in people's, uh, what they're exposed to, their behavior, whatever.

    Um, like did you, from which direction did you come or was it just a, a blend of everything? 

    Toby Wise: Yeah, it was, it was definitely the, the latter. So it was more using it as a natural experiment. Experiment. I mean, obviously, you know, we don't want to kind of, uh, just kind of make it sound like we are, you know, taking advantage of people's suffering or anything like that.

    It's obviously a, an awful thing that's happened, but you know, you may as well make the worst of, make the best of a, of a, of a bad situation. Um, it's this kind of a stressful life event on this kind of [00:05:00] scale. Which is experienced by this many people is not the sort of thing you can do in the lab ever.

    Like, I mean, you would not, you would want to be the IOB would probably never approve it. Um, so if we want to understand how people react to these kind of, they like really, really stressful serious life events. The only way to do it's naturalistically. So you have to kind of, you know, do what you can when they happen.

    Um, so, you know, uh, one other example of this is, uh, where people have looked at, for example, kind of PTSD and things in response to terrorist attacks. So nine 11 for example. Um, that's the sort of thing you, if you want to understand how those kind of responses to traumatic events develop, you can't do that in a lab.

    So you just have to kind of latch onto something bad that that happens to, to occur in real life. Um, so yeah, it's very much, this is a terrible thing that's happening. What can we try and learn from it essentially? 

    Benjamin James Kuper-Smith: Yeah. So what, what can we learn from it? 

    Toby Wise: That's a good question. [00:06:00] Um, 

    Benjamin James Kuper-Smith: in your particular case, maybe, 

    Toby Wise: uh, I mean, to be honest, we went into it, um, given, given how rushed it was, we went into it with quite vague kind of expectations and ideas.

    Um, so in, in, in the paper that we, we published on it, to Honest, that was not. The, the data we're describing there, that wasn't part of a primary hypothesis or anything. It's more just like, you know, we have this data on how risk perception is changing early on. We may as well put it out there so people can see it and use it.

    What we're more interested in is how specifically, um, people's risk perception and that anxiety develops as the pandemic, uh, goes. It goes, um, you, it goes on for, for months basically. Um, so we've been following people up, um, since March to look at that. And how, because how, at 

    Benjamin James Kuper-Smith: what intervals? 

    Toby Wise: So we've been doing it.

    Um, we started off quite, you know, densely sampling. We're doing it I think every two weeks. Uh, then we kind of dropped out to sort of a month now we're doing every couple of months, and I think we, we are just [00:07:00] doing our final one this week. Um, but the, I the, the reason behind that is basically a lot of the work that I do, in particular in the lab, looks at how people learn about threats and, you know, how they feel subjectively, subjectively in response to that threat that they're learning about.

    So I kind of wanted then essentially translate to that, that to the real world and say, you know, we have this external stressor that people have been experienced too. How do people learn about the, the risks that they face from that stressor and, and how do they respond to it subjectively as time goes on?

    Um, so that was, that was kinda the main thing that we, that we wanted to, to test. 

    Benjamin James Kuper-Smith: So one thing that we found also interesting to, I mean in our study too, that we found difficult to think about is that we, you often just don't know what the risk, what the actual risk is, right. As a scientist even. Yeah. It's not that you have some tightly controlled experiment and you can say the risk is 80%.

    Um, so how does, how do you think that affects people, not just this? Yeah. Complete [00:08:00] uncertainty in terms of what the actual risk is, because it seems to me sometimes it felt like the risk was, especially in the beginning, was quite high, but I feel like now it's, 

    Toby Wise: yeah, 

    Benjamin James Kuper-Smith: it's become clearer that it's much more manageable.

    Um, but from, if you're interested in this like fear and risk perception, how does, how does it change when you have very little idea what the actual risk is? 

    Toby Wise: Yeah, yeah. I mean, that is obviously a problem with any kind of naturalistic experiment. There is no objective kind of risk. Um, I guess what we do have is, uh, we can sort of compare to what is being floated around in the media.

    Um, you know, I think when we started this, there were reports in the media saying that, you know, something like 80% of people are gonna get infected with this. Things like that. Um, so you have that kind of, it may not be objective risk, but it's what people are, are learning what they're seeing in the news.

    Mm-hmm. Um, but the other fact is that it doesn't necessarily matter so much perhaps as the subjective risks. So what we'll actually [00:09:00] perceive, if we just take it that, you know, COVID presents. A, a risk of some of some extent. Um, then the interesting thing to me is more, you know, how how do people perceive it to be, uh, a, a risk for them and a risk for other people in the country?

    Um, so yeah, even if we don't know the objective risk, um, we can't, we can't, well, I know some people may deny it's a risk at all, but for the vast majority of us, we accept that it is some level of risk, and then it's kinda up to us to, uh, to decide, uh, how bad it's for each of us individually. And that's quite interesting.

    Yes. But I, yeah, it would be nice to have a more concrete idea of the risk. Um, and I think you're right that, uh, it was definitely, I, the thing is, I don't know whether it actually was that, um, it, it was kind of, it seemed more to be more of a risk at the start, or whether it's just that we are more uncertain about it at the start.

    Um, it may be that, you know, if, if you think about like the confidence intervals on our [00:10:00] estimate of risk, perhaps the upper bound on that was just far higher maybe, or something like that. Um. So what's quite interesting is we, we have tried to ask people this, um, I don't have, I've not read, looked to the data for this, but we did ask people, so how, how likely do you think you are to get infected and how confident are you about that?

    Um, I, again, haven't looked at the data, so I can't reveal any exciting results. See you there. But, um, uh, so we might, we might be able to get some kind of index of whether it is related more, you know, whether that anxiety is related more to their uncertainty. Um, uh, or, or the actual kind of risk, perceived risk of getting it.

    Benjamin James Kuper-Smith: Yeah. I mean, on a, from a like personal perspective, in the beginning it seemed to me that there was much more talk about getting infections from, uh, touching things. So like from door handles or whatever, something like that. Whereas now it seems to me that a lot more has switched to this, uh, what's the word, aerosol based transmission, um, through the air.

    Um. And to me some of that has made it like [00:11:00] just on a per like, personally, just so much easier to manage because you're like, okay, wear a mask. Yeah. Everyone else wears a mask, then it's pretty much not a much of a problem. Whereas in the beginning it seemed like anything you touch might carry the virus Yeah.

    And affect you. 

    Toby Wise: Yeah. And I mean, like that, that sort of kind of perception is presumably adaptive, right? You like it's best in that situation where you don't know where the infection is gonna come from to just be kind of scared of everything. Um, and you're right. As, as things have progressed, as, you know, the, the science on that side of things has, has, has become more advanced.

    We know that we don't necessarily should be kind of worrying about literally every, everything we touch. Um, but maybe we do need to be more concerned about aerosols and things. Um, so yeah, I guess, I guess like the, the exact nature of people's, the risks that people are facing every day has changed quite substantially.

    Um, both in that, I dunno, maybe it's either kind of greatened or lessened, but also that people have become more certain about where that risk comes from. Um, it's, it is interesting. I've, I, that's not [00:12:00] something we've, we've asked people about and perhaps we should have done because. It. Uh, yeah, I, I, we just didn't really think about that at the start.

    Yeah. Yeah. 

    Benjamin James Kuper-Smith: I mean, I mean, we also didn't ask, um, so we, the collected data three times in mid-March, beginning of April, mid-May. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: And we didn't ask about masks until May. 

    Toby Wise: Yeah. We 

    Benjamin James Kuper-Smith: didn't either. 'cause we just didn't know it was like gonna, like, it doesn't, it doesn't seem to us like a big factor at the time.

    Toby Wise: Yeah, not at all. It's, it's really strange now, like looking at the questions we're using then, and I mean, we've had kinda had to stick with them because we are doing this longitudinally and we can't just change the questions, but they, a lot of them don't really seem that sensible anymore. Like Yeah, we didn't ask about masks at all.

    Like that's a, an obvious thing to us now. Uh, but at, you know, back in March, no one was even considering that here. Um, it also would've been, I I, I know I've not really looked into it in too much detail, but I know, you know, there have been studies done in this, like in, in Asia for example, countries where [00:13:00] mask wearing is fairly a fairly normal phenomenon.

    So I dunno, you know, what, what those results have have shown, um, perhaps there, you know, there was, you know, people were kind of taking all these preventative measures anyway, whereas we had, we showed that, you know, they've kind of grown as a pandemic has, has gone on. I don't know. 

    Benjamin James Kuper-Smith: Yeah. So what, what, what question do you wish you'd have asked, uh, right from the start?

    Like, if you could have, 

    Toby Wise: oh, 

    Benjamin James Kuper-Smith: if you could return the time and insert some new questions. 

    Toby Wise: Oh, that, that's a very good 

    Benjamin James Kuper-Smith: question. Other than wearing masks, 

    Toby Wise: um, 

    Benjamin James Kuper-Smith: I mean like one thing that I just thought about just now was we were talking about it is also the, the duration of the interaction. Yeah. So that's something we didn't, I mean we, we, so we asked different social contexts and that to some extent, uh, you know, you're gonna spend more time with a friend or with your family than with someone you see in a supermarket, right?

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Like a random stranger you walk past. Um, so it's kind of implicit in there, but 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Uh, we didn't, yeah, 

    Toby Wise: yeah, [00:14:00] yeah. I think we, we are similar things where it's Yeah. Kind of implicit, but we didn't tackle that directly. I think it, yeah, would definitely be nicer to sort of asked about all these different sources of risk and how people perceive them changing as time goes on, or these, you know, things that we just didn't think about before.

    Um, also I think just people's level of knowledge about, um, kind of pandemics and, you know, respiratory viruses in general, um, which we just kind of, I know, didn't really think to us partly 'cause, you know, we ourselves didn't know much about it. Um, I think all of us were kind of a bit ignorant, um, at the start.

    Um, I think it would've been nice as well to get a better idea of, um. How people perceive the, the, the course of the, the pandemic to, to run. Um, so we did start asking questions about that later on. But, you know, I think at the start, so you mean like 

    Benjamin James Kuper-Smith: how long it's gonna take or, 

    Toby Wise: yeah. And how much then and Yes, and, and, and [00:15:00] how, kinda, how bad it will get, I guess.

    Um, because I think I, I, from what I saw, you know, in the, in the early stages, um, at, at least in, in America and I think Europe as well, um, people were like, you know, it, this is gonna be a couple of months. It might be a bit bad, but it's not gonna be too awful. And then, you know, what happened happened. Um, so it would've been quite interesting, I think, to see how people's expectations change.

    Um, so we, we have started asking people, uh, in later stages, like how long they think it's gonna go on for. Um, but we only started asking them, uh, that like, I think maybe like July or something. So, but it would've been very interesting to see I, 'cause I imagine, you know, in March people would've gone from thinking like, this will be over in a month gradually to, this will be over in a year or more, something.

    Um, 

    Benjamin James Kuper-Smith: yeah. Now it seems to be, I mean from what I've heard is like there will be. [00:16:00] Once we get a vaccine, whenever that's gonna be the case. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: It's gonna still take ages to produce it and then distribute it to first the people who actually need it on a daily basis. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Until the general population gets it, it might be quite a while.

    Right? 

    Toby Wise: Absolutely. And I mean there's still a lot of uncertainty about that because we don't yet have an approved vaccine and like the FDA, you know, threshold for effectiveness I think is, you know, 50%. So it has to for 50% of infections. So you could still have a vaccine out there that isn't massively effective.

    So 

    Benjamin James Kuper-Smith: you have these calculations, even if it's not mandatory Right. Then you get 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Like a lot of people can't be bothered or don't want to, then you have like 70 to 50% actually getting, getting it and then only 50% of that time it works. Yeah, 

    Toby Wise: yeah. So I mean there, you know, there are certainly potential situations where even having a vaccine doesn't really kind of stop it entirely.

    I, I mean, I think, you know, I think there are plenty of epidemiologists and, [00:17:00] um, saying that it's possible that it just becomes endemic and it never really goes away. Um, and I think like that, that I, we've not asked anyone, uh, about how they feel about that possibility. I think that maybe would've been something we should have asked about, um, because as far as I can see, it's reasonably likely.

    Um, and yeah, I think it's the sort of thing that people aren't necessarily, I think my, my, I dunno, I could be wrong, but my. My feeling is that the public perception is basically we'll get a, we'll get a vaccine, this will be over, we can go back to life as normal. Um, and while we hope that is the case, it might not be, I dunno how people, how much people think that that is a likely event or how they feel about it.

    You know, 

    Benjamin James Kuper-Smith: I think it's still, I would imagine it's still thought of as a kind of, there'll be a cutoff at in a few months and then it would just be gone. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Um, 

    Toby Wise: yeah, because I mean that, like, I, I think certainly, um, based on sort of [00:18:00] prior pandemics we've, we've built up like an expectation of, of what's gonna happen.

    We think about, um, like H one N one and, um, the original sars that, like, I know that obviously you caused a fair few deaths and, and spread fair fairly widely, but eventually they, you know, they just kind of died out of their own accord. Um, and I feel like that's probably what most people's expectations have been based on.

    Um, so we kinda expect it to, you know, run its course die out and that's it. Um, even, you know, going back to, uh, 1918 flu that, you know, it took longer, but that's eventually what happened. But we don't know for sure that's, how long 

    Benjamin James Kuper-Smith: did that take to. Kill everyone, uh, to go away. 

    Toby Wise: I can't remember idea, 

    Benjamin James Kuper-Smith: actually.

    Toby Wise: I think it was, was maybe it was like three or four years. I can't remember. I dunno, I, it, I'm not a viable historian at all. Historian. Yeah. Um, but I mean away, which is a shame. 

    Benjamin James Kuper-Smith: That would've been 

    Toby Wise: Yeah, [00:19:00] yeah. I know. Yeah. Definitely would've been. But, but then like you, there are, I think because this is quite, um, a, the, the way this virus behaves, the fact that it can spread so easily without killing people and then, you know, people die later on, means that it will spread far more, far more easily.

    And then. You know, it, it may be that it ends up just circulating the population forever and and you can't stop it. Yes. I mean, especially if you, if if vaccines, uh, it seems like vaccines are likely not to be the kind of vaccines that will just stop you getting infected forever, for example. So it may just be that this is a risk that we have to live for forever.

    I, I mean, this is it, this is kind of worst case scenario. It might not happen and I, I'm not someone who knows enough about this to make a decision. 

    Benjamin James Kuper-Smith: And also the first vaccine doesn't have to be the best, right? Yeah. It just has to be better than nothing. And then you're 

    Toby Wise: true. Yeah. Yeah. So 

    Benjamin James Kuper-Smith: maybe we'll get one that's 80, 90% efficacy, but 

    Toby Wise: possibly 

    Benjamin James Kuper-Smith: in three years or whatever.

    Toby Wise: Yeah. And the other possibility is that, you know, we develop treatments that [00:20:00] can, can, you know, mean that you, you're very unlikely to die from it anymore. And then there's not so much of a motivation to vaccinate people because if you end up in hospital, you get this magical treatment and you're fine. Um, so we dunno how things will run.

    Um, but, uh, yeah, I I, I, I feel like it would've been nice to ask people more about how they think things are gonna end. Um, and it would've been nice as well. I, I don't think we'll end up doing this, but then it would be nice to kind of follow people up maybe next year and see how their expectations, like at some point this year compared to what actually happens, um, next year and how they feeling about that.

    But yeah. 

    Benjamin James Kuper-Smith: Yeah, 

    Toby Wise: you could always do more. 

    Benjamin James Kuper-Smith: Yeah. Yeah. Yeah. I mean, that's what I found so impressive with your, uh, in your paper that you actually collected data for almost a week straight every day. Um, I thought that was pretty impressive. 

    Toby Wise: It was a busy week, 

    Benjamin James Kuper-Smith: I had to say, just from a, just from a [00:21:00] practical perspective.

    Um, 

    Toby Wise: also, like what, what's in the paper was maybe only like 20% of what we're actually doing that week, because we also collected data on like two, two or three, I think two like tasks that people did. Mm-hmm. Um, that I basically had to like code up and put together, um, in that week, uh, which took forever. Um, it was absolutely nightmare.

    Every, we kind of barely slept, I think, but, um, 

    Benjamin James Kuper-Smith: but did you actually say one thing? So, I mean, we had, so our preprint, I think was to some extent a similar story to yours that we 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Just had this, I, I mean, in our case it was, we had this idea on a Friday, I think, or my supervisor sends an email around saying, Hey, maybe we can like help in some way or something.

    Yeah. Like, you know, we know some stuff related to, especially he'd done stuff about optimism, bias and that kind of stuff. Mm-hmm. And he said like, maybe we can like, provide, like, help some, somehow, like provide some sort of information about this situation, how people are dealing with it. And [00:22:00] just sent, so you sent that email on Friday and then we talked about it on Saturday and then, you know.

    Created the study on Sunday, collected the date on Monday, analyzed on Tuesday, read on Wednesday, publish on Thursday. 

    Toby Wise: Okay. Yeah. We weren't the only ones being quite so rapid with it is 

    Benjamin James Kuper-Smith: No, but the thing is I really enjoyed it. It was, especially for academia, you have this, you know, you off have this project that take forever, right?

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: And here you go within, I mean, within six days from I idea to Preprint is stupid. Yeah. But, um, and I'm really glad that we don't seem to have made any major errors. That's also nice. Yeah. 

    Toby Wise: No, no, 

    Benjamin James Kuper-Smith: it seems to be We got that. Yeah. We dealt with that, but um, I just found it so almost exhilarating to 

    Toby Wise: Oh, absolutely.

    Benjamin James Kuper-Smith: Just have a project that just goes so fast and makes progress and 

    Toby Wise: Yeah. You, you right. It's not 

    Benjamin James Kuper-Smith: gonna happen again, I don't think. 

    Toby Wise: Oh, no, no. We never get to do anything that fast. And, and, and you're right, it's like, like the as well, you know, there's, I mean, I don't know if the work we've done will have had any real world [00:23:00]benefit, but there's potential that it could have done.

    Um, and you know, nor normally we are thinking with, uh, you know, either what we're doing is quite like basic psychology, neuroscience, whatever, which may have some long-term benefits way down the road. Or we're working in something related to mental health perhaps, which we hope might have more immediate benefits.

    But even then it's kind of uncertain. It's quite nice to be working on something where you're like, this could have a fairly immediate helpful effect somewhere. It might not do, but you know, there's potential for that. And it's better that we. Know stuff about this and put that information out there, then, then we don't.

    Um, and actually I'm beginning to like hear your thoughts on this because I know around the time, like, and since then there's only been a lot of people on Twitter who have been very critical of kind of rapidly produced COVID research. I think that's more aimed at, uh, people like more, more in the medical sciences who are perhaps, you know, putting out stuff about like, um, you know, antibody prevalence and stuff like that.

    Um, which is wildly [00:24:00] overestimated and things like that where they have more immediate public health, um, concerns. 

    Benjamin James Kuper-Smith: Yeah. Yeah. 

    Toby Wise: But I, I dunno how, like, I, I, I, I think kind of balancing, getting the work out there, ASAP, because it's important and timely versus making sure everything is done thoroughly is a difficult sort of line to tread.

    And, um, I think we've managed to do it FedEx successfully, successfully. And, you know, we've made all our data and code and everything open so that if there are mistakes that, you know, hopefully people can spot them. But, um, yeah, that's something that, that's kind of like always been on my mind since we've been doing this and I dunno how you feel about it, um, whether you've had the same concerns.

    Benjamin James Kuper-Smith: Yeah, I mean, so like, I think the, there's a few ways to answer that or to talk about that point. And the, the first thing, I think just in the initial week of actually doing it, before there were these criticisms, uh, or at least before I was aware of them, we, uh. I [00:25:00] mean, we almost did it in a way that my supervisor and I, so cost of Khan and I, we pretty much did the thing, and then the other three authors just made sure we didn't make any Yeah.

    We didn't anything really stupid. That was, that was almost the way we did it, because we were just so in, in the zone and just doing it. Yeah. That, um, we just needed people to just check that we weren't Yeah. Doing something catastrophically, catastrophically dumb. And so in a sense, we were very aware that, of that who were doing it, and we tried to, um, we tried to kind of build up a system that would ensure that we don't make errors.

    Yeah.

    Yeah. And then, but I, I actually, I think just quick, briefly about the criticism. I think a lot of the criticism was aimed towards people doing psychological research online. Right. I thought that was the main criticism. I, in terms of like, oh, now everyone has an idea about Corona. 

    Toby Wise: Yes. Yeah, you're right. I, yeah.

    Benjamin James Kuper-Smith: You know, puts a, puts a questionnaire online and publishes something. Yes. I thought that was actually a [00:26:00] criticism that, um, 

    Toby Wise: yeah, I actually, so I mean, in, with regard to that, yeah. I think, I think there were maybe two streams of criticism. One, one of which was. More into the medical profession where it's more like, you know, you're, you're putting out studies that could influence how we treat this and prevent it.

    And they need to be, you know, people's lives are at stake. And then a psychology where people's lives are at stake as, as important as we think we are. People's lives likely aren't at stake based on what, but yeah, there a lot of people, I think being precise for perhaps, you know, opportunism and kind of jumping on and maybe just not thinking things through properly.

    Um, and I mean, I, I know I, I dunno whether you and Christophe have had the same, uh, experiences, but I have received a lot of review requests, um, since publishing on COVID. Like I issue seem to get one every day pretty much asking me to review a paper about COVID. Um, and I mean, you know, just some of them I've [00:27:00] seen have been great and clearly like really interesting and others just don't really seem to be thought through at all.

    Um, so yeah, I don't think the Christians that may be entirely justified, but it's certainly partially justified. 'cause there definitely are, you know, examples of people just kind of, yeah, putting a questionnaire online. And without thinking it through at all. 

    Benjamin James Kuper-Smith: Yeah, I, so I haven't received any review requests, but I'm not, um, corresponding author on my email.

    Oh, okay. On the thing it's cost off. I dunno. I mean, he, he has received some, but, um, I don, I dunno whether it's what a day, I don't know. Um, but also I think your paper's also more prominent than ours. Um, so yeah. 

    Toby Wise: Sorry about that. I dunno, I dunno how that happened. 

    Benjamin James Kuper-Smith: I think, 

    Toby Wise: I think if anything, yours is probably better done, I 

    Benjamin James Kuper-Smith: think.

    I dunno. Well, you have the longitudinal thing in the, 

    Toby Wise: uh, I guess, 

    Benjamin James Kuper-Smith: right? We don't have that. So I think, I think that that's one thing I actually quite like is that they are in some sense complimentary, I [00:28:00] think. Yeah, they're about the same thing almost. Exactly. Yeah. But in different ways. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Yeah. Um, but, uh, wait, what did I wanna say?

    Um, uh, I mean, so I, I, I understand the criticism about opportunism because in a way, like these things do receive right now, or at least more, our papers both received a lot more citations than most papers would receive in 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Oh 

    Toby Wise: yeah. 

    Benjamin James Kuper-Smith: That short amount of time, right? Like there's def definitely some sort of benefit from it, but, um, I mean we also like talking about this like whilst we're doing and like, are we take just taking advantage of the situation or, and I think we thought that, well, as you said earlier, there is this trade off between speed and accuracy and, um.

    This. Yeah. I mean, that's the, that's the whole basis of what your paper is. And ours also to some extent, it's just this can't be repeated very well. Yeah. Um, I mean, in some sense it can be because there's a pandemic, like not a [00:29:00] pandemic, but there's some, some virus floating. Right. Every five years. Yeah.

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: If you think about, like, there was Zika at the Yeah. Olympics in 2016, right? Yeah. And then, well since the just house, there's been what? Swine flu. Oh 

    Toby Wise: yeah. 

    Benjamin James Kuper-Smith: Yeah. Bird flu. Like, you know, there's one every five years, but 

    Toby Wise: yeah, they're, they're common. 

    Benjamin James Kuper-Smith: Yeah. Um, actually, so I wanted to, uh, maybe actually, I kind of wanted to put this late in the conversation, but let's talk about it now.

    Okay. And this is the general thing of like how to do good science in this situation. Okay. Um, so whilst we're talking about it, yeah. So I mean, in some sense I feel like doing good science during a pandemic is like doing good science in any other situation. Yeah. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: You have to some things, but there are, there is a difference of that.

    You have to move fast to 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Get the early data right, because of, yeah. Um, I mean, I don't wanna ask too specific a question, but do you have any advice for like how, um, or something that maybe you. Would've done [00:30:00] differently or, yeah, I'm, 

    Toby Wise: I think 

    Benjamin James Kuper-Smith: so. Or maybe just broadly. So just to make this a more broader question than pandemics or something.

    Right. This is more specific or ge This is a general question about situations where there is some sort of natural experiment where you have a very short time window to react. It might be like you said, something like a terrorist attack or something, uh, where there's been lots of research or whatever.

    So it's not just about, 

    Toby Wise: yeah, I mean, I've, I, I feel like you, you can't necessarily expect it to be perfect, you know, if, if you were to spend, you know, months developing questionnaires and everything like that, you would get no data. Um, and so you have to just kind of accept that perhaps the, the science that comes out of it won't be perfect, but, you know, it's, it's, it's gonna be there, it's gonna exist, and it might be somewhat informative even if it's not perfect.

    Um, so first, yeah, I think like you can't afford to [00:31:00] be a perfectionist and you have to accept that it may not, may be suboptimal. The other thing is that like, um, it's, I think, I think it's important to be aware of what's already out there. Um, that's something I would've been, I, I mean, I say I would've done this differently.

    I, you know, would've fed more extra. But I didn't have time during week, but I still, yeah, exactly. But I think, like in hindsight, there are definitely things that I could have done better in terms of maybe reaching out to people who know more about, um, certain, you know, aspects of the, you know, pandemic psychology than I do.

    Um, for example, you know, when we went through, through the, the review process for the, for our paper, um, I think we had someone review it who was, uh, a social psychologist and kinda alert us to literature in that area that I just wasn't aware of, but would've been quite nice to know about when we'd done the study.

    Um, I think if I'd spent a bit more time kind of researching stuff, then it may be, I could have developed [00:32:00] some better questions, uh, to ask people, which would've been good. Um, uh, but yeah, you can't just kind of, again, you, you have to accept that it's not gonna be perfect and you can't just spend, you know, a month just reading every, every kind of paper that's out there.

    Um, and I think one thing that remains important, that I think is important all the time, but particularly in, you know, when things are inevitably gonna be somewhat rushed, is to make. Your materials, whether it's code, data, everything openly available so that if there are errors, hopefully they will be spotted.

    Um, and also, you know, posting preprints is helpful because, you know, work like, I think both ours got quite a lot of attention. Um, and presumably if there had been any major errors in there, they would've been identified. I know I saw at least one other Preprint put up. Um, and in the early days of this pandemic, where some I think fairly major errors were spotted and, um, I'm not gonna [00:33:00] name the names, but, um, but you know, spotted on on Twitter by someone who read the preprint, the authors were very good about it.

    They corrected it and yeah, and it worked out. It was fine. Um, but, you know, make, making sure things are available in Preprint form before they're finally published does at least mean there can be this kind of, you know, sort of dialogue between people and, and concerns can be addressed. Um, I, I mean this is a whole other topic, but um, you know, I think we've seen that across all sorts of fields.

    Preprints have been very important this year because, you know, in every field from, you know, medicine, epidemiology, psychology, people have kind of had to rely on them to get the information out there. And I'm, there's certainly been. A lot of activity on Twitter, discussing those preprints and, um, I think you've, for the first time perhaps you've had this interesting kind of like leakage of that, like scientific discourse, um, around these preprints into the public domain as well.[00:34:00] 

    Um, right. Yeah, 

    Benjamin James Kuper-Smith: yeah, 

    Toby Wise: yeah. The preprints are getting reported in the news. Um, and, and as well as I've, you know, discussions are happening on Twitter between scientists getting reported to the news. I see. Um, but the fact that, you know, whatever field you're in, the fact that, that those results are out there and can be discussed and criticized and everything before there is a final published article out there, I think that's been quite important.

    Yeah. Even if it to, maybe to the outside, it looks like everyone's fighting over everything. Um. Uh, yeah, I think that's beneficial. So, yeah. Alright. I think 

    Benjamin James Kuper-Smith: what I, 

    Toby Wise: yeah, 

    Benjamin James Kuper-Smith: what I, I found really funny or we found quite funny was that, so I completely agree what you said about the open si science stuff. I think especially in this kinda situation, it's, well, I mean, you could also argue everyone's doing it so no one has time to check on whether anyone else made a mistake.

    Um, so in some sense, if everyone, you know, it's, it's a flood of information, then again, to some extent Yeah. But, um, what, what I, what I found, well, in a way very good, but just a bit over the top was that [00:35:00]when we published our Preprint, as I said, you know, I mean people dunno this, but this was within six days, right?

    And when my supervisor posted this on Twitter, one of the comments was, your data's not open. Why is, where's your data? Basically, guys? Yeah. We've had it for three days. Yeah. Um, like our, our code is not exactly, I mean, it works and everything, but it, it's not like neat and presentable in that sense. Yeah. Um, well we actually haven't, we then decided like we're gonna, yeah.

    Not do it like bit by bit, but we're just going to 

    Toby Wise: fair enough, 

    Benjamin James Kuper-Smith: publish new preprint and then put everything out there. Yeah. Um, but in a way I was very encouraged that the expectation was if there's a paper, the the data should be out there, which I don't, you know, you wouldn't have had that kind of assumption a few years ago, right?

    Toby Wise: Oh yeah, absolutely. Yeah. I, and yeah, I've, I've also seen multiple things on Twitter, uh, where people have been criticized for this, and I, yeah, I think that's, I think that's fine. [00:36:00] Um, because like, yeah, that is the most obvious way to be able to, to check these things, right? I mean, you're right that we are kind of overwhelmed and not everyone has time to do this, but there, I mean, you know, I've seen examples of people literally going through papers and checking these things and finding errors or, or not, so, yeah.

    Um, I, I, I think, I think that, that we are now expecting this, um, it's probably a good thing, not just for pandemic science, but for science more generally. Um, and, uh, yeah, hopefully this is something that will extend to, to, I dunno, any other sort of natural naturalistic experiments that may occur. Hopefully we don't have anything like this again, but, um, you're right.

    That's very optimistic. They, they do, they do tend to come around quite a lot. Um, so yeah, I, I, I, I would say that yeah, the, these kind of open science practices have, have been very helpful, um, during this pandemic. Do. 

    Benjamin James Kuper-Smith: Is it, so I know [00:37:00] you've used preprints outside of this work, is it something that you do with everything you do?

    Or is there, do you follow specific rules when to use a preprint when not to publish one or, because it seems to me at least doing like nonclinical or it's, well, I guess irrelevant whether it's clinical or not, but for the stuff I do, it seems to me there's no reason not to just publish a preprint when you submit a paper.

    Toby Wise: Yeah, no, I, I, I, I would now put preprints up for pretty much everything. Um, there's, there's only one instance recently where I haven't, which is just because, um, I, I basically have used a task developed by someone else and that person hasn't got their original paper out about it yet. And I want 'em to, I want to, the first one to have their task out there rather than my me kind of stealing it.

    Um, um, so out of respect for their kind of, um, yeah, developing the task. Uh, but other than that, like I can't really see any circumstances. Like most journals are fine with it now. 'cause, you know, a couple of years ago, some journals were, were still [00:38:00] taking the, the view that, you know, if it's preprint, we won't kids to do it.

    Um, I know, but no important 

    Benjamin James Kuper-Smith: ones, right? No. Or are there 

    Toby Wise: journals that's, I think even a co I'm assuming they've changed this now. I think even a couple of years ago, science was, were taking that view. I think they put out an editorial or something. I, I, maybe I'm wrong, but I'm fairly sure I remember seeing this where they were saying.

    Or it's heavily implying that if something is a preprint, they won't consider it. Um, 

    Benjamin James Kuper-Smith: well that's gonna be a huge problem for all of my papers. 

    Toby Wise: Yeah, 

    Benjamin James Kuper-Smith: that's always the first journalism it to. 

    Toby Wise: Yes, exactly. I But now, you know, if you don't get a paper in science, you can explain that. It's just 'cause you put preprints.

    Yeah, 

    Benjamin James Kuper-Smith: yeah. I'm 

    Toby Wise: just, they won't, 

    Benjamin James Kuper-Smith: I'm an advocate for open science. That's 

    Toby Wise: right. Yeah. Yeah. But I mean the, like, if you, it depends how far you go back. But even, I think if you, to go back like four or five years ago, even though preprint servers were in existence, I think many journals would've been far more skeptical about it.

    There's also the, the fact that, um, they're in the preprints are new phenomenon and they're [00:39:00] not how science has been done, at least within our fields for many, many years. So people who are more established in the field, people who are in kind of more senior ranks, um, I know have been more resistant to it.

    Partly 'cause just 'cause they don't understand. And it's like, you know, in the past if you were to put some results out there before they're published, someone would get, would scoop you perhaps, or something like that. Um, which, but that's the 

    Benjamin James Kuper-Smith: opposite, right? Almost. 

    Toby Wise: Exactly. Like you 

    Benjamin James Kuper-Smith: can prevent being scooped by putting a, not a preprint out.

    Toby Wise: Exactly. Yeah. Yeah. If you, if your preprint is out there first, then. You're the first ones to have done it. So it like, it's, yeah, you're right. It, it kind of prevents you from being scooped, but I think 

    Benjamin James Kuper-Smith: there's, so I always feel like if your, if your work is so simple to copy that someone else could just copy it immediately and then submit, like publish it before you, like what is your research that if, if it's, if someone I 

    Toby Wise: know 

    Benjamin James Kuper-Smith: can copy it within like two months or something.

    Toby Wise: Yeah. And I mean, there's a distinction between like, maybe if you tell someone about a study you are going to run, you know, in the future [00:40:00] someone could perhaps take the idea and run it instead of you. And I dunno, I've, I've heard about that happening occasionally. Presumably does. But if you're at the stage where you have a paper and you're gonna put creeping up, like Yeah, you're right.

    How like you'd have to be, it'd have to be so simple for someone to go in and run it and get it published before. Yeah. It's, 

    Benjamin James Kuper-Smith: yeah. It seems like a very 

    Toby Wise: theoretical tool. 

    Benjamin James Kuper-Smith: Uh, 

    Toby Wise: yeah, yeah, absolutely. 

    Benjamin James Kuper-Smith: Yeah. Yeah. The seniority thing is something that I've noticed when, so I mean, I'm a PhD student now, and, um, in kind of the application cycle, uh, during which I got this position, I applied for quite a few other positions and talked to applying about other positions too.

    And I always asked how people thought about open science practices. This was just one thing I just always asked about, so how do you think about pre-registration? All these kind of things. And it did seem to me that there was like an infliction inflection point. Yeah. Right. At like, maybe 45 years or something.

    Yeah. I'm not exactly sure, but somewhere [00:41:00] around in the forties, people just, there was this, if they were younger than that, then everyone was like, oh, yeah, sure we do that. Yeah. And if it was older than that, then a fair amount of people said, well, uh, they started like talking very broadly about science once you brought up the topic, whereas if they were younger, they just said like, yeah, we do that.

    Um, I guess it, it, I, it's really weird because it doesn't seem that revolutionary to someone like me who more or less grew up with that being not the norm, but being talked about at least. 

    Toby Wise: Yeah. Yeah. Absolutely. Yeah. But I, yeah, I guess if, if that was just never something that was considered, then it's not gonna be on your radar.

    And, and I, like, I don't. Yeah, I wanna take a, uh, I wanna give them the benefit of the doubt. I might, I feel that people who are, who are more senior just are kind of unaware of it more than they, they're actually against it. Um, and I mean, I've seen that when people have become exposed to it, they've become on board.

    Like, like when, you know, people realize that actually s prevent you from getting scooped. And, and when, [00:42:00] like when I pre-register stuff, for example, um, that can becoming coming quite handy. Even if you put aside the fact that it's, you know, good practice. And in terms of doing good science, the fact that, you know, you get reviewers comments and they want you to do extra analysis and you can say, well, our preregistration said that we wouldn't do those kind of things because of this, that, and like, then, then the, the kind of more senior person might see like, oh, actually this is helping us deal with the reviewers comments that have always been a nightmare.

    Benjamin James Kuper-Smith: Wait, have you actually, um, preregistered what you won't do? I never thought about it that way. 

    Toby Wise: Um, not, not, not so much, but, um, or more that it's, so I sometimes you might get a, a reviewer who wants you to change your analysis and, and say, you know, you've done this in this particular way, but I think you should do it in this way, even when that might not be really the best way to do it.

    Um, and ordinarily you probably have to agree to them and, and that new analysis would gain some prominence when perhaps it shouldn't do when if you pre-registered something you said, I'm [00:43:00] gonna do it in this way, then you can. You have a bit more leeway to be like, you know, actually, sorry, we can't, we can't put your way of doing things at the forefront of the paper because we've already said we do it this way.

    Um, so there have been definitely been examples where preregistration has, has helped me with, with that also, actually, I, I've, I've remember that for one paper. I think I had a reviewer saying, you know, oh, the, they clearly, even the preregistration was mentioned in the paper. They clearly hadn't seen it or read it.

    And they said, you know, oh, this, this analysis looks like it was just done post hoc. I, I, I don't believe that you ever like kind of preplanned this and just be like, oh no seen preregistration. It's literally right there. 

    Benjamin James Kuper-Smith: Here's 

    Toby Wise: a link. Yeah. So, yeah, it's definitely, so I think when you get those kind of things, like even if you do have someone who perhaps doesn't see, you know, the, the actual kind of benefits of science, they'll see that, you know, from a selfish point of view, it's actually quite helpful as well.

    Um, so 

    Benjamin James Kuper-Smith: I mean, I do, I should [00:44:00] maybe also say, I, I am aware that there are many people who are older than 45 if who are into heaven since I don't want Yes, I know, 

    Toby Wise: I know. 

    Benjamin James Kuper-Smith: Um, and I do understand the, some potential problems associated with it because, I mean, I have one thing that I've, it's basically ready to.

    Submit, but I just haven't, I just want to finalize the code and make it look Yeah. Properly readable because I want to make it open and that's, you know, otherwise I probably would've, yeah, absolutely. Pub not published it, but I at least submitted it already, so I'm aware that it is a time effort involved in it, um, that you just wouldn't have otherwise.

    But then again, I also feel like I probably write my code better. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Because I know, like I'm very nervous that people will read this and just 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Share it to shreds. Right. So I want, I have to do well. 

    Toby Wise: Yeah. And I, I've, I've actually, I had one example where I, in doing that process, like trying to go through my code, make sure all time work properly was, was it looked nice.

    I [00:45:00] caught a bug that actually like changed my results and I would not have noticed that if I hadn't done that. So You're right. Like, it, it definitely has benefits, um, even though it is the early cost, but you, but also, yeah, like it's, it, you basically almost have to like write another paper just in code to go along with the actual paper, which is a bit frustrating, but 

    Benjamin James Kuper-Smith: Yeah.

    And, and then I am aware that ideally, independent of whether someone sees your works, you should do well. Right? 

    Toby Wise: Hmm. 

    Benjamin James Kuper-Smith: Yes. Um, but I think just knowing that people will actually see it or be able to see it. Yeah. I think in, especially if you have, if you deal with the reality of doing research on a time impression of these kind of things, I think it is very easy to not look over it again.

    Toby Wise: Oh yeah. These 

    Benjamin James Kuper-Smith: kind. 

    Toby Wise: Absolutely. Yeah. And I, I mean, the more, the more work I put into making the code and stuff openly available, the more the paper itself seems less important because what I try to do now is, um, and [00:46:00] at least all the kind of important analysis code I'll put in a Jupyter Notebook. So then you have the code, you have the outputs, you have everything, you have, you know, comments to explain things.

    And then when you have that, like the results section of the paper just like, seems so limited in comparison. Um, and I, I kind of always find that sometimes I'm putting more effort into the code than, you know, making the code look readable than I of the actual paper, which is Have you ever 

    Benjamin James Kuper-Smith: referenced a Jupyter Notebook rather than a supplementary information?

    Toby Wise: I haven't, no. I, 

    I'm 

    Benjamin James Kuper-Smith: sorry. Can you briefly explain what a Jupyter Notebook is for the people? Yeah, sure. Who don't, I mean, I dunno, Python either, but at least, I dunno roughly what it is. But for those, 

    Toby Wise: so. So the idea of a notebook, whether it's Jupiter or r Markdown or, or any of these formats, is basically that you, um, have this, uh, page that you can view in a web browser that has the code and the outputs that the code generates.

    So, you know, if you have a, some code to produce a figure, you'll have the code and the figure all in one [00:47:00] place. Um, and alongside that, you can use a language called markdown, um, to basically write comments and explanations, put in tables, everything like that. Um, so you have this document that as well as include texts, includes your code, includes what that code produces, so you can basically talk, you know, talk someone through an entire analysis pipeline with the code explanations, outputs, everything in one place.

    Um, it's, yeah, so there are the, the two major formats are Jupyter Notebook, which was developed, uh, with Python in mind, but um, also works for r and Julia and things. Um, then the, then there's, um, r Markdown, which is specifically for R but I think nowadays can do Python as well, so kind of they're, they're intermixing.

    Um, but yeah, either these formats do basically the same thing. Um, and so yeah, once you've kind of used this format, a results section of the paper that doesn't have any code or anything like that, just seems, um, yeah, [00:48:00] to be lacking half the information that's, that's actually necessary to understand what you did.

    Benjamin James Kuper-Smith: So you'd recommend reading your notebooks rather than your results section. 

    Toby Wise: Probably Yes. Yeah. Yeah. And the great thing is that like, you know, if you want to, then you can download notebooks on GitHub. You download it, uh, with the data, and you can then just run the whole thing yourself. So if you want to change something in the analysis pipeline, you do it and you'll see the results there.

    Um, so it's, it, then it becomes like almost an interactive results section. Um, so yeah, it has, so I really need to 

    Benjamin James Kuper-Smith: get into that. I'm still doing it the old way, uh, which is just so cumbersome. Um, 

    Toby Wise: I do you use R or, 

    Benjamin James Kuper-Smith: uh, MATLAB right now, but, but I mean, we've, I basically want to switch to Python for a while.

    Let's just leave it as vague as that. Um, but um, yeah. Yeah, I mean, I have used R before also, um, but so many years ago that by now I've almost forgotten how to use it entirely. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Um, so it's, it's been [00:49:00] mainly MATLAB for the last few years and. Yeah, but I be, I definitely want to switch to Python and I 

    Toby Wise: Yeah, because I, I don't think I, I'm not aware of any kind of notebook format really for matlab.

    I, I could be wrong about that, but, and or at least not as highly developed as Python. So like the, one of the great things about Python and uh, as well that I think maybe is underappreciated in the scientific community is that they are languages which are used extensively and developed intensively by people outside science.

    Whereas MATLAB is basically scientists and that's it. So you have all these tools that emerge that weren't designed for us, but can be quite helpful, like notebooks. Um, I think they, they, they came about from, you know, data scientists, uh, wanting to be able to explain what they're doing to be other people in the company, um, 

    Benjamin James Kuper-Smith: which is what scientists want to do.

    Toby Wise: Yeah, exactly. But when we don't have the time, or, I dunno, creative freedom, I don't know to, to be able to come up with something like that. Um, [00:50:00] 

    Benjamin James Kuper-Smith: yeah. But it, so I mean, this is now a very much a specula question about the future of publishing, but do you think that, uh, something like there will be online interactive papers where the notebook will be integrated instead of the results section or something like that?

    Toby Wise: Yeah. Uh, and people are trying it, uh, eLife for example, have started doing this. Um, oh, 

    Benjamin James Kuper-Smith: do they? Okay. 

    Toby Wise: Yeah. Uh, I think only recently, and I dunno how many papers they have using this format. Um, so yeah, in, in principle, I think people are trying to. I'm trying to make it work. Uh, the, like, I dunno how much it will catch on or how quickly it'll catch on because I think people quite like papers as they are.

    Um, and which I dunno, I, I, I'm not a massive fan of papers as they are, but I know lots of people who are, you know, of, of my generation who will still like exclusively read papers on paper. Like 

    Benjamin James Kuper-Smith: I 

    Toby Wise: know that, [00:51:00] yeah, it is not uncommon at all. I, I don't do it myself, but I know plenty of people who like to sit down and read a paper, you know, as a physical object.

    Um, and an interactive notebook is never gonna give you that experience. And so as long as, yeah, it'll 

    Benjamin James Kuper-Smith: be like two different versions I guess, right? 

    Toby Wise: Yeah. The, yeah. So I think that's what we'll probably end up doing. We'll have our interactive code in line, we'll have the paper version. It's a bit annoying 'cause then you have to like duplicate the workload essentially.

    But I don't know. We're already doing. Yeah, yeah. But, uh, 

    Benjamin James Kuper-Smith: but so what, okay, so, um, to slightly play devil's advocate here, so what, but why, what would be the advantages? Having the notebook version because isn't just, it seemed to me the results section should tell you what you found under the assumption that you did it correctly, right?

    Yes. So what exactly does having the code there add? Unless you, like, want to replicate it or use, I dunno. Yeah. What, what exactly does it add in terms of you understanding what the [00:52:00] person found? 

    Toby Wise: Uh, I guess one thing is that you no longer have to assume that the person did it correctly because you can, you can see it right there, um, a second.

    Uh, it's more, uh, feasible to include interactive elements. So if you want figures that are interactive, you can do that without too much of a problem 

    Benjamin James Kuper-Smith: interact, like what were being example. 

    Toby Wise: So I know for example, you know, you have a lot of data that you wanna show in a scatterplot or something like that, and you want to select various variables to show or hide things like that.

    Um, or like, you know, uh, like time courses that you might wanna zoom in on and things like that. Um, so that's another element that you, you know, you can't have a normal results section. Um, the other other thing that I found very helpful is, you know, from a, a learning point of view, if you are reading a results section where you can't see like how they've actually done stuff, you, you then have to kind of delve into the methods and try and figure out what they did.

    And, um, if you're trying to learn to, is that kind of analysis yourself, it's not [00:53:00] very helpful if you literally have a result section that includes a code when you are trying to learn yourself how to do this kind of analysis, it makes it very clear to you. So just from the point of view of kind of, you know, uh, you know, learning how to do science, um, and you know, us teaching other people how to do what we've done, uh, it can be very helpful from that perspective as well.

    So, yeah, I mean, you're right. Like it's not always gonna be necessary, but it's. Certainly nice to have. And the the other thing as well is that, you know, if you want to do, you have a notebook, you just hide the cells that, that do the code and you just have the results section. So, I mean, you could just, you 

    Benjamin James Kuper-Smith: necessarily, in a way it almost, it could make results section much more compact.

    So I was just thinking, so one problem I'm having with this study is that, uh, it's, it's been kind of difficult to, there's so much data, um, to, to present, to, to present that in a usable format, um, without having 50 pages of figures. Right? 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: And so it would've been, it would be kind of cool if [00:54:00] you could have something like, I mean, so our preprint we have power plots, but in, we don't wanna use them in the, for most of the figures, at least in the later things.

    And I guess it would be kind of cool if you could just show the bar plots and then you like, click on it. Yeah. And then it opens up like a, a, a density for, uh, plot and the, or like something like a rain club plot or something like that. Yeah, that would be super cool, because then you can like, would it be like nested or whatever you'd call it?

    Yeah. And you can, um, so you have like one figure and three interactive elements. It's, you can see whether the Yeah, yeah, yeah, yeah. That would be 

    Toby Wise: super cool. Yeah. Yeah. And yeah, it also like, it's, um, it, you end up kinda writing out a lot of stuff in the results section anyway. The, all the details, the statistical tests you did and all the, um, you know, the results from them and stuff.

    And you know, if you just have the code, then that kind of can almost. Replace the results section because if you're saying, you know, I'd used an ano to look at this, or you just have the, the line of [00:55:00] code saying you'd run an ano to do this, it's the same sort of thing anyway. 

    Benjamin James Kuper-Smith: Although, I mean, I guess is, is the criticism or one of the obstacles here that then everyone has to learn that programming language 

    Toby Wise: potentially?

    Yeah, that's a downside. Yeah, no, I, I, I, I'm perhaps going a bit extreme there. Perhaps you shouldn't kinda replace 'em all the, the, the text with, uh, with code, but certainly, uh, in the, the output of the code, um, where you can, you know, for some people say you've done loads and loads of tests for whatever reason, and, you know, in a result section have some very big table, um, that gives you all the output.

    If you just have something interactive where you can kind of click around and see what the results are, then, you know, it's a lot, lot simpler to use. Um, I think like as, as a, a kind of top timely analogy, like, I dunno if you've been following the American election, but there are lots of Yes. Um, I think yes, more than I wanna, yeah.

    Most of the world have, but like many like news websites, like New York Times, Washington Post, whatever, [00:56:00] have had all these nice interactive like graphics where you can see how the results are changing, how they're coming in, you know, 5 38 have their. Their, um, prediction model where you can change things and see how things work out.

    Now if you imagine like if they were to have instead done that and written it up as a traditional results section in a paper, like how, how inferior that would be to these interactive, um, things that we get online instead. 

    Benjamin James Kuper-Smith: Yeah, that's, that's actually the perfect analogy because I guess the, the, the in, I think I looked at the Guardian for this.

    Um, they have this and assume most will have something like this. You'll have the, whoever won whatever state and then you click on it and then you see for the counties or whatever it is in the US 

    Toby Wise: Yeah, yeah. 

    Benjamin James Kuper-Smith: For the small sub parts. And I guess that's kind of what I'm thinking about with the figures. You have like one overview of what the main result is and then you can zoom into the, the more granular level.

    Toby Wise: Yeah, and actually we've seen the same thing with, with going back to COVID as well, where, uh, many, you know, there've been [00:57:00] lots of kind of online dashboards people have created or that governments have created where you can view testing results and things like that. Which, uh, part of the reason that's useful in that circumstance is because the data is constantly being updated.

    So that's not necessarily the case in a scientific paper, but it also means that you can, you know, you can look at different areas quite easily in one interactive kind of site. You can zoom in on if you want to see what the test positivity rate of this particular county was in, in April, you can do that, just zoom in and, and you see the data.

    Uh, whereas again, like. You would miss, you would lose so much information if you had to write that kind of stuff up as we do with ordinary result sessions. 

    Benjamin James Kuper-Smith: Yeah, yeah. Then you just have to cut so much stuff, right? 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: You just have to read 

    Toby Wise: it. 

    Benjamin James Kuper-Smith: I, 

    Toby Wise: yeah, I, I'll admit, I've, I've not used these interactive things as much as I should have done.

    Um, it's something I wanna do more. The problem I face right now is that because I have to do make loads of figures for the paper, it's then annoying to, [00:58:00] to go and basically duplicate them, to make them interactive online. But I think as in the future, I hope, I'm hoping this will become a more common thing to do.

    Benjamin James Kuper-Smith: And it seems that with Python, you are on a pretty solid path to being use, like using one of the things that will be used in the future for, or is already. 

    Toby Wise: Yeah. Yeah. So that's helpful I guess. Yeah. Yeah. But what, what I'm hoping we'll do at some point is, um, uh, with the data we've collected from our COVID study, we'll put together like a interactive dashboard where you can view all the results.

    I already have a prototype that sort of works. Um, okay. And, uh, I mean, just from me to use when I've been, you know, looking at, uh, other data we've collected so far has been so nice. 'cause we have so many questions done at different time points and different samples and, um, 

    Benjamin James Kuper-Smith: yeah. 

    Toby Wise: Yeah. It's, it. I would highly recommend learning languages that can do these things.

    Benjamin James Kuper-Smith: Yeah, no, I definitely want to switch. It's, it's, yeah, I mean, it's the same conversation as always. It's [00:59:00] just more convenient to do what you know how to do. Yeah. Even though long term, it's probably beneficial to, you know, make the 

    Toby Wise: switch. I, I, if you're currently doing a PhD, that's the perfect time to do it.

    It might not seem like it, but it is the time when you are supposed to be learning these new things. So, I dunno, just tell Christophe that you're gonna take three months off and, uh, I, Python. 

    Benjamin James Kuper-Smith: Yeah. I, I think I'll just start with some very small projects I have that are very easy to analyze and just get my, what's the word?

    What's the phrase? Fees wet. Feet wet. 

    Toby Wise: Yes. 

    Benjamin James Kuper-Smith: Yeah. 

    Toby Wise: But my, my way of doing it was basically like, anytime I have anything that, that could be done with Python, do it with Python, um, I would, to be fair, I wasn't coming from a lot another language. I was coming from not knowing anything, but still like, just kind of forcing yourself to do it, even though it's gonna take you way longer to do it in Python.

    Um, 

    Benjamin James Kuper-Smith: yeah, I mean, what annoys me, and maybe I'm just being really dumb here, but, uh. I really don't like these things [01:00:00] where you have to configure something yourself. What I like about MATLAB is you just download the thing and then it's there. 

    Toby Wise: Yeah, 

    Benjamin James Kuper-Smith: yeah. Whereas with Python, I, I have used it actually, and I, I kind of half made the switch then reverted back again.

    Um, which is also in part because our testing labs here run on matlab. So that was part of the reason why I did that. Um, and then I just never switched back again. Not, never, not yet. Um, uh, but yeah, I don't like the whole, like you have to, it was always f uh, yeah. I dunno. I had to always think about what to download and Yeah.

    It was always, it wasn't just, you download a program when it runs, you have to kind of make it run. 

    Toby Wise: Yeah. No, I, 

    Benjamin James Kuper-Smith: I get that. That's not my kind of cup of tea when it comes to electronics especially. 

    Toby Wise: Yeah. Yeah, yeah, yeah. No, you're right. I, I'm 

    Benjamin James Kuper-Smith: just, I'm just whinging, 

    Toby Wise: I, I, I keep mean to like, write a quick blog post, just like explaining here is how you get it set up without too much trouble.

    And, uh, because [01:01:00] I, I've, like, I've had multiple people say the same thing to me. Um, and yeah. 

    Benjamin James Kuper-Smith: Yeah. 

    Toby Wise: I feel like I've got the stage now where I've used it so much that I can kind of get everything installed and running without problem whatsoever. But I know when I first started out, it's like, how do I do, how do I even make anything work?

    So, 

    Benjamin James Kuper-Smith: yeah. Well, I, yeah. 

    Toby Wise: Yeah. But 

    yeah. 

    Toby Wise: Uh, the, the other, the other reason to make that switch is that if you end up wanting a career outside academia, um, you know, say you want to go into data science or something like that, no one's gonna be using matlab. So, uh, helps out in that respect quite a lot. 

    Benjamin James Kuper-Smith: Yeah. Is that actually true?

    I've heard that's, I've heard that quite a bit, but I've only heard it from scientists saying it as a hypothetical. I've never, as in that wouldn't be useful. Uh, yeah, like that, the, the exact argument just said, I've only heard from scientists who are in academia. I haven't yet so far, but I don't know. I haven't heard.

    Toby Wise: Yeah, I mean, I, I, I've, I've not worked in [01:02:00] data science, I don't know, but I mean, like, like I said earlier, many of the tools that we're using were developed by data scientists. So tools in Python and our, um, like that, that is just, I mean, like the reason the machine learning stuff in, in Python is so good nowadays and deep learning and everything like that is because, you know, that's just the language people use to do that kind of stuff.

    Um. So, yeah. Yeah, 

    Benjamin James Kuper-Smith: yeah. I think I just have to, uh, do it. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: I have to follow Nike's slogan and just do it. I think that might be the secret to learning Python, um, and not be so caught up in your own comfort. 

    Toby Wise: Yeah. Yeah. 

    Benjamin James Kuper-Smith: The comfort of the comfort of a program I don't really like either. 

    Toby Wise: Yeah. Yeah. I, I'm, I'm not sure anyone really loves the, the stuff they're using.

    Think, yeah. Yeah. 

    Benjamin James Kuper-Smith: Uh, uh, 

    Toby Wise: sorry, I went a bit on a tangent there. 

    Benjamin James Kuper-Smith: Yeah. But again, as I said earlier, that's kind of not the point of the podcast, but, um, actually, uh, [01:03:00] I, I think I've already abandoned any structure for this conversation. Okay. So I'll just, uh, I mean, it's still related, but one thing this, I mean, it goes back to the slightly to the quality in terms of quality stuff that came out during COVID, uh, or about COVID.

    And I've, I've, I'm curious what your experience is. So we have these, I think something like 20 papers now that have cited us, and a shocking amount of them to me, didn't really cite me properly or us properly. Uh, I don't know whether this is just a general thing in academia that people don't really read papers properly.

    Is it? Okay. But I, I kept reading these things and thought, I mean, some, some were just bizarre. Um, yeah. Uh, I mean, luckily, I don't remember. I, I can't even name names, which is good. Um, there was one where I read it and it cited us. I thought, no, we didn't, we didn't say that at all. Yeah. Like, I mean, we're the same topic.

    It's about [01:04:00] COVID. Yeah. Like we didn't say that thing at all. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: And others getting like, details wrong and 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Yeah. It's, I mean, yeah, 

    Toby Wise: I, I've, I've not looked at this for our own citations. I mean, I've definitely looked at it for other papers of mine, and it's not uncommon at all. There was, I, I'm very sure that I saw a study about this recently where they'd actually kind of investigated, uh, the quality of citations.

    I, I, I can't remember what it's called at all. Um, and found that, yeah, citations are abused, basically. I mean, I found it myself. If you, you know, read a paper and they make some claim and reference someone, you're like, oh, that's, that's sounds like an interesting paper. I should read it. You go and read it and it doesn't make the claim that the author said it did.

    It's, I don't think it's uncommon at all. It's a, a more widespread problem. I, yeah. I I definitely don't think it's unique to, to COVID. It may be that happens more often, perhaps if people are kind of rushing things a bit. Um, uh, yeah, I actually, I've, I've not looked at who has cited us that much. Perhaps I, [01:05:00] I should do a bit more.

    Um, I feel like our work is from a brief look anyway, right. It seems like it's probably just been more cited than that. Like there has been some research done in into COVID. Yeah, cite my paper. Like, that's basically it. Yeah. 

    Benjamin James Kuper-Smith: I mean one, one paper that cite, I mean, so I'm looking into this because, you know, this is basically my first, first author.

    Toby Wise: Yeah. I guess, yeah. 

    Benjamin James Kuper-Smith: Preprint. So this is all, uh, kind of more exciting than if you've already done this a few times. Um, so I've, especially in the beginning, I looked at like exactly like what they, well just what the pa like who was citing us Right. And what they were doing. And I mean, there was one paper that say it cited our us for our assumption, basically.

    So we said, you know, just something like, it might be useful to just know, like how people react to this thing if you, because ultimately any government policy depends on people actually doing it. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: That was pretty much, you know, that's just the fundamental assumption behind doing this kind of research.

    And people cited us saying that like, 

    Toby Wise: yeah. 

    Benjamin James Kuper-Smith: Uh, we, we said that, [01:06:00] I mean, we did say it, but it's just not really what we found. It's just the assumption behind it. 

    Toby Wise: Yeah. But I mean, like, I, I find myself doing that sometimes. 'cause like you cite a paper for, because I mean, sometimes you want to make just a really obvious statement, but you can't really do it without a citation.

    So I, you know, sometimes when you're just like making a blanket statement, like people are are anxious quite a lot, you know, people often suffer from anxiety. It's like there's no paper really. Just, it's like saying that. So sometimes you just like cite some review that's just like, like a statement of 

    Benjamin James Kuper-Smith: the topic.

    Toby Wise: Yeah. Yeah, exactly. Um, I mean, at least, at least it sounds like you've not had any negative citations. Um, that'd be, 

    Benjamin James Kuper-Smith: uh, you mean like say, 

    Toby Wise: I don't know if we have Yeah. We 

    Benjamin James Kuper-Smith: did something wrong, or you mean something like that? 

    Toby Wise: Yeah, yeah. Or you saying 

    Benjamin James Kuper-Smith: like, uh, no, they've, I mean, there've been some that were just bizarre the way they cited us.

    Um, 

    Toby Wise: I kind of wanna look at our, but 

    Benjamin James Kuper-Smith: uh, yeah, I feel, and also, so this is something, uh, I feel kind of almost mean saying this, but I was, one thing I found super exciting was when [01:07:00] papers about this topic came in that a lot of them were not from Europe or US and Europe, right? Yeah. Not from the western world, where people in, uh, like Southeast Asian countries, African countries, south American countries, did these studies.

    And we've had quite a few of those studies that did something similar to what we did, uh, but in a different country. And I found that super cool just to, you know, read papers from all, all over the world, but a lot of them were not that great. Like, that was the thing that I found so dispirit and that I kind of don't wanna, I don't wanna trash people, but a lot of them would just like, yeah, there's maybe a reason why I don't read those favors usually because 

    Toby Wise: yeah, 

    Benjamin James Kuper-Smith: they're just 

    not 

    Benjamin James Kuper-Smith: that good.

    Toby Wise: It's, uh, like, it, I don't think it's unexpected because we, we are lucky in the western world in that, you know, we have governments that put huge amounts of, well, first of all, we've been relatively wealthy countries. Those governments put huge lot of money into science. We end up being the 

    Benjamin James Kuper-Smith: training of science also.

    Right? 

    Toby Wise: Exactly. Yeah. Um, and you know, in, I've seen papers Yeah. From countries that [01:08:00] I, I know government doesn't really spend any money on science, so, you know, you can't really blame them. Um, 

    Benjamin James Kuper-Smith: yeah. 

    Toby Wise: Uh, which is kind of, yeah. Unfair, and I think that's, especially, it's kind of frustrating because, you know, COVID is a global phenomenon and, um, ideally it needs to be studied globally.

    There's no reason why you would expect our results in the United States to generalize to Europe, even not that, let alone Indonesia or India or, or somewhere. Um, so I think it's important that we do get this research done all around the world, but you're right, not every country has the infrastructure in place to be able to, to kind of produce such quite high quality work.

    Um, I, I, and I dunno how to kind of solve that because Yeah, I, it's a kind of, it's, it's not, it's not something we can directly help with. So easy. I don't know, I dunno, you're 

    Benjamin James Kuper-Smith: not gonna solve structural inequalities in the world on your end. 

    Toby Wise: Yeah, it exactly. Sadly. Um, yeah, 

    Benjamin James Kuper-Smith: I [01:09:00] mean, some of them was, it was also more linguistic thing where their English just wasn't that good.

    Um, which in a sense is more forgivable. Um, and it's of course very mean if, um, I mean I didn't, I grew up like half English. Um, but um, yeah, for people who are native speakers or close to, 

    Toby Wise: oh yeah. 

    Benjamin James Kuper-Smith: Um, it is of course like an unfair competitive advantage. 

    Toby Wise: Yeah, absolutely. 

    Benjamin James Kuper-Smith: Yeah. But I just found it so sad to think like, oh, cool, these foreign papers and then, 

    Toby Wise: yeah, 

    Benjamin James Kuper-Smith: I mean, some of them also good, like clearly, right.

    Um, 

    Toby Wise: yeah, and act actually, so interestingly, um, a lot of the work that had looked at responses, psychological responses to pandemics prior to COVID did come out of China and places because that's where they've been hit by recent pandemics. Um, so, you know, various, I can't remember which ones now. Swine flu for example, I think, I think started in, in Asia.

    I don't wanna make [01:10:00] famous now. Maybe, um, uh, I know there at least been a couple, uh, pandemics have started in Asia. So, um, the same reason essentially that that countries like China and Japan and South Korea have been more, uh, kind of agile in adapting to the circumstances of pandemic has caused and putting in place measures to, to prevent the spread of disease.

    Um, you know, that also means that they've done more psychological research into it in the past because they've had to, they've been exposed to it more than we have. Um, and there are some, you know, there's some really nice papers that, um, I think we, we cited in, in our, um, that had had shown similar things that, you know, he was kind of optimism bias, for example.

    I can't remember which pandemic it was, but, um, that had been, had been shown before. Um. So, yeah, I think a lot of the research that came that had existed prior to COVID on psychological responses to, to, to pandemics had actually come, come from, from Asia and the Yeah, there are some [01:11:00] really nice papers actually, so, 

    Benjamin James Kuper-Smith: yeah.

    Toby Wise: Yeah, yeah. Yeah. 

    Benjamin James Kuper-Smith: I feel like we are at the, as you you mentioned, like the preparedness of countries. I feel like that is something that in Europe we just lack completely. Yeah. I mean, I know that there are people who thought about this and are aware of this, but as a general population, I, I mean, anytime there was something like that going on, it was always somewhere else.

    And yeah, I mean, like buying masks was just not like, this is basically the first time I've been wearing face masks in my life. 

    Toby Wise: Yeah. Same. Yeah. I think it would've been really nice to be able to, again, this is the sort of thing you couldn't really set up because of the timing and everything, but if you'd been able to do something like some kind of cross-cultural comparison between country, like a country like South Korea for example, versus maybe like the United States or Germany or the uk, um, when the pandemic kind of first hit and see how people adjust psychologically, because I imagine in countries that have, they've done this before.

    You'd see less anxiety, perhaps you'd [01:12:00] see people just being like, well, okay, it's this again. We'll get on with it. We, we know what we're doing. We're sensible. We, we know how to cope with this. Whereas in, in like America, people have no idea what to do. Um, and they have a government who's never really had to deal with this kind of scale of pandemic before and, and presumably feel a lot more anxious.

    But I dunno if that would've been a nice comparison to be able to do. 

    Benjamin James Kuper-Smith: Yeah. Who maybe there will, there must be some data out maybe where you can do some sort of comparison, but then again, the questions might be a bit different and then 

    Toby Wise: Exactly. Yeah. Yeah. I, I, I, and, and yeah. I dunno. 

    Benjamin James Kuper-Smith: I mean, to some extent that's what we try to do in our study because we collected data from Yeah.

    The us, the uk and Germany. Um, this, I mean, so I don't want to be, how should we say, I haven't looked into the country differences too much in the longitudinal stuff and everything, but as far as I can tell, most of it's pretty similar. Um, even though, I mean, so these countries are, um, according to what you just said, similar in that they [01:13:00] haven't had this kind of pandemic before, but they are quite different in terms of how the government reacted to it.

    Toby Wise: Yeah. Yeah. 

    Benjamin James Kuper-Smith: And when they were hit by it also because in Germany it was probably. Two weeks earlier, or I don't know exactly how much earlier, but it was earlier than in the uk, which I think was again, pretty much earlier than in the US 

    Toby Wise: I think so, 

    Benjamin James Kuper-Smith: um, and, but it seems that there wasn't really that much of a difference.

    Uh, but we don't have the, the slight problem is we don't have huge amounts of people per country. So 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Then it suddenly becomes a bit, you know, um, difficult to make these proper claims. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Although, so this is one unexpected finding we had, which is, uh, maybe not that unexpected, but it's something we didn't really plan for, or we didn't expect the results to be this extreme.

    So we asked people how much they trusted the government to deal with the situation. 

    Toby Wise: Oh yeah. 

    Benjamin James Kuper-Smith: And, uh, three countries had very different responses. So the Germans, I bet, I bet, were very, very supportive of the government, like almost ceiling effect level, 90% plus almost. [01:14:00]Um, they were like, yeah, they've, we've got, and to be fair that it has been my reaction.

    So I live in handbook and this has been my reaction too. Like, they seem to take this seriously, do it properly, and 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Well, I 

    Toby Wise: mean, I trust your government as well. 

    Benjamin James Kuper-Smith: Um, the UK had a kind of, they started off like mid-level, I think at the beginning in terms of trust. Then it went up actually because I think the UK didn't, oh yeah.

    So that's my assumption. 'cause you didn't have it that early, but then once the headache dropped lower again than initially. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: And the US just doesn't trust at all. 

    Toby Wise: Yeah. Yeah. 

    Benjamin James Kuper-Smith: They're like, I think they were below 30% or around 30% or something. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Um, although part of me also wonders. Whether that's just a general divide between in the country in general and not related to pandemic response, 

    Toby Wise: I imagine it would be.

    'cause yeah, it'd be nice to that, that's, I think, the thing that is most annoying about, I mean obviously there's no way around this, but it would be very nice if we had had baseline measures like pre pandemic or all these pandemic related things. I mean, [01:15:00] maybe we should just keep surveying people every year just in case the pandemic happens.

    So we can do that. But, um, yeah, 'cause like I imagine Americans are less distrustful of their government generally than, uh, than Germans are. I, I don't know for sure, but I guess that, yeah, 

    Benjamin James Kuper-Smith: I dunno. I mean, one interesting thing is that if I'm, I think I'm correct on this, all three countries had conservative governments.

    Um, yeah. So in that sense they are, I mean, very different kinds of conservatism. But, um, they, it's not like one, you know, had like, I mean, in, in principle I think within the society there were somewhat similar. The governments just, they reacted very differently to it. But yeah, we didn't have any baselines of, yeah.

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: And also we also, we also didn't ask for their political affiliation. 

    Toby Wise: Yeah. No, we didn't either. We, we've already done that more recently. Um, which is, I, I should have done that from the start, but didn't really consider it. I think the, the other, the other problem in the US is that, um, [01:16:00] it's not. Even though it means technically United, the United States, the states are so different from one another.

    Like, um, for example, in California, even though if, you know, there has been a, basically an absence of leadership at the top of the, the government, you know, Californian governments have been dealing with it fairly well. So, you know, if I asked what my overall opinion of the government here is in terms of dealing with COVID, like I'd probably be relatively positive, um, because the local government has been quite good.

    Um, you don't have that so much in, at least in the UK because, you know, there's not, um, I mean there are, there are some level dilution, but it's, yeah, I 

    Benjamin James Kuper-Smith: mean the UK if 

    Toby Wise: anything, yeah. 

    Benjamin James Kuper-Smith: What's the counties, right? 

    Toby Wise: Yeah. But 

    Benjamin James Kuper-Smith: they, but there's huge amounts of them, right? 

    Toby Wise: Yes. And they don't have that much power individually.

    The, like, the larger cities have some degree of power and then, you know, there's obviously the, the four kind of constituent countries of the UK which do have [01:17:00] more power. So they, I've seen that more recently there's been the issue where, you know, Wales completely locked down when England hadn't, and then people like couldn't go from England to Wales.

    And um, yeah. So you may even have, yeah, a similar kind of effect there where people perhaps in certain parts of the UK feel more trust in their government. Um. Than than elsewhere. 

    Benjamin James Kuper-Smith: Yeah. In Germany it's also been, so we have the federal states, there's uh, 16 I believe. Um, I hope I'm not wrong on this anyway.

    I think by now I have the people who know me know I know nothing about politics, so I think I can get away with it. Um, but each of the different states have had different responses like the barrier or in Germany, they in general said like, if it's more than 50 per a hundred thousand, then we, I dunno what exactly what was gonna happen, but they had to at least think about it or talk about it and maybe decide what to do next.

    I mean, that's through the window now because everything's way higher. But a few months, [01:18:00] that was the state, but the various said we're doing this at 35, I think, or something like that. So there were some minor variations in how they dealt with it. And there's also like some states which are so much less densely populated and that kind stuff that they just never had many cases anyway.

    Toby Wise: Yeah, but you also get interesting kind of, uh, points of frustration over here as well when, because essentially like the, the federal government overrules the, the state government, which overrules the kind of county level government. So then, you know, if you have, uh, for example, there are bits in California where that are quite Republican and so perhaps, uh, agree more with the federal government's approach, not their state level government approach, but they can't, like overall their state level go.

    Like it's, um, yeah, I think, uh, there's, yeah, people have, have become a bit frustrated with different kind of levels of government and things like that. And so there are, I I think essentially the, the, the, the question of of whether, [01:19:00] you know, they feel the government is handling it properly is, is a very complicated one, at least in the United States.

    Benjamin James Kuper-Smith: Yeah. Yeah. I mean, we also had a question about what your local authorities, so we did kind speci, uh, divide this a little bit, but the problem is just for, so we have this one longitudinal data set, which is 430 people across the three countries. So, you know, you're not gonna allow, get a lot of people from each state or whatever, and.

    Uh, I mean, you divided, so actually you divided in your paper, you, this geographical analysis where you divided the country into four parts, right? 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: The, or was it uh, uh, was it South California? It's basically, 

    Toby Wise: I can't, I can't remember. Basically northeast, Southwest, 

    Benjamin James Kuper-Smith: yeah. So what is that based on? Um, I'm, 

    Toby Wise: um, so that is, I, I'm not sure who came up with that, but I think it's, I think it was, um, a division that's used in the, the United States census and various kind of government statistics.

    They tend to divide the states up into these [01:20:00] four regions, um, which I guess is done based on, I dunno, cultural factors and things like that. I mean, like, just kind of intuitively those four regions do kind of divide themselves culturally to some extent. Um, I'm not American so I don't know too much about this, but, uh, um.

    And politically as well. Uh, for example, like the, the Northeast, it's very left-leaning as is the west coast, whereas the sort of, uh, the Midwest and the South are, are generally more conservative leaning. Um, so yeah, but I, I mean we just took those categories 'cause they already existed and we, we, um, we weren't really able to look that much at, um, well, we weren't able to look very well at differences between them because it was kind of confounded by the case numbers.

    So the Northeast, um, so like New York for example, was hit really badly quite early on, so they were socially distancing far more than, [01:21:00] uh, you know, people on the west coast were for example. But that's just because they needed to. Um, whereas I mean, to be fair, we probably needed to at that point as well.

    We just didn't know it. Um, so yeah, we, we weren't able to look at that properly. We also didn't have any finer grained location data than, than the state. Um, uh, so we couldn't really, uh, I mean it's not something we probably should have done, but we, uh, we couldn't look any, in any more detail, um, than that, which is a bit annoying.

    Benjamin James Kuper-Smith: But even, uh, so we actually asked for the city they live in, but even that's. I mean, what are we gonna do with that? In hindsight, it's got, yeah. Now it's, it's a bit like, okay, you have a city and a state, but well, number one, we don't have enough free peop enough people to really do anything with this information.

    Um, but the other thing is also, even if we, did I still, yeah, I mean, for the political, I guess it depends on what the question is, but so for trust in the government, you could maybe use the average, yeah. Like you, you take the [01:22:00] percentage of who voted for which party. Yeah. And take that as a proxy for this person, is that likely to be in favor for the government or not?

    Um, but of they, uh, it's, that's still very, very time consuming and probably not very, I'm not sure how, what, how likely it is. You'll get an interesting result on the end. 

    Toby Wise: Yeah. The, the other thing that we've, um, we have in our data that we've not really looked at yet, um, is there have been multiple significant kind of cultural, political, social events that have happened in the United States the last, I mean, the last few months.

    So since the pandemic began, you know, obviously there are the, all the, you know, massive, you know, kind of protests against racism, police brutality and things that happened early in the year. Um, and then there's been the election and the whole round up to the election. So. Um, we, we will presumably be able to look at some effects of those to see how they have perhaps kind of interacted with [01:23:00] the things, um, related to the pandemic, uh, which will be interesting, interesting to see.

    Um, 

    Benjamin James Kuper-Smith: how, how would 

    Toby Wise: you 

    Benjamin James Kuper-Smith: do that? 

    Toby Wise: Not sure. Have not thought about that yet, but, um, 

    Benjamin James Kuper-Smith: you've got 

    Toby Wise: the data 

    Benjamin James Kuper-Smith: and a lot of 

    Toby Wise: hope. Yeah. So, yeah, I think like for example, um, I, I imagine there'd be some degree of kind of, uh, perhaps divides along kind of political lines, but also along racial lines, um, particularly around when the protests were happening because that was kind of, part of that amongst a load of other things were high, was highlighting the fact that, you know, black people, other minorities in the United States have been far worse affected by the pandemic than, than white people.

    Um, so I dunno whether we'd, you know, perhaps see some kind of effect there. Where I, I, I dunno what the, the kind of splits are in our sample. Actually, I've not checked that in a while. Um, but perhaps, you know, that may be [01:24:00] highlighted to black people that they are. You know, at a high risk, sadly, and therefore should, you know, maybe perhaps made them feel more angry or anxious, uh, I don't know.

    Or maybe, you know, you had sort of more conservative leaning white people who kind of felt like, you know, the protests are gonna make everything worse and therefore show heightened anxiety because they think that these are super spreading events. I don't know. Um, it'd be interesting to look at. Um, it's been a very, a very interesting year.

    Definitely. Um, in, so, uh, this has been such a 

    Benjamin James Kuper-Smith: weird year. 

    Toby Wise: It, it really has. So, uh, yeah, there's a lot of stuff we were to look at in the data, and I, I'd like on, on that topic, I think, I think, um, there have, there there's not really been enough work considering these kind, the, the effect of these in inequalities that exist, particularly in the United States on how people are psychologically dealing with pandemic.

    I mean, it might just be that this, this work will come out as, as, as [01:25:00] people kind of get it published. But, um, yeah, the fact that you have these particular really kind of entrenched racial divides in the United States and where, you know, black people are at far worse, far high risk of getting the, the disease of, of dying from it once they do get it.

    Like presumably that's having a far greater psychological toll on them than it is on white people who are relatively safe from it. I mean, we, like in, you know, where I live in la, um, it's incredibly obvious that the areas that have been affected are the poor Latino. Areas mostly. Um, whereas the wealthier white areas, uh, have much lower case case loads.

    Like this is not like a a, a small split whatsoever. And presumably you'll then get kind of massive psychological effects that differ between those, those kind of groups. You also have the issue of, you know, where, where kind of poorer people generally have to kind of be confronted with risk of getting the virus 'cause they work in occupations where they have to be amongst other people.

    Whereas we are very fortunate, like we can stay home, I [01:26:00] isolate ourselves. We don't have to come into any good contact with other people if we don't want to at all. Um, and like I think that's something that's not really been, we've not addressed that partly. 'cause we didn't really have the, we didn't collect the data.

    That's another thing I probably should have done to begin with. Um, uh, data 

    Benjamin James Kuper-Smith: on what you mean like on ethnicity or on socioeconomic status or? 

    Toby Wise: Yeah, because I, I mean, to be honest, we, later on we did collect more data on socioeconomic status and I've not looked at it yet. Um, and also data on ethnicity, but we didn't do that to begin with, um, in that very first week, which I think in hindsight we probably should have done.

    Um, at first, to be honest, part of it was just kind of practical. Like I, I wanted to, um, just for the sake of avoiding any questions with the IRB, I wanted to reduce the amount of like personal, potentially personal information I collected. So the, the minimal extent, um. But I think, yeah, that would've been something that'd be quite interesting to look at in more detail.

    We, yeah, we did a little, uh, some kind of analysis on that based on like education levels in our data. But that's not a [01:27:00] perfect measure of SES by any means. Um, and I, I mean, personally, I think, like I know I, I'm not for the United States. I've only lived here like a year, and I didn't fully appreciate until this year, like just, uh, how divided the country is in terms of how well off people are.

    Um, and so I didn't really necessarily. Uh, consider it to be quite such an interesting question. Um, as much as I should have done until, until the whole year happened and, and it was, uh, made blindly obvious to many of us. 

    Benjamin James Kuper-Smith: Yeah. But 

    Toby Wise: yeah. 

    Benjamin James Kuper-Smith: But then again, there's only so much stuff you can think of in the beginning, right?

    Toby Wise: Yeah. And the other thing as well is that I, yeah, I I I didn't wanna like necessarily stray into other people's kind of research areas because I don't wanna end up, I I'm sure you had the same kind of feelings where you don't wanna end up kind of doing something that you don't really know anything about just because you can.

    Benjamin James Kuper-Smith: Yeah. 

    Toby Wise: Um, like there, there are a lot, there are quite a few things we discussed in the earlier [01:28:00]days of the, of doing this work where it's like, oh, it'd be interesting to look at that. But then we're like, actually, we don't know anything about that. It's not really our area. We should probably let people, I mean, the one example is mental health, for example.

    At first I was kind of thinking, you know, it'd be interesting to look at mental health effects of this. But then I thought, you know, there will be clinicians out there who are gonna do this far better than I could. Um, and you know, we've seen those studies, um, done and published. Um, so we end up staying away from mental health, for example, because yeah, we, we would just be doing something far more poorly than, than other people would.

    So, yeah, there are quite a few questions that we, we perhaps also ignored just 'cause we thought we wouldn't do them very well. 

    Benjamin James Kuper-Smith: I mean, I think one thing that just occurred to me that I think is also important to bear in mind is that, at least in the. To me it was always very important that the questionnaire doesn't, or the study doesn't become too long.

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Um, so we try to keep it up to like 20 minutes or something and, um, 

    Toby Wise: [01:29:00] yeah. 

    Benjamin James Kuper-Smith: There's only so much you can ask with a 20 minute experiment. Um, 

    Toby Wise: yeah, I 

    Benjamin James Kuper-Smith: mean, everything seems very easy to add on, but then suddenly, you know, it's another five minutes longer and 

    Toby Wise: yeah. Yeah, that timing was definitely an issue.

    'cause like I'm, I think all, any of us who have seen someone complete questionnaires in the lab or do a task in the lab, know that after not very long, people start falling asleep essentially. And, um, data quality diminishes rapidly. Um, so like from that point of view, you don't really want someone to be sitting there answering questions for like an hour and a half.

    Um, also obviously we had the issue of money, like we, we have to pay people to do this. And, uh, I 

    Benjamin James Kuper-Smith: mean the, the mortgage tricky thing was not to have the money to pay them, but to have the money on the prolific account. That was the, that was the more tricky part. I think Kof pretty much put like his private money up for the first 

    Toby Wise: Yeah.

    Benjamin James Kuper-Smith: At least first entire data collection because 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: You know, with German bureaucracy, you're not gonna get, [01:30:00]because we just started like online data collection. We were, it was, we just kind of started doing that with prolific and, um, yeah, with German Pro, you're not gonna get 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: A thousand or two, 3000 euros onto an account within like a day.

    That's just not gonna happen. 

    Toby Wise: Yeah. I, yeah, I think I was, I think I was like doing it myself to begin with as well and yeah, a bit of a nightmare. But, um, I think it's, we were running 

    Benjamin James Kuper-Smith: back by now, but 

    Toby Wise: I hope so. Really. We, we were very lucky to, to, so Dean, um, managed to get a grant from Caltech. So Caltech actually put out this funding call saying, what is the Merkin Institute at Caltech?

    Put out this funding call saying, you know, we want studies looking into the effects of COVID. 

    Benjamin James Kuper-Smith: Oh, really? 

    Toby Wise: Um, so we applied for that? 

    Benjamin James Kuper-Smith: Or was that 

    Toby Wise: Yeah. Oh no, it's very fast back at the beginning. So it was I think within a month or two afterwards, so Oh. After the first part we had to find other sources of money for, but um, I see.

    Benjamin James Kuper-Smith: Okay. 

    Toby Wise: But from then on, uh, we were, uh, [01:31:00] yeah, it was very generous of them to, uh, to give us the money to, to cover the rest of it, um, which obviously allowed us to extend it far longer than we would've done otherwise. Yeah. Um, so yeah, we were fortunate in that sense. 'cause otherwise yeah, you, you can't really just keep going for months, um, with no money.

    Um, 

    Benjamin James Kuper-Smith: especially 

    Toby Wise: another, 

    Benjamin James Kuper-Smith: you've been doing it every two weeks, right? 

    Toby Wise: Yeah, exactly. Yeah. Um, it's cost a fair bit of money. Um, and I mean, I, hopefully it'll turn out to be worth their money. Um, I, I, hopefully it will. I, one, one other thing is that I, I've seen a lot of studies that haven't paid people for participating.

    Um, and that's something I always feel kind of like ethically on the fence about, because like, I mean, I know that people are happy to do it because they want to contribute and that that's fine, but equally, I kind of feel bad not like compensating people for their time. Um. Well, that's 

    Benjamin James Kuper-Smith: what prolific is also so [01:32:00] adamant about.

    Toby Wise: Yeah, exactly. Yeah. 

    Benjamin James Kuper-Smith: You have to pay them what, like five euro an hour, five pounds 

    Toby Wise: an Yeah, something like that. Yeah. Um, but then, you know, when you have like, uh, some of the, the really large scale studies of mental health, for example, that I've seen particularly in the uk, um, which I don't think have paid people just because, you know, they, they've ended up getting tens of thousands of people to do them.

    And, you know, paying people would be a kind of a hindrance in that respect because, I mean, A, you need to find the money, but b, just like the infrastructure of like setting up payment system to pay like 80,000 people, it's just like, yeah, you can't, you can't get everyone to sign up to prolific just to do it.

    It's, yeah. So I dunno, that's, that's something that I've been kind of uncertain about over, over, over time. I didn't want to myself, not pay people, but I appreciate, you know, other circumstances. That's kind of been the only way to do it and people have been apparently very happy to, to kind of participate for free.

    Benjamin James Kuper-Smith: I feel like in this situation, it does seem to me it's a bit different. I mean, in a way also, I, I feel [01:33:00] like slight, so my PhD is not on this topic. Exactly. Yeah. Um, so I don't, in that sense didn't, I didn't have the, or still don't really have the precise knowledge and expertise about, uh, risk perception of these things.

    That's, I mean, we have other people, um, Christoph and Julia, Ian. I don't dunno whether gobble, but the other two definitely have published on this before. Um, so that comes from their perspective and I'm more know how to run these online studies and do it fast and yeah. Kind of generic science stuff. Um, yeah.

    And now I forgot where I was going with this. Um, wait, what were we talking about? Um, uh, 

    Toby Wise: paying people money and stuff and 

    Benjamin James Kuper-Smith: Oh, yeah, exactly. 

    Toby Wise: Online. 

    Benjamin James Kuper-Smith: Exactly. So in some sense, I also feel like I am, uh, giving away time for my PhD for this that's not related to my thesis. I mean, in some sense it is a, actually this is, oh, I'm so smooth in my transition.

    If I hadn't just said how smooth I am. Um, because it's something I did want to ask you about is whether [01:34:00] this, um, how much this relates to your kind of normal work or to, it seems to me, maybe to you it's a, it's more related than to me, where it really is just something that's completely different. 

    Yeah.

    Toby Wise: Um, 

    Benjamin James Kuper-Smith: but I was just curious how you, how you think about this also in terms of, uh, or maybe let's just keep it at that and then I'll ask other questions later. 

    Toby Wise: Okay. Yeah. I mean, it's, it is definitely not like literally what I had planned to do, um, during this postdoc that I'm on. Um, but it's ended up being, it fits very nicely with what I was kind of intending to do because everything I'm doing normally is, is in the lab looking at how people learn about kind of threat basically, um, and how it relates to mental health.

    And this essentially is kind of allows me to test a similar, you know, similar hypothesis in the real world. So it is not quite the same, but um, it definitely allows me to kind of answer the same questions in different ways. It, one other thing that we're, that we, uh, have collected data for, we plan to do is, um, I've, [01:35:00] I've used a task that I've used in the lab before, online in, in the people doing the study, and that looks at how people learn about threat and safety.

    And so I've not looked at the data yet, I don't know what the results are, but this will give us an opportunity to look at whether what we find in that task kind of translates to reality. Whether people who learn about threat more in this kind very contrived task do actually learn about real world threat more quickly.

    Um, and that sort of external validation is something that we don't do very often, mostly 'cause it's just very hard to do because, you know, when, how often is it that people are exposed to a serious like, threat like this in real life? It's very rare. Um, so I think as well as being kind of generally just sort of another way of testing the hypothesis I want to test also gives me a way to validate stuff that I can otherwise validate.

    Um, the other thing is that I, I, sorry, I, uh, like, I think we bring [01:36:00] perhaps a different perspective to this kind of research because generally this work on risk perception, um, is done by people in other. Kind of areas of psychology, particularly in social psychology, for example. Um, whereas a lot of work on, on how people perceive risks from various things in their environment.

    Um, but it's quite interesting then for us to come in from this more kinda lab-based way of doing things and think about how we might be able to explain how people, um, react to, to threaten the real world, uh, when it's not usually our, even though we know a lot about it in the lab, it's not really normally our area.

    Benjamin James Kuper-Smith: Yeah. Um, I was, so, I was just want to ask about the validation. You mentioned, um, if you can't control this real life experiment of the pandemic properly and you can't measure everything properly, how do you know whether, uh, you are not, like, if it doesn't validate, are you not validating it because you can't cont [01:37:00] like the actual 

    Toby Wise: Yeah.

    Benjamin James Kuper-Smith: Do you know what I mean? I'm, I'm not making a clear sentence here, but Yeah, I know what 

    Toby Wise: you mean. Yeah. I, but I, I, yeah, I mean, there's always gonna be that caveat isn't there, that, that this is a very uncontrolled real life situation. Um, and so yeah, if something doesn't predict real world behavior, it may just be that it's because the real world is so unpredictable.

    Um. I still think it's worth doing though. Um, and you might end up finding kinda interesting patterns in the data regardless. Um, yeah, so I, I don't think it'll be conclusive, but I think it's still important and interesting. Um, I'm hoping that it, the tasks we use to predict real well behavior, um, maybe they won't.

    Uh, but, uh, I, I think yeah, it's valuable to do either way. It's something that we should be doing anyway, but it's just very hard to do, particularly with threats anyway. 

    Benjamin James Kuper-Smith: Yeah. Um, God, I keep forgetting what I [01:38:00] wanted to say. Uh, anyway. Okay, then I actually wrote one thing down, so don't forget, and this is, I wanted to ask her, this is, I mean, this relates to the general question of how much this relates to your normal work, and that's the, I, so I was just, so you said you're gonna continue collecting data on this.

    Um, but I also, is this something that where you think this maybe adds like a new perspective to your own research and that you almost, you do want to collect data, uh, or you are almost waiting for some sort of event and you are like more on the look at now for these kind of natural experiments or, um, yeah, 

    Toby Wise: yeah.

    Yeah. I, I mean, I think I, I would be more so than I ever have been before because yeah, I've definitely learned that this is actually quite a. Effective way to, to, um, yeah. Validate things in the real world. And, uh, and it has, yeah, it definitely has caused me to think more about, um, how the work we do in the lab actually relates to real world threat [01:39:00] perception.

    Um, 'cause often it's, it's like the kind of behaviors and responses we're looking at the lab are so unrealistic. Um, this is, this is, uh, it's kind of gone hand in hand with a broader area of work that I've been doing with Dean, where we've been kind of thinking about how we can assess threat responses more in more ecologically valid ways in the lab.

    Um, so I've, I've kind of been thinking about that in our lab, more lab based work. How do we make things more ecologically valid? And then it's kind of gone hand in hand with the COVID work, where we're thinking about how does what we do in the lab actually affect, affect how people pa behave in the real world.

    Um, I think specifically one thing that I, I realized is that, um, uncertainty in the real world is not very easy to, um, kind of conceptualize. We, we have these neat kinda definitions of uncertainty that we use in our, our kind of models and our lab-based tasks that like, uh, all seem [01:40:00] very, very, uh, neatly understood.

    And, uh, yeah, we, we, we, we, uh. We kind of know what we're talking about. We know how to measure them, we know how to model them and everything. When you get to the real world, you are, it almost seems like you're more uncertain about the type of uncertainty people are experiencing. Um, it, everything is a lot more complicated.

    And I've been thinking about ways, ways to kinda measure that in the real world. It's like, how do you measure how uncertain someone is about their threat from COVID? It is actually a very difficult question to ask people. Um, because like I said, it's, it's difficult for us to conceptualize where the sources of uncertainty are in the first place.

    And secondly, like people are very, are not very good at reporting their uncertainty. Um, it's difficult. I mean, it's, that's on us as much as them. Like we need to find better ways to, to understand how uncertain people are feeling in the lab. This sort of thing is far more controlled, so it's far easy to do in the real world if we want to look at how people are kind of perceiving this kinda uncertainty.

    Um, I've realized through doing this that it's very difficult and we need to [01:41:00] figure out better ways to do it. Um, so yeah, I think it's definitely helped me kind of, uh, think more about these things and um, so maybe I'll be looking for, just do it again. 

    Benjamin James Kuper-Smith: It's problems rather than offering solutions 

    Toby Wise: or. It's, I mean, it's offered solutions in terms of, you know, perhaps it allows us to, to validate things and things like that.

    But yeah, it's, I mean, most science causes you more problems than it solves, right. The whole, every, every result you get raises more questions than it answers. So that's a better way of 

    Benjamin James Kuper-Smith: phrasing it. 

    Toby Wise: Yeah. This is, this is done exactly the same. Um, uh, and I think it's also just maybe aware of, I think you, you kind of get like wrapped up in, in the lab basically, and you kind of ev ev your, your way of thinking of everything is about how things work in the lab.

    And you sometimes forget that the outside world actually exists and that people aren't necessarily gonna react in the same way as they do in the lab. And you can maybe just think [01:42:00] about that a little bit more when you're doing these, these tasks, um, that we do. Yeah. 

    Benjamin James Kuper-Smith: Yeah. You don't want to become a a, a, a, what's sort word?

    Like a lab behavior. Yeah. Psychologist. 

    Toby Wise: Yeah. I mean, you, you don't wanna spend years developing these neat tasks that work perfect in the lab, but don't have any relevance to the real world. Um, which I think is quite an easy thing to do if you don't have, have things occasionally that make you, kind of, force you to think about how people actually react in, in reality.

    Benjamin James Kuper-Smith: Yep. Yep. 

    Toby Wise: So not, not to put a downer on everything, everything we do, but 

    Benjamin James Kuper-Smith: I mean, it is just a, I mean, I also feel like to some extent, if you have a. Task in which you can very precisely manipulate variables. I feel like it should have some sort of real life effect. Right? I mean, at least you'd, so 

    Toby Wise: that's a 

    Benjamin James Kuper-Smith: fundamental assumption, is that 

    Toby Wise: Exactly.

    Yeah. It, it is the fundamental assumption, but it's when [01:43:00] we don't test. So it 

    Benjamin James Kuper-Smith: has it, uh, yeah, I think that 

    Toby Wise: as, as a, 

    Benjamin James Kuper-Smith: this paper from economics, which is this from 2004 five or something, which is sold something like what do laboratory results tell us about the real world or something like that. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Uh, I haven't read it yet, but it sounds like a very important paper.

    Toby Wise: I haven't either. I'm 

    Benjamin James Kuper-Smith: vaguely afraid of it. Yeah, exactly. That might be why I've not read it yet. 

    Toby Wise: I, I think like, one, one thing that's, uh, quite salient for me is, so most of the work I do is more like in the field of computational psychiatry. So it's understanding how, if we learn and make decisions and how that might relate to mental health.

    And one thing that, you know, I've seen my work, other people have seen this, is that, um, the way people, which, the way in which people learn about positive and negative events that happen to them, uh, is associated with symptoms of anxiety and depression. So it seems like people who are more, uh, kind of depressed but more anxious in certain ways tend to learn more about [01:44:00] negative than positive outcomes.

    So I 

    Benjamin James Kuper-Smith: think that is Christoph's paper. 

    Toby Wise: Yeah, I think Christophe has that. I think, um, I dunno whether he looks it specifically in response to anxiety and depression, but there have been other people in the house. 

    Benjamin James Kuper-Smith: I think he had, I think he did a thing with. Depressed patients, I 

    Toby Wise: think. Okay. 

    Benjamin James Kuper-Smith: And, yeah. Anyway, 

    Toby Wise: I need to, I need to look at his paper.

    Um, but, you know, there've been a few studies Chinese, now it's, it's, it's quite like, it's quite a neat kind of, um, explanation of these symptoms because it makes sense that, you know, if you are learning about, sorry, if you're learning about, it was 

    Benjamin James Kuper-Smith: a fly in my face. Okay, now I've got it. Okay, now I've got it.

    Okay. 

    Toby Wise: Okay, great. Um, yeah, this makes a lot of sense. 'cause if you're learning about bad things more than good things, you are probably gonna feel more anxious and more depressed. That makes sense. Um, but what's kind of implicit shoes there? That's the 

    Benjamin James Kuper-Smith: police siren you warn me of, right? 

    Toby Wise: It's, yes, yes. Let that go.

    I'll, I'll let that go. There's always police. This has been a very 

    Benjamin James Kuper-Smith: disrupted response. 

    Toby Wise: [01:45:00] Yeah. First the 

    Benjamin James Kuper-Smith: fly, then 

    Toby Wise: the police. I, I, I might just start all over again. Yeah. Um, yeah, maybe I'll just start again. So, uh, so yeah, lots of studies have shown that people learn more, uh, from negative than positive outcomes if they're more anxious and depressed.

    Um, and that makes sense because if there things that happen to you in your everyday life that are bad and that are good, if you are kind paying more attention to the bad than the good, you're gonna probably feel worse. You're gonna feel more anxious and more depressed. But that kind of implicitly assumes that, that the way in which you learn about these things does kind of influence your mood by kind of, uh.

    Uh, biasing you to, to, to, to pay more attention to the good thing, the bad things in your life than the good things. Like that's a very clear assumption there. We, we've not been able to test that in the lab. All we've basically shown is that people who have this kind of bias in learning have these symptoms, but that, that critical link we can't really test very easily.

    'cause it, you know, that's something you need to do in the real world. Um, and so despite the fact that as a fundamental assumption, [01:46:00] we've never tested it. Um, and so, you know, this pandemic allows us to do that to some extent. That's what I'm hoping to be able to do with the data. We have to see whether the way in which we learn doesn't, how we learn about the threat of the pandemic, which is a real world threat.

    Um, and if it doesn't, then that would be slightly concerning. But, but if, even if it doesn't, it is, it's not a nice result to get. And maybe there will be issues around the controllability of, of the pandemic that could explain it. I think it would be important because it's something we really do need to think about if we want to, in, in, in a, in a more clinical sense, if we want to translate these, um, these kind of results to, to helping people who suffer from, uh, mental health problems.

    Uh, so yeah, whether, whether, whether the outcome of, of this, this, uh, this research is good or bad, it'll be important, I hope. 

    Benjamin James Kuper-Smith: Yeah. Uh, it just occurred to me when you talked about. Your research or this kind of research, you said earlier [01:47:00] that you're not an expert in mental health or anything like that, and that's why you didn't include it.

    But it, from what you just said, it sounds like you're doing stuff that's at least pretty adjacent. 

    Toby Wise: Yeah. Sorry, I, I should have been more clear on that. I, I wouldn't say, I mean, my, my PhD was in a mental health related field, so I know a fair bit about it, but, um, I'm not clinically trained. Um, and also, um, it's not really in terms of the kinda more epidemiological aspect of it, just kind of producing sort of more descriptive information about how people's mental health is affected by the pandemic.

    Um, and, uh, how various factors affect that. That's not my area of, of work to learn more about the mechanisms that lead to those symptoms. So 

    Benjamin James Kuper-Smith: Okay. 

    Toby Wise: In that sense, it's not really my, my kind of area to, to focus on. Um, uh, so I'll leave, I'll leave that to the psychiatrist and the clinical psychologist to deal with 

    Benjamin James Kuper-Smith: something to do.

    Toby Wise: Yeah. Otherwise 

    Benjamin James Kuper-Smith: I'll just get bored. 

    Toby Wise: Yeah. I think like, what, what [01:48:00] hopefully will be nice is we end up with this kind of convergence of findings where. There'll be research out there showing that people's mental health has sadly deteriorated because of this. And then we'll have our more kind of mechanistic research showing that, you know, perhaps it's the way, the way in which people have learned about this threat has kind of perhaps had some impact upon that kind of deterioration of people's mental health.

    Obviously there's a lot of other stuff going on, but perhaps that'll be something that comes out of it. Um, 

    Benjamin James Kuper-Smith: yeah. 

    Toby Wise: And yeah, actually as well, something on a slightly different topic, but something that this has made me acutely aware of is, uh, in some sense how unimportant, how it work is in the field of, um, mental health research because, um, it may be that the way in which people are learning about their environment, about the good and bad stuff that happens to them does play some role in how anxious or depressed or whatever they become.

    But there are a huge number of other factors going on right now in the [01:49:00] world, such as their, like actual risks, their, you know, whether they've lost their job, things like that. You know, people, you know, being thrown into poverty and things that, that are gonna have a far bigger effect on some bias in, uh, than some bias in their learning.

    I think it's also made me appreciative that, about that in my everyday work. Like ultimately I think what we're doing is important, but it's not, it's, there are far bigger issues that that could be addressed than, um, than what we are looking at. Um, 

    Benjamin James Kuper-Smith: then again, isn't it just that in some sense. The work you are doing is less obvious and less trivial.

    I mean, knowing that if you lose your job, most people feel bad about afterwards, it's, you know, it's kind of obvious. Um, yeah, 

    Toby Wise: and I agree, I agree. It's de it is definitely kinda less obvious and, you know, there is always the fact that some people who don't lose their jobs will become depressed. There are people who lose their jobs and would be fine in terms of their mental health.

    Um, and that's where what we do kind of is helpful. But, uh, I, it's, I dunno, [01:50:00] it is made me feel that from a more, it's more of a political issue I guess. But if you think the, the amount of money that is poured into what we do is what, as it's nice to, as it is that keeps us in jobs. And I think it's important, um, there are, uh, more salient things that, that could be addressed, that would have a bigger effect.

    Um, so I mean like in United States for example, you know, U European countries have generally done a lot better at keeping people paid, whether that's through kind of, you know, furlough schemes, things like that. Whereas in the US they've not done that really at all. And in terms of people's mental health, that's a, you know, that's not, that's not something we can help with, uh, in our work.

    That's something the government has to do, but it's arguably gonna have a much bigger effect than anything we can ever do. Um, and so, yeah, it's, I guess it's a more a political question of funding priorities. Do you, is that something you want to invest in? 

    Benjamin James Kuper-Smith: But then again, I mean, that's the whole thing with basic research, right?

    That you never know where it's gonna lead. [01:51:00] Yes. Yeah. My, my favorite example of this is still that, you know, in, I mean, to, to, to paint a very, very broad story here, to paint a story here, whatever. Um, let's paint stories. Um, you know, in the sixties or seventies, John O'Keefe put a electrode in a, in a rat. And now with, in this paper that came out a few years ago from Cas group, where they found that they can use grid cell-like, um, activity in enter cortex to, or they found differences between people who are likely to develop Alzheimer's and those who won't.

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Which seems to offer some sort of initial glimpse at how you might develop some sort of neuroimaging based, um, uh, what's the term when you find out who will develop a disease? Prognostic or 

    Toby Wise: marker Prognostic. 

    Benjamin James Kuper-Smith: Prognostic biomarker. Some sort of, yeah. Um, [01:52:00] so, you know, you start with something completely random in, in rats, and then you end up with something that might help people figure out who's gonna develop Alzheimer's.

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: 20 years before they have symptoms. So. 

    Toby Wise: Yeah, I, I'm, I, I imagine we, we may not be quite the same, but we may have some similar effects that come out of the work that's happened during COVID where it, it, we, we discover things that we didn't expect to discover. We come across interesting findings about how people perceive risk in the real world, that then cause us to think differently about what we're doing in the lab.

    And, uh, who knows where that will lead us. Um, so yeah, I, I think yeah, you, I, you can never, you know, basic research is, is crucial and, um, you can never predict where it's gonna go, what it's going to be to, and I, I don't think it'll be any different with the work we're doing. I think it's, it's more when you get to the applied stuff, I think that, um, if anything it's important, almost diminishes.

    I dunno. 

    Benjamin James Kuper-Smith: I mean, it seems to me it has, it's, it's just a matter of, uh, timeframe, right? The immediate importance of [01:53:00] what we do is zero and the long term, 50 year down the road kind of thing. I think then even without, maybe not evens out, but yeah, slightly balances in our 

    Toby Wise: Yeah, yeah, yeah. Definitely. 

    Benjamin James Kuper-Smith: That's the hope.

    Toby Wise: Yeah. We, well, 

    Benjamin James Kuper-Smith: that's what we 

    Toby Wise: tell us. We'll never know, but yes. Yes. 

    Benjamin James Kuper-Smith: I just imagined, like after we retire and go, this was all so pointless. I wasted government money. I, 

    Toby Wise: I mean that's, I, that's almost what I'm expecting. But that's like, that the, I mean, that's how science works, right? You fail hell of a lot for you.

    You get anywhere, um, uh, you, I mean we, we know about the successes, but we don't, we always remember the failures, but the failures were just important 'cause they showed us what didn't work. Exactly. Um, and you also always get, you know, it's because we're human. You always get the kind of like hype cycle in, in research where new methods come about.

    We get massively excited about them. You know, there are [01:54:00] examples, you know, Riss one, uh, genetics to some extent, there's been another, um, and people think it's gonna save the world and we're gonna learn everything with a cure, psychiatric, neurological disorders, whatever. And it turns out that's not true.

    Um, and you might k kind of perceive that as, you know, we failed. We, you know, things didn't work out as we hoped we was a load of money, but that was kind of part of the process of discovering the truth. So, but 

    Benjamin James Kuper-Smith: then again, 

    Toby Wise: I think it's just how things work, 

    Benjamin James Kuper-Smith: isn't it? I don't know how true this is in general, but I feel like a lot of the examples I've heard of, of actual breakthroughs in science happened after some methodological development.

    So it, it was the case. I mean, well, using John O'Keefe, I think he was just one of the early people to use this electorate system where mice could run around. That was, you know, yeah. Um, so in some sense I think it's, I mean, there's always a hype cycle, but I think there's some truth to it. Um, yeah, 

    Toby Wise: I, yeah, I, no, I, I agree.

    Um, I, it it's often the case that. I think sometimes these, [01:55:00] like new methods that come about are perceived to have failed more than they actually have because, um, you're right. Like they, they will inevitably, inevitably lead to discoveries that could not have been made without them. They also probably lead to a load of rubbish.

    But it's like once you, I mean you kind of have to try everything right. See what works. And some of those things will be ex really important. Some of them might be Exactly. 

    Benjamin James Kuper-Smith: I was about to say the, the, the importance of the one discovery you might get from it might 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Be like, make everything else basically irrelevant.

    Toby Wise: Yeah. Yeah. I mean with, with FMRI, like a, a lot of stuff that's been done with it is either just false positives or not particularly informative or anything, but there are some really interesting things. I mean like on the, you know, in the sense of grid cells, you know, Tim Baron's work showing the abstract, um, relational concepts are seem to be encoded by some sort of grid cell-like system.

    Like that's something that was achieved with FM i that I think is really important. Um, [01:56:00] and you wouldn't have got that without fm I, even if you did get all the, the rubbish that, that came outta fm I as well. Um, 

    Benjamin James Kuper-Smith: so, so the hope is just that maybe just maybe will be one of the non rubbish things. 

    Toby Wise: Hopefully.

    Hopefully. You never know. You never know. But I mean, inevitably. I think most of what both of us produce will probably be towards the rubbish side. I think that's the same with most scientists. Even like the greatest scientists I imagine have done a lot Yeah. Stuff that they would perceive they would consider to be rubbish like 20, 30 years down the line.

    Um, you may believe it in at a time, but you would always learn things later on where you're like, oh, actually maybe what I did before in light of what I know now was not quite so good. And I mean, that's just how things work. It's just kind of, 

    Benjamin James Kuper-Smith: yeah. I mean, to the, the best example to me is then that, so I played the piano a lot, um, especially in my teenage years.

    There's a lot of music by very great composers that's really not that great. Yeah. And, um, where you [01:57:00] really wanna, like, this is the same person that did all this other great, like 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Sure. Um, so yeah. 

    Toby Wise: Yeah. It, it, it, and, and like no one out there is gonna be a genius a hundred percent of the time. Right.

    You're gonna, you're gonna get things wrong. You're gonna do things badly. It's much as we, we want to try and stop ourselves from doing that, but. You know, it, it happens. And partly a lot of it is just that you don't know what what you were doing is wrong at the time because I mean, like, so, and, and again, FM I is a good example of that, you know, a lot of early work and fm I was using statistical methods that were questionable at best.

    Um, but I don't think they, they knew that at the time. Uh, it's wouldn't do you 

    Benjamin James Kuper-Smith: do it right, if you knew it was 

    Toby Wise: Yeah, exactly. Yeah. And, and then, you know, people started to realize, actually maybe we should be a bit more careful about these things because of, you know, all these reasons. 

    Benjamin James Kuper-Smith: I mean, 

    Toby Wise: um, 

    Benjamin James Kuper-Smith: let's [01:58:00] be honest, most of the stuff we're doing right now is terrible in retrospect.

    Right? 

    Toby Wise: Yeah. 

    I 

    Benjamin James Kuper-Smith: that because there'll be completely different way of analyzing MRI and then 

    Toby Wise: Yeah, exactly. And I mean, it, it doesn't mean we haven't learned anything. Um, and it just means that, you know, not everything is gonna be perfect. Um, there'll be some interesting results that come out of it. There'll be enough that comes out of it to, to move the field forward.

    Um, but we can't expect everything we do to be absolutely perfect in, in hindsight. Um, 

    Benjamin James Kuper-Smith: not everything. 

    Toby Wise: Not everything. We, we can hope we can 

    Benjamin James Kuper-Smith: like 90%, but not everything. Yeah, 

    Toby Wise: yeah, yeah. Uh, but I think also, I mean, we we're like, in that sense, um, I. There are still things we could do better, um, that we can try and preempt those concerns more than we perhaps do.

    So one, one thing that I think, um, happened with fmri is that people didn't necessarily [01:59:00] consider what they were doing as much as they should have done. It's, you know, this really exciting method. You just kinda run with it and do everything you can without what you're thinking about whether what you're doing is right.

    Um, so like in the field of computational psychiatry, for example, um, I think we, we may end up having similar concerns because, um, maybe, hopefully not quite as bad, but, um, and I, I think I'm guilty of this as what, as anyone is. Um, but you know, we should probably be doing more kind of validation on, on the methods we're using.

    That means it's partly what I'm trying to do with the COVID stuff, but in terms of like the, the statistical methods we're using, so I had a, a conference papers that's been just accepted, um, with, uh, Vincent Volton and Ali Robinson from UCL, uh, where we've looked at it how, how, what's the best way to fit hierarchical baying models to data when you have multiple patient populations.

    Um, and the way in which you do that affects your false positive rate. And that's something that we hadn't really cons no one had really [02:00:00] considered before. So we are trying to, like, we are trying to get in there and, and address those things. Like now, but rather than, you know, it being 10, 15 years time, um, when everyone's done these studies and suddenly we realize that we've done them all wrong.

    Um, so I think, yeah, we need to be kind of proactive in, in that there will always be the case that we'll learn things and stuff we did before. Uh, we'll realize what's not right. But we, we do need to be kind of proactive in trying to do the best we can and get as, as well. Sorry, I just w waffle forever about random.

    Benjamin James Kuper-Smith: No, I think this, the last quarter of an hour has been a very good therapy session or both of us convincing ourselves it is fine that we fail a lot. 

    Toby Wise: Yeah, it is, it is. It's, I mean, I, I I guess how long, how long have you been doing your PhD for? 

    Benjamin James Kuper-Smith: Uh, almost two years now. Although, you know, I mean, it happened, I started January last year, uh, and it's now early November.

    And [02:01:00] then again, it does feel like I'm, you know, the last half year been spent more on the COVID research than on my actual PhD research. So in a, in a sense it feels like I've been doing my PhD for one and a half years at most, but 

    Toby Wise: yeah. Yeah, 

    Benjamin James Kuper-Smith: I've been here almost two years now. 

    Toby Wise: Okay. All right. You've not, you've not yet reached a stage where you've realized that everything you've done before is wrong.

    Benjamin James Kuper-Smith: Wait, what 

    Toby Wise: not. 

    Benjamin James Kuper-Smith: I'll just cut that pad out. Just pretend 

    Toby Wise: you said 

    Benjamin James Kuper-Smith: that. 

    Toby Wise: Um, 

    Benjamin James Kuper-Smith: no, 

    Toby Wise: I mean, but I mean, 

    Benjamin James Kuper-Smith: it's kind of interesting because we did like one study and then. The, the one I mentioned earlier where I still just have to finish the code. Um, and there, the interesting thing is this, I had like one idea and tested it in a, in a specific context.

    And the other question was how much does this generalize to other contexts? That kinda thing. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: And the amazing thing is this generalized really well. Oh, great. So I, I've always been in the, in the other way. I was like, we did the second like, pilot for something that would then come later. I was like, wow, this actually, [02:02:00] this looks better than the initial results.

    That's brilliant. This is amazing. Yeah. It seems like we looked at for some, yeah. For some reason we almost looked at one of the harder context to find the effect. Uh, I mean this is still based on 

    Toby Wise: brilliant 

    Benjamin James Kuper-Smith: preliminary pilot, but yeah, so I'm in the moment. I'm still in the o in the other opposite direction that I feel like I'm getting better.

    We'll see how long it lasts, but 

    Toby Wise: oh my God, I'm, I don't think it's ever happened to me, but yeah. 

    Benjamin James Kuper-Smith: I mean, 

    Toby Wise: but I think 

    Benjamin James Kuper-Smith: this is with a, with a pilot, right? It's not, it's a, I guess yeah, it's a somewhat decent pilot. It's not like a tiny thing. 

    Toby Wise: Um, I mean, I've certainly had pilots that didn't end up working out when I did the real thing.

    Benjamin James Kuper-Smith: Yeah, exactly. Exactly. So it could still, it could still all come for crumbling. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Um, 

    Toby Wise: yeah. I, I think you, I, I, I mean, I know I feel this way, I feel this way and other people do, where you always end up being very, feeling very critical of your, your prior work, even if it's not entirely justified. But because you'd, you'd always do it differently if you did it again, you'd al you'd always know there are things you could have done better.[02:03:00] 

    Um, 

    Benjamin James Kuper-Smith: quite a few. 

    Toby Wise: Uh, yeah. Uh, so even it's not a complete failure, like you would do it better a second time round. You'd probably do it better if you did it a third time round. Like we, it, it's not a bad thing that what we did before, it could have been improved. It just means that we've learnt something, the message we're using have improved.

    You know, that happens. And I'm sure you'll reach a stage where you look back on the study and think, God, what was I thinking? I could have, I should have done this completely differently. But 

    Benjamin James Kuper-Smith: yeah. Yeah, I mean, I, I do have those thoughts in general quite a lot. So guys, is this really good enough? Um, but then again, I think there is also quite a utility in just starting, I mean, as with almost the COVID research, um, 

    Toby Wise: yeah.

    Benjamin James Kuper-Smith: You know, as we said, like if you wait and theorize forever, you'll never do anything. So 

    Toby Wise: Yes. 

    Benjamin James Kuper-Smith: You just have to do something and then you realize the error and then you're correct. 

    Toby Wise: Yeah, I, I definitely tr tend to go through things in more of a trial error away than I think some people do, which might [02:04:00] not be good.

    Um. But I, I dunno, I'd like to just kind of get stuck in and try stuff and if it works, it 

    Benjamin James Kuper-Smith: seems have worked pretty right so far, right? For you? 

    Toby Wise: Yeah. I, I mean, like I, I've had multiple things that have not worked though as well, so, yeah. Uh, 'cause I mean, yeah, the number of like, things I've, like rough kind of things I've piloted and haven't worked.

    It's been, uh, 

    Benjamin James Kuper-Smith: yeah. 

    Toby Wise: Too much. But 

    Benjamin James Kuper-Smith: by the way, have you ever made a failure cv if you've heard Oh, 

    Toby Wise: I did. Yeah, I did actually do it a while ago. I've not updated it though. Um, I probably should do 'cause I 

    Benjamin James Kuper-Smith: failed. I found it so helpful for like, because then it was suddenly, so soon as something didn't fail, like, yeah, I'm building my cv.

    It actually felt like I was, I was achieving something. 

    Toby Wise: Yeah, no. 'cause I mean then like, sadly, like, it's a numbers game, right? Like everything we do basically is down to, it's for the result of a weighted friend and number generator. Um, 

    Benjamin James Kuper-Smith: yeah. 

    Toby Wise: So, uh, you just need to try as many things as possible and yeah, adding things to your failure CV means that you've been trying it more [02:05:00] things.

    Um, I mean, like, I've, I've done pretty much entire studies that, that just didn't work. Um, and like also I, I've done studies that like didn't work for annoying, for like stupid reasons as well. Yeah. Like, not that I didn't, I I checked the hypothesis well, and I didn't find the effect just that I realized I did the study really poorly and so I, I got inconclusive results and it's like, you can't be, you can't really publish it because it's just like, it wasn't done very well.

    Um. You know, uh, but yeah, 

    Benjamin James Kuper-Smith: it happens. Yeah. That's the most s I've had, I had that with one of my master's thesis where we, we ran it, it seemed every, basically all the control tricks, everything worked. It's just we didn't have the power to detect the effect. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Um, and, but it's not, it's not necessarily that we, so this was, it was basically, um, with using MEG decoding within person.

    Oh. So, um, it's not, it seems like [02:06:00] maybe adding huge amounts of trials per person would've helped, but probably not. It's, it seems like there might actually be just some sort of limit to, but not, but the thing is not even necessarily as going back to your point, not necessarily even in a particularly interesting way, it just seems to have not worked, but not because you made a huge error.

    It just doesn't really work. And that's just so frustrating. Yeah. 'cause you have this thing and you, it feels like maybe we should publish this. It's like, because it's kind of useful, but then it's so much effort to just show that something didn't quite work. 

    Toby Wise: Yeah, 

    Benjamin James Kuper-Smith: yeah. 

    Toby Wise: Yeah. I know. Yeah. I, I also had an MEG decoding study that Oh, really?

    So I, I had one that did work, which, which will be coming out soon. But that was basically a response to the first one, which like, uh, I kind of realized eventually, I, I was like, I did the study as I really wanted to kind of cling onto it and, and do something with it. But, uh, I, the more I did, the more I realized that like, if I tried to write it up, I was just trying to try to like explain away all the things that.

    You know, [02:07:00] it didn't work because the experimental design was not, not good enough. Like I should have designed it completely differently. It's like things were inconclusive because, I dunno, I, I was maybe like expecting to find stuff, um, in conditions where in hindsight I designed experiment in a way that it would never produce those effects, like it was.

    Yeah. Um, it's that sort of annoying thing. Uh, 

    Benjamin James Kuper-Smith: have you, uh, learned some sort of way or procedure to avoid that kind of stuff in your future? Current research? Uh, 

    Toby Wise: yes. Um, be more careful. Don't rush into things. I, I think, um, I think in sometimes I have definitely kind of rushed into running studies, um, without thinking things through as much as I should have done kind of theoretically.

    And, um, yeah, I mean I think that that was one example of that. And so what I've tried to do more [02:08:00]nowadays is do more kind of like simulation based work. Beforehand, make sure I have some, you know, kind of formal model of what I'm expecting that I can use to simulate data. And, you know, you can, then you get a better idea of what is what of whether it should work, um, before, before you try out.

    Uh, I think I would've had less failure if I had done that before. Obviously that, you know, you may be simulating data from an incorrect model and whatever it is not, uh, it's not bound to work. But, um, I think that would've been helpful. Um, but also it's, uh, it is often tempting I think just to, as much as some I've, I've found of just kind like trying things out and seeing whether they work.

    Sometimes you get a bit too excited about something and don't necessarily stop to, to evaluate whether it's working before you just gonna run with it. Um, 

    Benjamin James Kuper-Smith: yeah, the simulation thing is actually the one thing that I guess I'm at the stage where I've. Notes that I should be doing that, but I haven't actually done it yet for the next projects.

    But I definitely plan for the next projects to, [02:09:00] um, yeah, just simulate exactly what I, you know, these results I more or less expect and then what tests I would do and all this kinda stuff because 

    Toby Wise: yeah, 

    Benjamin James Kuper-Smith: it seems to me that it's so easy to think you know exactly what you're doing until you actually sit down to do it, and then 

    Toby Wise: Oh yeah.

    Benjamin James Kuper-Smith: Absolutely. I realize I had a very vague idea of what I was gonna do. 

    Toby Wise: Yeah. Um, and preregistration helps with that as well, right? Yeah. Like when you have to sit down and formalize your hypotheses. Um, I think sometimes when things haven't gone so well, I mean, I've, I've had, I've preregistered stuff that has not worked as well, but, um, but it does definitely help you think a bit more about whether what you're doing is sensible, um, as well.

    Benjamin James Kuper-Smith: Yeah. And like what exactly, what, what exact results you're expecting and how you're testing. 

    Toby Wise: Yeah, 

    Benjamin James Kuper-Smith: yeah. 

    Toby Wise: Yeah. And I mean, like, I have a study that I'm doing right now where, um, I was kind of, I'm, I'm testing this basic kind of behavioral effect that I think should exist. Um, but I realized that, um, I don't know whether [02:10:00] this, this kind of basically decision making paradigm, I dunno whether using this kind of planning mechanism that people that I'm hypothesizing people would use is actually gonna be helpful.

    Um, because you know, in certain tasks you may expect people to do, to make decisions in a particular way, but if it doesn't benefit them to do so, then why would they? So there's, um. I dunno, you know, the, the two step task that stood out by Nathaniel do? 

    Uh, 

    Benjamin James Kuper-Smith: yes, but I need a reminder. I've definitely heard it a lot.

    I'm, it's, 

    Toby Wise: it's, I can't explain it 'cause it's confusing, but essentially it allows you to look at this distinction between, uh, sort of goal directed or model-based behavior and habitual versus, you know, habitual is the thing 

    Benjamin James Kuper-Smith: where you have to choose between two option and then depending which one you choose.

    There's different possibilities for two things happening. 

    Toby Wise: Yes, exactly. Yeah, yeah, yeah. So the idea is that, you know, it allows you to, to determine how much someone's behavior is influenced by this model based planning system versus a model three planning system, the goal directive versus the habitual. Um, but in the original version of the professional task, it [02:11:00] doesn't pay to be more goal directed or more model based like, so in that sense, you know, if you don't, if you see that someone isn't doing that, then actually they're being perfectly sensible.

    They're using this kind of less computationally, uh, intense, expensive decision making method than they're doing just as well. Um, and then that kind of affects how you interpret results in that task. In like psychiatric populations for example. There's some nice work, um, by who, who kind of developed a task where they kind of mo incentivize, um.

    Goal directed behavior. Um, so 

    Benjamin James Kuper-Smith: is he the guy who does effort based stuff or something around that? 

    Toby Wise: I think so, yeah. I've 

    Benjamin James Kuper-Smith: never really read his 

    Toby Wise: stuff, 

    Benjamin James Kuper-Smith: but I've seen his name a bit. Yeah. 

    Toby Wise: Um, but yeah, he, he has this incentivized version of the two step task, which they then use in psychiatric patient or to look at relationships with psychiatric symptoms and things.

    Um, and then you, you see that there are effects of how much we incentivize this thing. Anyway, long story short, like, um, with that task, I feel like the original authors maybe should have [02:12:00] checked to whether it's actually beneficial for the, the subject to, to use the, the strategy they're expecting to see.

    Um, and I've had that same thing in my own task. I like designer's task and like, actually, wait, is, is it worth people doing this? Do, is it? Um, thankfully it turns out it is, but I, yeah, through using simulations I'd be able to kind of verify that actually, you know, helps people to do that. Um, 

    Benjamin James Kuper-Smith: yeah, 

    Toby Wise: because that's the sort of thing where, you know, if I then did that and did not find the expect, the expected effect, it may have just been that there was no reason for people to, to do what I wanted them to.

    Um, 

    Benjamin James Kuper-Smith: yeah. Yeah. And then you get this huge data set, oh, not huge, but a, a substantial data set on people doing exactly what you'd expect, but not what you want. Yeah, yeah, 

    Toby Wise: exactly. Yeah. Um, so yeah, I, I definitely think, yeah, simulation is, is a helpful way to do, do these things. Um. But then I, yeah, it's not always gonna work.

    Benjamin James Kuper-Smith: Yeah. But I think it's just about, you know, in increasing the probability of it [02:13:00] working, right? 

    Toby Wise: Yeah. It's not about, yeah. And it depends, it depends what kind of work you're doing as well. Like with, I couldn't we done with the COVID stuff, um, it, this is more helpful for kind of lab-based. Yeah. Um, yeah, stuff, we have a formal model.

    Benjamin James Kuper-Smith: That's one thing that I've really noticed is that there's just different types of doing research, right. The more exploratory kind of, um, you know, going out into the woods and seeing what you'll find. Um, yeah. And then the more theory driven, creative specific experiment and analysis plan to test that.

    And I've real, and I've realized I'm really not that good at the first, and it's not, I really don't like doing it. Um, and I don't, I don't think I'll do it a lot more in the future. Fair enough. Maybe on a small, maybe on a small scale, uh, like with small questions you have, but like, not with like large data sets.

    'cause that, that, oh 

    Toby Wise: yeah. 

    Benjamin James Kuper-Smith: That whole thing is not for me. Um, yeah. 

    Toby Wise: No, I, I agree. 

    Benjamin James Kuper-Smith: Um, so yeah, I think what you just outlined is probably more correct for the second type right? Where you 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Have a clear hypothesis and what you expect [02:14:00] and 

    Toby Wise: Yes. Yeah. Yeah. Definitely. Yep. 

    Benjamin James Kuper-Smith: Yeah. Okay. I think we've been talking for quite a while now.

    Toby Wise: Yes, we have. 

    Benjamin James Kuper-Smith: I'm not sure. Uh, 

    Toby Wise: is there anything more, any more specific stuff that you need to get covered? Let's 

    Benjamin James Kuper-Smith: see. I had a few. Well, in theory, I wanted to talk about how our studies fit together. I think that was initially how I contacted you. Uh, we talk about, I 

    Toby Wise: I think we did cover that a little bit.

    Benjamin James Kuper-Smith: Yeah. I think, um, very slightly. I think, um, yeah, 

    Toby Wise: we can 

    Benjamin James Kuper-Smith: actually, I mean, so this is now completely unstructured, but there's one question I had about how those toast fit together, and that's what I've found. And this is actually, have you are, have you looked at further data than those that weak from the longitudinal stuff or, 

    Toby Wise: yes.

    Benjamin James Kuper-Smith: Okay. Because one thing that's striking to me is the difference in terms of risk perception. And that's, I've got, so you have this increase between week one and week two, basic [02:15:00]or time 0.1, time 0.2, where especially for self, it jumps from like 40% to 50%. 

    Toby Wise: Yes. 

    Benjamin James Kuper-Smith: Um, and thereby because it stays, it, it increases also for other people, but less so, and because of that, you get this reduction in optimism bias, right?

    Yeah. Or comparative optimism. Yeah. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: So we don't have that at all. We have this completely different. Okay. So we, so I said we looked at, uh, we collected data on, I think the,

    so I think we might have collected our first data on the last day that you collected data for this. 

    Toby Wise: Oh, okay. 

    Benjamin James Kuper-Smith: So I think that's how they, but yeah, I think that's how they relate temporarily. But then we found there's a reduction in overall perception for everyone and their optimism that. Comparative optimism doesn't change at all across any of our Oh, interesting.

    Across any of our, we have three optimism questions, getting infected, infecting others, and getting severe symptoms that require hospitalization. 

    Toby Wise: [02:16:00] Yeah. 

    Benjamin James Kuper-Smith: And across none of those, we find any change in optimism bias at all. Interesting. Um, you could of course, argue that the relative shift changes, right. If you go from 40, 60 to 2040, 

    Toby Wise: yeah.

    Benjamin James Kuper-Smith: Then that is a difference. But if we just, right now we're just subtracting them, that's what we use as compared to optimism score. So just the subtraction doesn't change at all. Um, okay. It just reduces from time 0.1, two, three. Uh, no, not like perfectly linear. Um, but it seems Yeah, almost the exact opposite of what you found.

    Right. Like we find the reduction in 

    Toby Wise: interesting 

    Benjamin James Kuper-Smith: in, in these probabilities and then no change in the, um, optimism. So, 

    Toby Wise: so wait, so you found that people became, their perceived risks became lower? 

    Benjamin James Kuper-Smith: Yep. Quite 

    Toby Wise: a bit. Yeah. Uh, I, so. Yeah, that's, that's interesting. And was that across every country? 

    Benjamin James Kuper-Smith: Haven't looked into [02:17:00] that, I don't 

    Toby Wise: think, because it might be that where like Europe was hit a bit earlier than the US 

    Benjamin James Kuper-Smith: Yeah.

    So we did have, so I mean, most of our participants in this 430 people are from Europe. Um, the US citizens are fairly, fairly few. We had a annoyingly low retention rate for the Americans. Okay. We had a pretty good one for the, the Germans were very German and just completed the survey again, as asked. It was very stereotypical and the Americans just started and stopped and then we didn't invited.

    Yeah. Um, 

    Toby Wise: but now, no, to do all longitudinal research in Germany. 

    Benjamin James Kuper-Smith: I mean, we, we had a, uh, at least two third retention rate from each time point, and we didn't tell people about that. This was a long-term thing. We just asked them again and two thirds of it again. Whereas in the US we went from something like 250 people to 90 or something over the, yeah, over the three.

    Yeah. 

    Toby Wise: Ours, ours wasn't amazing either. 

    Benjamin James Kuper-Smith: Yeah. But with the Germans from 200 to [02:18:00] 130 or something was pretty good. 

    Toby Wise: Oh. 

    Benjamin James Kuper-Smith: Um, anyway, but uh, yeah, so I haven't really looked at the differences between countries there, but as I said, most of the stuff I've seen has been pretty stable cross country. So my guess would be that it didn't change.

    Toby Wise: In, in our sample. We did actually see then a decrease in perceived risk from like starting like the beginning of April to mid-April. So it was basically like kind of mid-March or sort of early March. It, it's kind of at lowest. Um, it kind of peaked towards it sort of, uh, late March. Um, and then by mid-April it had gone down quite substantially as well.

    So it may just be that you kind of got to that um, 

    Benjamin James Kuper-Smith: bit early. Yeah, 

    Toby Wise: yeah, possibly. 

    Benjamin James Kuper-Smith: Yeah. I mean, so the second in our case was 1st of April and we already had a decrease there from 16th to the 1st of March, the 1st of April. Um, but yeah, that might 

    Toby Wise: expand a little bit. Yeah, 

    Benjamin James Kuper-Smith: I guess [02:19:00] 

    Toby Wise: if they were kind of two weeks ahead maybe then that would match up with our pattern in the us.

    Um, also it was, yeah, it was kind of like it went up to a peak and then sort of plateaued for, for like two, two weeks or something. Was it two weeks or three weeks? I remember, um, before getting, going down again. Uh, so yeah, it, it, it definitely kind of changed over time. It sort of increased, stopped for a bit, then went down and it seems to, and then went back up again.

    Benjamin James Kuper-Smith: And so that's the like general, uh, probabilities, but the differences. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Did you 

    Toby Wise: Yeah. Uh, that's, sorry, that's not the differences. That's just the overall. Yeah. Um, 

    Benjamin James Kuper-Smith: but do you also have something where you, I mean, hours are just so stable. It's, um. The curves. So I, I have these curves for time 0.1, two, and three for each.

    So we have three optimism questions and they're just, they just, they look identical almost. Um, so 

    Toby Wise: ours, ours, look, they, they have, I'm just looking at the data now. They have a broadly similar pattern. Um, the only difference is [02:20:00] that, um, for example in as things go towards, 'cause we followed people up. I mean we, we followed up until yesterday actually.

    Um, 

    Benjamin James Kuper-Smith: okay. 

    Toby Wise: But, uh, I don't have the latest data, but like going to the summer for example, people's ratings, uh, of, uh, other people's likelihood of getting infection kinda went up, whereas their own state quite low. I think that's largely because many, so many of our, um, 

    Benjamin James Kuper-Smith: I guess it did go 

    Toby Wise: up respondents. Yeah.

    So cases in a lot of the US did go up quite highly in the summer, but some states they didn't. So, um, in, in states in particular, uh, on the East coast where we have quite a lot of people from, so that's just probably because people are kind of realistically thinking the average person in the United States has a higher likelihood than I do.

    Um, 

    Benjamin James Kuper-Smith: yeah, 

    Toby Wise: maybe not the average person, but like someone else in the United States. Um, so hard to tell how they actually, 

    Benjamin James Kuper-Smith: yeah. I mean we also found, so like one thing I tried to look at is really to what extent, [02:21:00] I mean basically where this risk perception comes from and there are a few. Very obvious candidate things, um, for the different questions.

    And it seems like pretty much people are aware of these things as far as we can tell. So it seems all pretty, I mean, whether the, whether the bias is in that sense, then rational, I'm not entirely sure, but it seems overall that the probabilities seem to, um, you know, I mean some of these things are very obvious, like the older you are, the more likely you are to get severe symptoms, that kind of thing.

    People, that's all in the data. Um, 

    Toby Wise: yeah. 

    Benjamin James Kuper-Smith: So it, it seems that it's, people are being pretty sensible about this in terms of what was known at the time. 

    Toby Wise: Yeah. The other thing is that we don't know, it's hard to tell kind of how representative the people you're sampling are in terms of their actual risk.

    Because it may just be that people who tend to do these things are genuinely at lower risk than the average person possibly. Um, 

    Benjamin James Kuper-Smith: yeah, we had, oh, we had this one huge, what was it? Talking about, [02:22:00] finding errors. There was this one thing where I thought the Germans were just so different. Um, and I think it turned out I would just coded a column incorrectly or something.

    Then I was like, oh, no, they're not. It's fine. I just messed this up. But there was this one where I was like, oh, this is really fascinating. Why are the Germans so different? Actually, no, it wasn't, it wasn't actually fascinating. It was this, this questions the quality of our data. It was more that it's like I, because of Germans, I was like, geez, are they really that I think it was, yeah.

    So we asked 'em how much contact they usually have. The Germans just reported very low, extra super low. And I thought, oh, did we, 'cause the German population on prolific isn't super huge. So I thought maybe they advertised in like some specific thing that they didn't realize was very specific to socially isolated people or something.

    But then it turns out I just included a very stupid extra column or something very, like something random in the way the data was collected. So then I was asking for [02:23:00] a binary variable, whether they're married or something, like completely. And then it turns out everyone said, or like a lot of people said no or something.

    Toby Wise: Okay. 

    Benjamin James Kuper-Smith: Not married, but like, just something like that. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Uh, yeah, I mean, that was good to know that, that, like talking about the earlier the, the, the length of the survey and people taking part in these surveys and how well they do it, it seems that like data's consistent over the stuff that should be consistent.

    Yeah. Um, pretty much it, it makes sense what people said. The the variables relative to each other. Yeah. Um, 

    Toby Wise: yeah, 

    Benjamin James Kuper-Smith: I mean at a group level, so they're always out biased, but 

    Toby Wise: Oh, yeah. 

    Benjamin James Kuper-Smith: Um, at a group level, it seems it's kind of good to, you know, there's some of those tests you don't wanna do. It's like, uh, yeah, I 

    Toby Wise: know.

    Yeah. 

    Benjamin James Kuper-Smith: How much is the age gonna differ or something between people, between examples, but they all came out pretty well. 

    Toby Wise: Yeah. Yeah. Same with those. Yeah. Yep. Yeah, it's reassuring. 

    Benjamin James Kuper-Smith: Um. [02:24:00] Yeah. And then the, the other other question I had, uh, was, so did you only look about optimism for getting infected or also for other stuff?

    Um, 

    Toby Wise: yeah, no, we only looked for the, the risk of getting infected. I think. I think we had a question about severity of infection perhaps as well. 

    Benjamin James Kuper-Smith: Mm-hmm. 

    Toby Wise: Um, I don't remember. 

    Benjamin James Kuper-Smith: Did you, 

    Toby Wise: it was a long time since I did this now, but, uh, it was, yeah, it's mainly around the likelihood of getting infected. Yeah. I, we probably should have done it for other things as well, but, um, 

    Benjamin James Kuper-Smith: yeah, I mean, because we in theory had three slash four questions on that.

    Um, one was the getting ect, the other was infecting someone else. Um, and then the third was severity of symptoms. And we kind of had that in two ways, getting severe symptoms or getting mild symptoms. But then during the pandemic, it turned out a lot of people were asymptomatic, so I was no longer sure that mild symptoms was a good thing.

    Yeah. So we kind of dropped that one because it seemed like Yeah, it's a kind of a mediocre to good [02:25:00]kind of thing. So anyway, so we can, with hospitalization is definitely bad. Um, and what's kind of interesting there is that there's just this quite a difference between those questions, um, in that there's actually no optimism for getting severe symptoms.

    Toby Wise: Okay. 

    Benjamin James Kuper-Smith: Um, 

    Toby Wise: interesting. 

    Benjamin James Kuper-Smith: And there's more optimism for like this, the, the Yeah. The strength of optimism is more for infecting others than forgetting infected yourself. And so what's kind of interesting that, and this, it is this three things. So infect others is more than get infected is more than severe symptoms.

    That's interesting. And it, what the kind of thing that is interesting that is like right now still a kind of guess in there is that there's a kind of perceived level of control there. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: So you can't really affect how much your, what your symptoms are gonna be because it largely depends on your immune system once you're infected.

    Yeah. And you can, uh, you can kind of affect how [02:26:00] much, um, you're gonna get infected because you can minimize stuff, but you can't, you have to go shopping and someone might just not wear a mask and coffee. You Right. There's the stuff you can't control. But infecting others is probably the thing you can control the most.

    Yeah. In the sense that you can wear a mask and stay at home, that kind of thing. And I know that's kind of something that we didn't expect at all. Um, something we haven't planned for. Yeah. That's really interesting kind of in there. But, um, but I guess you don't have any data really to compare that No, 

    Toby Wise: we don't.

    No. As another thing that we should have done better, I think. But I 

    Benjamin James Kuper-Smith: mean, this wasn't planned. 

    Toby Wise: Yeah. Well we, we should have accidentally done better. Yeah. We just focus on likelihood and kind of ignore the rest. But, uh. 

    Benjamin James Kuper-Smith: Yeah, I mean 

    Toby Wise: part of this 

    Benjamin James Kuper-Smith: actually came up when someone contacted me again talking about the cool thing about Preprint.

    Someone contacted me saying, Hey, did you look at this comparison? Um, I mean, I would've done it probably, but I just hadn't done it at the time because she said that they didn't find this in their sample. [02:27:00] I think, I think they, I think they were, I dunno whether they were looking for it and didn't find it, but at least they made that comparison and didn't find it at all.

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: And uh, we found it. Yeah. But we did. Um, 

    Toby Wise: interesting. 

    Benjamin James Kuper-Smith: I dunno. Yeah, it's interesting. But then again, it's also like, yeah. Not quite sure what to do with it. 

    Toby Wise: Yeah. I dunno. At this point you just kind of put it out there and see what people make of it. 

    Benjamin James Kuper-Smith: Yeah, 

    Toby Wise: I guess. Yeah, 

    Benjamin James Kuper-Smith: definitely. 

    Toby Wise: Um, yeah. Uh, I wish I could try and replicate it.

    It's, uh, sadly I can't, 

    Benjamin James Kuper-Smith: well, you can always click more data if you want to. 

    Toby Wise: Yeah. Um, 

    Benjamin James Kuper-Smith: uh, that's very, uh, oh wait, actually you were also contacted by the, the replication thing, right? 

    Toby Wise: Yeah. Were you as well? 

    Benjamin James Kuper-Smith: Yeah, not me. Oh. I mean, like, we were contact was customer was contacted for our project, uh, which makes interesting when I saw, I think you tweeted about it and then 

    Toby Wise: Yeah.

    Benjamin James Kuper-Smith: Um, and my first thought, well, it's looking less random now. 

    Toby Wise: Yeah. '

    Benjamin James Kuper-Smith: cause I said 

    Toby Wise: that. Okay. That's weird. Selected 

    Benjamin James Kuper-Smith: stuff randomly, [02:28:00] right? 

    Toby Wise: Yeah. Okay. I don't, that's interesting. 

    Benjamin James Kuper-Smith: I mean, our assumption was that it was about, um. Because there was this criticism towards research being done quickly during the pandemic that they Yeah, 

    Toby Wise: actually, yeah.

    Benjamin James Kuper-Smith: Um, 

    Toby Wise: yeah, I, I mean I, so they, they, uh, they got in touch with us once they'd sort of formalized their replication plan. I dunno what's then become a bit 

    Benjamin James Kuper-Smith: Yeah. We have anything once we gave 'em our questionnaire, basically. 

    Toby Wise: Okay. But yeah, they, they like gave us the protocol, we could comment on it and stuff.

    And, um, it seemed like they were doing a really good job of it and they were gonna add in some extra things that were quite interesting to test as well. So I'm quite looking forward to seeing the results. I mean, I, I highly doubt that our result replicate because it was in this fairly unique window of time, um, which you just can't.

    Benjamin James Kuper-Smith: Sorry. What is the, what are, what do they want to replicate of yours? Because, 

    Toby Wise: so 

    Benjamin James Kuper-Smith: yeah, 

    Toby Wise: for, for us, they want to replicate our finding that, um, uh, people's [02:29:00] engagement in, um, social distancing and hand washing were predicted mostly by their perceived personal risk, um, rather than, uh, their kind of risk of infecting others or anything.

    Yeah. 

    Benjamin James Kuper-Smith: Okay. 

    Toby Wise: So like, I, 

    you 

    Benjamin James Kuper-Smith: know, Charles was just laughing. Yours is a very specific one. And then ours was just optimism bias. 

    Toby Wise: Oh, really? 

    Benjamin James Kuper-Smith: Which is almost, it's, which is a pretty ob like it's, it's not controversial to find optimism by it is a pretty set thing. So when we heard that they wanted to replicate and that they want to replicate that finding, in particularly, we thought like, well.

    Yeah, they are. They, they will, yeah. 

    Toby Wise: Oh, yeah, that 

    Benjamin James Kuper-Smith: always, several samples that always came up. Yeah. We, we have like several replications in our own dataset. 

    Toby Wise: Yeah. Oh, 

    Benjamin James Kuper-Smith: um, 

    Toby Wise: okay. That's, that's a lot nicer, Allison. Yeah, no, I like, I, I highly doubt, I mean, I doubt, I, I have no confidence that it would replicate now, I mean, partly because, you know, like, sorry, 

    Benjamin James Kuper-Smith: sorry, say again.

    Toby Wise: The way 

    Benjamin James Kuper-Smith: they want to predict that the, [02:30:00] the, how much they wash their hands. 

    Toby Wise: So hand washing and social distancing. 

    Benjamin James Kuper-Smith: Okay. That's predicted by, um, 

    Toby Wise: by personal risk. Perceived personal risk. Yeah. Um, uh, 

    Benjamin James Kuper-Smith: we, I mean, we can look at that too, but, and we want to, but I don't think that's the part of the analysis I haven't done yet.

    The longitudinal 

    Toby Wise: thing. Okay. I mean, if you, 

    Benjamin James Kuper-Smith: if 

    Toby Wise: you can, it'd be great. It'd be cool to see. Um, I mean, 

    Benjamin James Kuper-Smith: that's basically what I'll be doing tomorrow, I think. 

    Toby Wise: Okay. 

    Benjamin James Kuper-Smith: Running that analysis. Yeah. 

    Toby Wise: Yeah. I, but yeah, I, I think, I mean, it'd be interesting to see whether it replicates, I don't know. It will because, um, you know, the, the ways in which we, we now know that, you know, hand washing isn't quite as maybe as important as it's important, but it's not like quite as critical as we felt.

    Maybe it was back in March, you know, um, the, the, the ways in which people are social socially distancing and that, like back in, back then, nowhere in the United States was it mandatory, for example. Whereas now it is, it's kinda outta control to some extent. Like there are so many factors that I think, [02:31:00] you know, that relationship right now will be, would be completely different.

    Um. Uh, but I mean, that's partly what what I think makes our results particularly interesting. Yeah. Yours, yours as well. It's that very unique window, window of time when people were just, you know, being hit with this thing. And there was relatively little government involvement, at least in the United States.

    Um, so it, our, our own kind of psychological responses had a greater impact on our behavior, I guess. 

    Benjamin James Kuper-Smith: Yeah. I think you also had the advantage, maybe that because you were earlier, you might not have had some of the healing effects we have because a lot of pe Well, it was kind of interesting. So we asked this one question, how much did you reduce your social, inter your physical contacts, whatever exactly the phrasing was, and pretty much everyone said they did it.

    I mean, it, it really was a thing that where a hun, like a hundred was the most, um, was selected by, I don't even know, like 90% of people or something. Oh. Or something like that. It was, it was very, it was a kind of thing [02:32:00] that was so skewed that basically everyone said they did it. But then we also asked like, how many, how, so we have this question like, how many physical contacts do you have with this?

    Like separated for different social contacts, uh, per week usually. Uh, and then during Corona, and then you did find that most people reduce quite a lot, but obviously not to zero. Yeah, yeah. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: So that's kind of the measure we're using now as, 

    Toby Wise: okay. 

    Benjamin James Kuper-Smith: Uh, social reduction. That's good. Um, 

    Toby Wise: yeah, 

    Benjamin James Kuper-Smith: of course it's still, the problem is still reported.

    Aft once the pandemic started. Yeah. Which is still problematic. And it's, I mean, as all of this, it's self-reported anyway. Um, yeah. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: But yeah. Okay. I'll, I'll have a look and see whether we can, I mean, that's one of the Yeah. Things. Yeah. 

    Toby Wise: But 

    Benjamin James Kuper-Smith: it is the weird thing about replications where it can seem like, it seems to me always you'll get this label of whether your study replicated or not.

    Toby Wise: Yeah, 

    Benjamin James Kuper-Smith: I know. And then so like, just based on that, you'll [02:33:00] probably come off looking way worse than we will. 

    Toby Wise: Yeah. Yeah. 

    Benjamin James Kuper-Smith: Which is fine by me. 

    Toby Wise: Yeah. I, I mean also like I, you know, this very, uh, fine line between, you know, what's a replication and what is a, I've forgot what the other words, but, you know, some kind of like extension basically.

    Yeah. And so I, I wouldn't consider this a replication because our, you, the claim we claim we were making was that in the first week of the pandemic, this is what happened. So without repeating the first week of the pandemic, um, you can't entirely replicate it. I still think it's interesting and worth doing, but I wouldn't class it as, as in the strictest sense of replication.

    Um, but who knows? Who knows? 

    Benjamin James Kuper-Smith: Yeah. We'll see. But I mean, I think it is cool that they're doing it right and they must have 

    Toby Wise: Oh yeah. 

    Benjamin James Kuper-Smith: Huge amounts of, I dunno where the resources come from if they want to replicate all sorts of studies. Yeah. Um, 

    Toby Wise: I dunno. But, um, 

    Benjamin James Kuper-Smith: yeah, it is. Yeah, I mean, I, I, I think when I [02:34:00] saw your tweet, I coincidentally a week or two before, um, found this thing by, I mean, this is complete different research about scarcity and how that affects cognitive performance and all sorts of things.

    And they, they were part of, so that was part of that replication effort in nature and science where they had like social science Oh yeah. In nature and science. That was probably from 2010 to 2016 or something. And yeah, so the scarcity paper was one of those papers and that didn't replicate. And the authors were kind of a bit annoyed with the replication because they said like, I mean, the replication people said, we're replicating the first study because that's the most important thing.

    And they said, well, for us, the first study was more like the thing to get the ball rolling and the actual mechanism follow later. So then what I thought was really cool, they then actually, so this is by I think the first author's, Shah, SHAH or something, um, anyways, about scarcity. And they then actually did their own replication effort of all the other stuff they did.

    And they also Oh, [02:35:00] wow. They found then, I mean, not everything, but like the study, right? And they found that, so they just replicated their own study and they found that the one that didn't replicate, they also couldn't replicate, but all the others they could. Um, so it seems like this one. And, but then again, it leads to this thing you see, like, you know, you read this.

    Nature, human behavior paper I think it was, that says, oh, these people didn't replicate. And it just looks like, yeah, 

    Toby Wise: yeah, yeah, yeah, yeah. 

    Benjamin James Kuper-Smith: It's difficult to do these things. 

    Toby Wise: I know. Yeah. And people, I've seen lots of disagreements where, you know, the original authors don't think that it's being replicated entirely properly and stuff.

    And I dunno, it's, it's good when they have this kind of, like, they allow the authors and the replications to go back and forth and Yeah. 

    Benjamin James Kuper-Smith: I mean, we didn't have that much back and forth though. I think for us it was more we, they said, Hey, we want to do this replication. And, um, I, I think basically asked for the methods and we just like wrote them all of our [02:36:00] questions in a Word document and sent them to them.

    And we basically haven't heard from them. It's, it's been, 

    Toby Wise: oh, so yeah, we, we got through like a full kind of protocol protocol document, like they, 

    Benjamin James Kuper-Smith: I think, 

    Toby Wise: oh yeah, 

    Benjamin James Kuper-Smith: I think, and, and there wasn't that much. 

    Toby Wise: Okay. 

    Benjamin James Kuper-Smith: Yeah. There was a few times when KOA was like, I haven't heard anything from 'em, I'm assuming. Oh, 

    Toby Wise: okay.

    Benjamin James Kuper-Smith: Yeah. 

    Toby Wise: May maybe they, 

    Benjamin James Kuper-Smith: but I mean, they've got a huge thing to do also. 

    Toby Wise: Yeah. 

    Benjamin James Kuper-Smith: Just organizationally. 

    Toby Wise: Yeah. Yep. Up. So yeah, we'll see what comes with it. 

    Benjamin James Kuper-Smith: I think I

    Toby Wise: some point.