Hanne Watkins is an adviser for the Behavioural Economics Team of the Australian Government. Previously, she was a PhD student and postdoc studying how humans think about morality in the context of war.
In this conversation, we talk about Hanne's move from academia to governmental work, and about her previous work on morality in a war context. We also talk about her experience of running a Registered Report, something I've read a fair bit about, but haven't yet gotten around to doing myself.
BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith. New conversations every other Friday. You can find the podcast on all podcasting platforms (e.g., Spotify, Apple/Google Podcasts, etc.).
Timestamps
0:00:57: How Hanne's work in behavioural insight for the Australian government relates to her previous academic work
0:35:34: Registered Reports: Hanne's practical experiences writing a Registered Report
0:56:16: Morality in war
Podcast links
Website: https://bjks.buzzsprout.com/
Twitter: https://twitter.com/BjksPodcast
Hanne's links
Old blog: https://myscholarlygoop.wordpress.com/
Google Scholar: https://scholar.google.de/citations?user=e5jaGiUAAAAJ
Twitter: https://twitter.com/hm_watkins
Ben's links
Website: www.bjks.page/
Google Scholar: https://scholar.google.co.uk/citations?user=-nWNfvcAAAAJ
References
Thaler, R. H., & Sunstein, C. R. (2009). Nudge: Improving decisions about health, wealth, and happiness. Penguin.
Watkins, H. M., & Brandt, M. (2019). The moral landscape of war: A registered report testing how the war context shapes morality's constraints on default representations of possibility. Journal of Experimental Social Psychology.
Watkins, H. M. (2020). The morality of war: A review and research agenda. Perspectives on Psychological Science.
-
[This is an automated transcript with many errors]
Benjamin James Kuper-Smith: [00:00:00] So I, it's kind of funny, I think how I, you know, you asked me in the, in your first email, like how I came across your stuff.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: And it was, it was kind of funny to me that, um, in a sense you are an unusual guest in this sense that I initially was looking for something else.
Hanne Watkins: Oh yeah.
Benjamin James Kuper-Smith: Um, like often it's a paper that I read for specific reason, because it's about my own res, like it relates to my own research and I want to know something from it.
And here I was looking for something and I thought, maybe this is relevant, that I thought, nah, not really, but I just continued reading because I thought I was suggesting. Oh,
Hanne Watkins: okay.
Benjamin James Kuper-Smith: That sounds good. I mean, yeah, exactly. Um, and in a, in a roundabout way, it is actually kind of relevant again. Um, it's just somehow I, I expected something slightly or I was hoping for something slightly different 'cause I was looking for something very specific.
Hanne Watkins: Mm-hmm.
Benjamin James Kuper-Smith: Um, but yeah, I just thought that was kind of funny.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: Um. Actually curious now is the, so you know, I, I [00:01:00] mentioned that the, the two papers on morality of war
Hanne Watkins: mm-hmm.
Benjamin James Kuper-Smith: Um, was what I was interested in. Is that actually something I think I wanted to talk about the government stuff later.
Hanne Watkins: Yeah,
Benjamin James Kuper-Smith: that's fine.
But just for now, is that actually something that is also relevant to your work now or is that kind of something you've done and left behind you? Uh,
Hanne Watkins: it's not at all relevant to my work now. Yeah. Ah, okay. I, uh, completely switched tracks when I got this government job. So, um, because the academic pipeline is quite, uh, long, I am still working on some papers related to war, but that's again, like just in my spare time, um, yeah.
Not related to my job
Benjamin James Kuper-Smith: because I was cur uh, I was curious then why. What exactly Australia's planning. Yeah,
Hanne Watkins: exactly
Benjamin James Kuper-Smith: How
Hanne Watkins: they hire.
Benjamin James Kuper-Smith: Did
Hanne Watkins: they end up
Benjamin James Kuper-Smith: in
Hanne Watkins: government when Yeah,
Benjamin James Kuper-Smith: exactly.
Hanne Watkins: When my research interests are, uh, morality and war. Um,
Benjamin James Kuper-Smith: yeah, it's like Australians trying to figure out what they can get away with.
Hanne Watkins: That's awful. No, no. I think, um, [00:02:00] the, the reason I was a good fit for the, my current role has to do more with the skills that you pick up during a social psych PhD. So data analysis, uh, just the general literature on social psychology more broadly. Um, and then also I guess, um, uh, I've been told that, that my application stood out because I mentioned open science and like pre-registration and, um, yeah, like using pre-prints, um, open access, those kind of things, which is something that, uh, beta does as well.
Benjamin James Kuper-Smith: Uh, so what does beta stand for again?
Hanne Watkins: Oh, yeah, sorry. Uh, it stands for the Behavioral Economics Team of the Australian government. So there's no g it, it's a slightly weird, yeah, it's acronyms. B
Benjamin James Kuper-Smith: ttag
Hanne Watkins: acronym. No, beta. Much better. Yeah, it does. Yeah. So we publish all our, we pre-register all our trials and we publish, uh, by default.
So often we can only publish a project like after it's well and truly wrapped up. But [00:03:00] at some point it makes its way onto our website. So all our projects that are completed, you can read about at behavioral economics. Dr. Go au.
Benjamin James Kuper-Smith: So I said I want to talk about this later, but I think now we're in it. Yeah,
Hanne Watkins: yeah,
Benjamin James Kuper-Smith: that's true.
We'll, about the, about the war stuff then later. Yeah. Uh, so yes, as I mentioned, like I have no idea what. Kind of work you would be doing. And one of my questions was whether you actually still plan on publishing things or not, and
Hanne Watkins: Mm.
Benjamin James Kuper-Smith: Um, so can you just kind of talk about how, maybe just how your work now relates maybe on a practical, like day-to-day level, like what you do, how that relates to kind of what I assume before was kind of standard academic research?
Hanne Watkins: Mm mm So, um, well, I guess, have you heard of like behavioral economics, behavioral insights, nudge units?
Benjamin James Kuper-Smith: Uh, yeah. So I mean, nudge units, so, let's see. So I've, I think actually once, [00:04:00] like ages ago, read the Nudge book.
Hanne Watkins: Nudge. Yeah.
Benjamin James Kuper-Smith: And, but I'm not sure whether I read it. So, uh, but I think that's about, um, how should we say, depending on how you frame questions and make options available to people that behave differently and there, so I think the idea is to.
Basically make people adopt things that are good for them without forcing them to
Hanne Watkins: Yes. Yeah. It's kind of marrying, yeah, the marrying. Just the idea that like it's good for people to be able to choose. Like, uh, whatever they want, whatever option they want. The classic example in Nudge is of a canteen wanting to have like sweets available and chips, but also apples and bananas or whatever.
'cause they want to give people the choice, but at the same time they have, the canteen has to decide how to like present those choices and they can make those decisions in a way that kind of helps people choose, um, the apples and the, and the [00:05:00] bananas. In this example, uh, on the assumption that most people, if they were kind of reflecting on what they would want to want for themselves, it would be to choose the healthy option.
Um, so. Behavioral, like nudge units or behavioral insights units have kind of come into government to try and help, um, policy makers think about the way that they present choices to the public. It's like there's no neutral way to kind of give people options. You have to pick one. And so you should, um, try and pick ones that will do people like, you know, the most people, the most amount of good, but also, um, you should evaluate and find out, well, if we do it this way, what effect does that have?
And if we do it that way, what effect does that have rather than relying on the assumptions from sort of classical economic models. So that's at the broad level what we do.
Benjamin James Kuper-Smith: I mean like one example that came to mind for me with nudging is that of organ donations.
Hanne Watkins: Oh yeah.
Benjamin James Kuper-Smith: Mm-hmm. Where I think the example they [00:06:00] used was something like, I dunno whether it was Germany and Austria or, but like two countries that are culturally very similar, have completely different, uh, organ donation rates.
Mm-hmm. Because one of them has an opt-in and the other has an opt out. Um, and that Yeah, because people are lazy basically. Yeah. Yeah. In one, in one case, like in the opt-in where you have to make an effort to be an organ donor, something like 20% of people or whatever are organ donors. And in the other case where you have to opt out, if you don't wanna be on something, like 70% of people are organ donors.
Something like that. Yeah,
Hanne Watkins: exactly. So those are, those are the kind of things. Um, as another example that's pretty well publicized, one of beta's early sort of high profile projects had to do with, um. Anti, uh, microbial resistance. So doctors overprescribing antibiotics and, and building resistance. Um. Uh, to those antibiotics.
And so the project was to try and, um, the government was going to write a [00:07:00] letter to doctors in Australia to say like, oh, by the way, this might not be a good thing you should think about. You should think about like maybe, uh, reducing the, um, amount of antibiotics you prescribe. And so to tie this back to your question earlier about how this like relates to the academic work, I wasn't at, uh, in beta when this project started, but there were other social psychologists there.
And so they were thinking about the things we know about how people's, uh, behavior is influenced what by what others are doing. Um, so what they suggested was that, you know, there was one version of the letter that was just a standard information letter, and then there was another one that said that also gave the doctors information about how much antibiotics they prescribed relative to other.
Doctors. Uh, so, and they only gave that information to doctors who, um, were in the highest, like beyond the 70th [00:08:00] percentile, um, of prescribers. So really high prescribers would told like, you prescribe this much antibiotics, whereas the average is like this much. And the difference was sometimes quite large.
Um,
Benjamin James Kuper-Smith: it's like, this is the problem and you are the problem.
Hanne Watkins: Yeah, yeah. Um, in, in ni nice, uh, just sort of descriptively, right? Like not, not making that judgment. Yeah, yeah, of course. But leaving it to the doctor to, to make that judgment.
Benjamin James Kuper-Smith: I mean, there are many reasons why you might prescribe more than others.
Hanne Watkins: Yeah, yeah. And, and in some, in some, um, this study was recently replicated in New Zealand and they actually looked more, uh, carefully at the different segments of the population that are generally prescribed more antibiotics or less antibiotics. And so yeah, found like, oh yeah, there are some groups. For whom this matters more than others and so on.
Anyway. Yeah. So there were also a few other versions of the letter, but the short story is that, um, the letters went out before the cold and flu, uh, season, which is when you see this big peak, uh, usually in antibiotic [00:09:00] prescription. And that peak was just like chopped off, uh, by, I already actually worked for, for the group that, that, um, yeah, it worked really well.
Um, and even we recently published an update to this report, uh, like at 12 months later. The report didn't get that much attention 'cause COVID, but, but it does seem to have persisted a little bit. Um, the, the effect of getting a,
Benjamin James Kuper-Smith: you might have to remind them.
Hanne Watkins: Yeah. Might have to remind them. Yep. But, um. So that was, that was a, a trial that was, you know, registered ahead of time, what the plan was for the, for the data analysis, um, what the main outcome measure was going to be, which in this case was the number of scripts per thousand visits, um, how it was gonna be analyzed.
And then, yeah, the report is, um, available on, uh, on the website.
Benjamin James Kuper-Smith: Is, this is one thing I wanted to answer. Do you, uh, to answer, to ask. So the, the, the publications [00:10:00] you do, it's, it's, it doesn't have to be like a paper or something. It's, it's, um, you know, you don't have to, do you have to go through the whole peer review process or how does that whole thing work?
Hanne Watkins: No. So I've discovered something much more rigorous than the peer review process. It's called the clearance process. Um,
Benjamin James Kuper-Smith: okay.
Hanne Watkins: Which, which just means like, um. I guess, you know, academia, you think of it as fairly hierarchical in some ways. Like there's a very clear, like undergrad, grad students, postdocs, uh, associate professors, you know, like there's, there's, there's clearly levels, but in terms of just the interactions that you have, um, and who makes what kind of decisions, it's a lot flatter.
But in government, one big difference is just that the, the hierarchy. Um. Really matter for, like we write a report, then it goes to the next person in the hierarchy and they have to read it, uh, and say like, oh yeah, this is okay, but take this bit out. And then you do that and then it goes to the next person and they say, okay, this is fine, but I don't like your phrasing here.
I change that. And so, and that's, and [00:11:00] that, I guess I should also point out that a good thing is that that process doesn't happen only at the end, like after you've done all the work, it also happens right at the start. So you say like, oh, maybe, maybe we wanna do this. And they think like, oh, okay. Yeah. Is that a good idea?
Is it gonna benefit the Australian public? Is it value for money? Um, does it have any like, risks associated with it? Okay. Like if we've balanced up all of those things yet. Okay. Go ahead. And then as you're making decisions along the way about the sample of participants or the analysis strategy or the outcome measures, we often have to, um, partner with different agencies.
So that's one thing that I can also, uh, talk about at length, how beta works as like, just fair warning. Um, how Beta is in a, um, central agency. So we're in the department of the Prime Minister and Cabinet, which doesn't, which is a, a central agency, which means it doesn't have direct responsibility over a specific area, like for example, [00:12:00] health or transport or education.
It sits at the center. And so if we wanna do something with health. We need to talk to the Department of Health. If we wanna do something with taxation, we have to talk to the Australian Taxation Office. If we wanna do something with, um, education, talk to Department of Education. And um, most of the time it doesn't actually happen in that direction.
It's the depart the other departments coming to us and saying, oh, we have this behavioral problem. Um, do you think you can help? So, yeah, sorry, I lost track of what your question was, but,
Benjamin James Kuper-Smith: uh, yeah, me too. I never, uh
Hanne Watkins: oh, I was gonna say, um, the publication was what you asked about. Um, so I asked when I started like, oh, will I be able to publish in academic journals while, uh, while working at beta?
And the people I spoke to said like, oh yeah, in theory you can, um, you know, again, you have to like get it cleared by a bunch of people, but, um, it's not, it's not a problem sort of at a [00:13:00] theoretical level. But what I've discovered is that once we have the report, um, on our website that is like the beta report in, it's much more readable than an academic article.
Um, it's immediately, uh, public, um, and. I just haven't really felt like any great need to then restructure everything like, so that another, like five people can, can read it, you know, when it's published in Journal of blah. Um, so it doesn't, it Yeah. Even though I thought that I would want to continue to do that, um, it just doesn't seem like necessary, I guess.
Benjamin James Kuper-Smith: Hmm. Okay. Is, um, um, is, I mean, is this. Move from having a, a postdoc position I think you had before, right? Mm-hmm. Um, from that to working, um, in your current position, is that for you, a kind of permanent move or, because I, I, my question right now is [00:14:00] like, okay, so if you wanted to return, let's say in at some time, would that be a problem that you hadn't published anything, you know?
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: And quotation marks, you hadn't published anything.
Hanne Watkins: Um, so I asked the same question from when I was considering whether to take this, this position. I asked my old, um, supervisor that question like, oh, but what if I don't like it? Like, what's gonna happen for my prospects of getting back into academia?
Um, at the time my prospects for staying in academia weren't that great, so just because there were no jobs. Um, so, so like, I don't know if, if it doesn't really make sense to then start thinking about going back to academia, but yeah, I raised the same question and what he pointed out was that, um, your, uh.
Output, for want of a better word, um, is evaluated relative to opportunity when you go for lectureship positions. Um, at least in, at least in Australia. Is that true?
Benjamin James Kuper-Smith: Yeah, that was
Hanne Watkins: my question about that. I mean, yeah, [00:15:00] maybe it's like a theoretically true thing, but, but so he, he followed up to say that, um, he thought that I could make an argument that it's not like.
Say, I stayed in this job for like three years and then decided I wanted to try and get back into academia. I could make the argument that I hadn't just been sitting there for three years. You know, I'd been publishing non-academic articles. I could point to all the other things that that would stand in place of publications and my old publications wouldn't expire.
So, you know, you would say like, oh, I have these pub, these solid publications. Then I went and got a government job and had like a direct impact on the wellbeing of the Australian public. And now I would like to, uh, uh, like be a lecturer again and just talk to undergrads. Um. Yeah,
Benjamin James Kuper-Smith: that's a great way of phrasing it.
I didn't publish, but actually help the world. Yeah,
Hanne Watkins: yeah.
Benjamin James Kuper-Smith: But are these reports also then linked to your name and that kind of thing? Or is it just the, the [00:16:00] beta team?
Hanne Watkins: Uh, they have my name, uh, uh, so I haven't been at beta long enough to see a report actually published. Yeah. Um, again, clearance processes and so on.
Yeah. But, but yeah, it says in, in the front matter, it says like the beta, um, team members who contributed to this report were, and then it just has our list of names. Yeah.
Benjamin James Kuper-Smith: Okay. So there there is still this clear like, I worked on this thing and
Hanne Watkins: Yeah. Yeah.
Benjamin James Kuper-Smith: You know, my, my name's on it. It's not just
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: You know, because otherwise it's like this team published something.
Hanne Watkins: Right, exactly.
Benjamin James Kuper-Smith: And it's very unclear what you. Well, I guess that's what references are for, but yeah.
Hanne Watkins: Yeah. Yeah. And I think also, um, you know, you, yeah, it, like you said, references and, um, beta has, uh, some psychologists, some economists, some sort of public policy people. Um, but the thing, one of the key things that the psychology, um, uh, PhDs and masters, uh, people contribute is actually the ability to [00:17:00] design experiments and analyze the data from them.
So for the projects that I've been a part of so far, um, I've been the main data analysis person for all of them. I say that and, and then there's been a more senior data analysis person checking all my work as is good and proper. Um, but, but yeah, so, so that's kind of like, that doesn't come out in the reports, but it's something that, that, like my reference could say, I could say in a cover letter or whatever, and then it would be very clear when you go to the results section, like, all right, so this is exactly what this person, um, contributed to the report.
Benjamin James Kuper-Smith: Yeah. Yeah. Um, I had one, um, when you said earlier that you have to, um, get clearance from, from above. Um, the first thing I, I mean, I'm sure this, I hope that, well, I think this was just, uh, uh, the way [00:18:00] you phrased it, but there was one thing that immediately made me think of C Sense.
Hanne Watkins: Oh. Oh, yeah. Mm-hmm.
Benjamin James Kuper-Smith: How do you say that?
Sensor cen uh,
Hanne Watkins: c
Benjamin James Kuper-Smith: Yeah, yeah, because mm-hmm. Yeah. Um, because you said, you know, like, we take this part out or something. Oh, yeah. Yeah. And I'm just, I I was just curious, like, do they then sometimes tell you, like, don't write it's, I mean, I'm sure you don't, didn't mean like results or something, so like No, this is not, yeah,
Hanne Watkins: it's more, um.
So my experience is a little bit, uh, limited here. I've been at beta for 10 months, so I haven't quite figured out like exactly what kinds of things are considered sort of sensitive, um, is the phrase that often gets used. But, um, often it'll have to do so in the experience I do have, it seems to have a little bit to do with kind of consistency across different, um, [00:19:00] uh, agencies and different publications on the same topic.
Um, so for example, um, I don't know if this is the case, so, um, because I wasn't on the project about antimicrobial resistance, but say, I'm imagining that like the Department of Health would've had some sort of campaign more generally about like antimicrobial resistance and how to use antibiotics and so on.
And so then making sure that beta's publication like doesn't say anything that seems like it's gonna contradict. Um, what health are saying in, in, in terms of like, just, yeah. It's not censorship so much as just making sure messages are clear and people are sort of, um, yeah. It's, it's a bit hard to, hard to get at.
Uh, I wish I had a,
Benjamin James Kuper-Smith: like, avoiding certain things that might just seem confusing even though they aren't.
Hanne Watkins: Yeah, yeah, yeah. 'cause often it's not, it's often it's not like, take this out. It's like, oh, this is really unclear. Explain what you mean. Or take it out [00:20:00] like
Benjamin James Kuper-Smith: I
Hanne Watkins: see.
Benjamin James Kuper-Smith: Okay.
Hanne Watkins: Yeah. Um,
Benjamin James Kuper-Smith: okay,
Hanne Watkins: so yeah, but lemme think if there's a.
I'll, I'll, if something comes up, I'll, I'll come back to it. Yeah, yeah.
Benjamin James Kuper-Smith: But also, and this then relates to me, to the whole thing. You mentioned earlier that one of your, um, one of the good things about you and your application was that you had experience with open science and preregistration, these things.
Mm-hmm. So I'm assuming you have to, you know, those things then have to be in the report that you pre-register or how
Hanne Watkins: Yes. Yeah. So we pre-register, um, uh, so, so we, we publish the pre-registration plans as well. Mm-hmm. So everything that we say we're gonna do is, is right there. You can go and look at what we said we were gonna do, and then you can look at the report and see like, well, did you actually do that?
Um, is that
Benjamin James Kuper-Smith: done via like the standard OSF format or the
Hanne Watkins: No, just go. Just be website? Yeah. Just on our website. Yeah. Um, and, uh. And then in [00:21:00] the, in the report we will also say, have like very clearly, like, this is the, um, as pre-registered or, you know, for our primary outcome measure, we found blah, blah, blah.
And then it'll have a section that says additional analysis or secondary analysis or, and then we, um, just make it really clear that, that, I guess 'cause one, one difference, uh, between, um. Uh, between academic reports and the reports that we write is that it's often quite clear that someone is going to read the report and try to do something on the basis of our conclusions.
And, and like it is actually going to inform some decision, even if it's not directly doing something. It's, it's like a few steps down the line and someone is going to make some change. Right. So that's another
Benjamin James Kuper-Smith: thing thing I how that's a weird concept almost.
Hanne Watkins: Yeah. Yeah. I mean, it's a great, it's all
Benjamin James Kuper-Smith: like, oh, that's, that's so different.
Hanne Watkins: Right. But, but it's when you, when I feel like it's a really good thing to have to be like, well, [00:22:00] we can't just be doing these studies like, 'cause we kind of wanna know. Like we are spending taxpayers money. It has to be something that we can justify in terms of like, why do we wanna know this? What impact is it gonna have?
Um, why is it important to people?
Benjamin James Kuper-Smith: And I'm also spending taxpayers money, but I'm, I'm thinking about those questions.
Hanne Watkins: Yeah. Yeah. I mean, when I was, when I was a, a PhD student, I was also funded by the taxpayer, but the pay was a lot less. So I guess, uh, I was, it was more okay that I just did something I was really interested in and yeah, not very practical at all anyway.
Um, but I was just gonna say that, um, it means that, so the, the reports I've worked on, the biggest, um. The most kind of editing that's been happening has been in the results sections where we just wanted to be really clear about what we feel like confident in. Um, again, like and that's because the effect sizes are big or because the analysis were sound, the analysis were pre-registered.
Like there wasn't anything weird. You know, [00:23:00] it's the, those are the things we're confident in. And then things where we're like, well, it looks like there's a difference here and here, but we're not really sure. Like, just being really clear about the conclusions that we think can be drawn from those sort of in-between things, um, I think is important.
'cause as soon as you like, put it in a report and say, oh yeah, this is what, what we found, then someone is gonna run with it. Um, so just, yeah.
Benjamin James Kuper-Smith: Right. Yeah. Yeah. I guess there's more of a. How should we say responsibility to make sure that you are also just, I mean, I think in general, academic research is also going much more that way to, you know, report effect sizes and all that kinda stuff.
Um, but somehow it's not quite as important as if, you know Yeah, that's, someone's gonna just read your thing with the implicit, uh, not implicit with the explicit purpose of then using it to actually inform policy or something. Then I feel like there's also probably you don't wanna like bog them down in like a hundred details [00:24:00]that don't really matter.
That don't
Hanne Watkins: matter. Yeah, exactly. Yeah. But
Benjamin James Kuper-Smith: that are interesting for like basic research, but not for actually I know writing that letter to doctors or,
Hanne Watkins: yeah. Yeah. I mean, I should also say that it does happen that our reports just kind of get ignored. Um, whether that's, whether that's because, um, the, uh, like we just didn't find anything like, you know, we're comparing two different.
Presentation formats or something, and it's like, oh, well this doesn't really matter. They're both equivalently good. Like then obviously there's not that much interest in trying to build on it, but also if the priorities, um, move on or the budget's cut or, you know, hypothetically, um,
Benjamin James Kuper-Smith: yeah.
Hanne Watkins: Yeah. Those kind of things mean that, that, uh, even if a part, a particular study seemed to show something promising, it's not always carried forward.
Benjamin James Kuper-Smith: Mm-hmm. Yeah. By the way, I think we heard your dog there for a second in the background a few seconds ago. I said, I said something always happens in these vodka.
Hanne Watkins: Yeah,
Benjamin James Kuper-Smith: I think I have to [00:25:00] leave that in though, because you were talking at the same time, so I can't take that out. Yes,
Hanne Watkins: you can't. Yeah. Hopefully it wasn't too, uh, uh, too loud.
Benjamin James Kuper-Smith: No, I, I dunno. I thought it was nice to have a dog in the background.
Hanne Watkins: Yeah, yeah.
Benjamin James Kuper-Smith: Um, anyway, the, uh, another thing I was wondering is, so are you then. Do you only do research about certain topics or is it more that, uh, your kind of, your team or the, the meta team is like a general research purpose team where just people say like, Hey, we want to know something about this, and then you just go off and do that?
Or is it, yeah. I'm just curious like how specific or limited the research you do is, or whether that's more Yeah. Whatever has to be done.
Hanne Watkins: Yeah. Um, in theory we could do, we could work on anything across government, but yeah, in practice it's sort of a balancing act, um, with people coming to us saying like, oh, we have this problem.
[00:26:00] Can you help us solve it? Or, oh, we have this question about that. Can you help us answer it? And then, um. This is something that my managers do a lot more than, uh, I do. 'cause they're kind of in charge of like managing beta's workload and projects as a whole. They have to figure out like, is this a pro problem that we actually, that is like a behavioral issue, so therefore something that falls within our, um, scope.
Is it? Um, if it is, then, then, okay, good. But do we have capacity to deal with this right now? Um, is it on a topic that we sort of already have some expertise in? Um, I'm just trying to think of like the other considerations. Uh oh yeah. What are the timelines like? Like sometimes an agency might come to us with a project and say like, oh, we really want, um, you to help us design this website.
And then we're like, okay, that sounds interesting. And then they say like, oh yeah, but we have to finish it next month. It's like, well then there's a limit to how, [00:27:00] how much we can actually contribute. Um, you know, that, that was just a random example, but. Yeah. So there, it's kind of a balancing act, but we've worked on, um, uh, trying to, um, we've worked on retirement savings, we've worked on projects in health, like I mentioned.
Uh, we worked on projects on add-on insurance. We've worked on a project on helping students, um, like not drop out of university. We've worked on cybersecurity, uh, reminders for credit card, uh, like to make your credit, credit card payments on time. Yeah. And, and a lot of them are, are, you know, randomized control trials in the field.
Um, but sometimes we'll also write advisory reports where we just go in and kind of analyze a problem and suggest recommendations for addressing it. But we don't actually test those recommendations. And those reports are also available on the website.
Benjamin James Kuper-Smith: Okay. [00:28:00] And. Um, so what kind of, so let's say you do in empirical study to see how people react or whatever.
Do, is it, is it like, can it, can it be anything that you might do in an, in academic research context or is it, uh, somehow imagining you have more ability to contact citizens or something to do Oh, yeah. Like to get like specific representative samples or something like that, but
Hanne Watkins: yes, no, definitely. Um, so our like gold standard is running a trial in the field so that that trial with the doctors, the letters went to the doctors, uh, like the, the actual reel.
Think it was a sample of like 3000 or something. Um,
Benjamin James Kuper-Smith: you didn't let undergraduates imagine they were doctors.
Hanne Watkins: Imagine that they were doctors. No. Um, but um. And I'm just trying to think of, oh, yeah. And the project with students, again, that was a randomized control trial in the field where we got, uh, partnered with two universities, um, and recruited students at those [00:29:00] universities.
But we have also done some more like framed field, uh, experiments so that when, where you try to get the, the. A sample of people who would be sort of engaging in the kind of decision that you are, uh, interested in. So an example is a project that, uh, about retirement planning. So there they got, um, people who I think were recently retired and so they spend a bit of time like, um, trying to, like, making sure that they could recruit those people.
And then they took them through a process that was hypothetical. Like they didn't, in that case, they weren't actually choosing their, um, their a new retirement plan. It was just kind of taking them through, I think it was a website, um, informing them about different, uh, retirement plans. And you could make like a, okay, so now that you've been through this, you know, was it helpful?
Do you think you would switch retirement plan, savings plan? What? [00:30:00] Whatever it might be. So yeah, we do a little bit of that, um, kind of more hypothetical work as well.
Benjamin James Kuper-Smith: Hmm. It sounds like these. Especially when I consider, when I compare it to the study that you, uh, that we're going to maybe talk about in a, in a few minutes.
Mm-hmm. Um, which is an online study, right? If I remember correctly. Mm-hmm. Yep, yep. Is that right? Yeah. It sounds like these are just involve a huge amount of effort in terms of like getting the data, like because it's in the field and that kinda stuff, like contacting all the doctors and
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: Yeah. I guess you have to measure their.
Antibiotic output, if that's the word. The
Hanne Watkins: prescription. Yeah. So, so, so yeah, A lot of the challenge, um, comes from not just like finding, figuring out the sample and how to contact them, but also like where do we have access to that data? So, uh, the data on, on doctor's prescriptions, rate prescription rates is collected in Australia, but of course it's [00:31:00] like, oh, like, you know,
Benjamin James Kuper-Smith: automatically, or
Hanne Watkins: Yeah.
Yeah. Okay. Um, I can't remember like what the organization is that collected, but it's collected. Um, and, but of course there's like huge, um, privacy and kind of confidentiality issues around that. So like, it's a pretty long process of negotiating to, to, um, convince the agency that, you know, we're going to be treating that data, um, as it should be treated securely.
Yes. Um, yeah. And that it's not gonna compromise individual doctors, um, or put them, you know, at any disa particular disadvantage, things like that. All our projects still go through an ethics review. Board. Um, and they're also reviewed by the like partic, um, teams that just deal with privacy in our department.
Um, so there's a lot of things like that that, that, um, you go. So yeah, on the one hand, like because it's the government, it has access to, uh, amazing stores of data. On the other hand, getting [00:32:00] access to those amazing stores of data is appropriately difficult. Um, yeah, yeah,
Benjamin James Kuper-Smith: yeah. But I'm also assuming you are, you are not doing the actual data collection and sending email, uh, letters, things you are, you said more the analysis parts.
Yeah. Good
Hanne Watkins: question. Um, yeah, so for that one. No. So one project that I'm on at the moment, I,
Benjamin James Kuper-Smith: I guess that wasn't your, yeah, you said before.
Hanne Watkins: Yeah, that one was mine. But so, so just as an example, a project that I'm working on at the moment, and I won't tell you the details, but there, the agency is doing all the data collection.
We've specified what kind of, um, like what variables we want to collect. And then once we reach the target sample size, which is in this case is taking a few months, um, they will do the initial data cleaning, make sure that no individuals can be identified, and then send it to us. Um, and we'll do the, the data analysis.
Benjamin James Kuper-Smith: Hmm. Yeah.
Hanne Watkins: So
Benjamin James Kuper-Smith: it's, from the way from describing it, it sounds like it's. [00:33:00] Kind of like academia, but without a lot of the annoying parts.
Hanne Watkins: But, but, but with a different,
Benjamin James Kuper-Smith: but probably with other different
Hanne Watkins: parts. Yeah. Yeah, yeah.
Benjamin James Kuper-Smith: I guess peer review can be annoying, but clearance doesn't sound like it's the easiest process either.
Hanne Watkins: No. Yeah. I mean, I really appreciate it. So maybe this will be a nice pivot to, to the registered report that I wrote in academia. Right. I really appreciated having, um. In that case, the peer review before we'd actually run the study, because of course, the, the, the, you can take on board like feedback in a completely different way when you haven't just like, you know, like, oh, you should do this thing differently in your experimental manipulation.
It's like, oh, well thanks I already had 3000 people complete this study. Like Yeah, yeah. You know, it's, it's much more useful earlier on. And, and one of the really great things about, um, working the way, the way that beta works, and this is probably true for some academic labs as well, is just that it's so intensely [00:34:00]collaborative and you are always having to, because we work in partnership with other agencies, you, you have to talk to people who are not like so used to thinking about experiments and research.
And so you have to justify like every decision and rethink it and explain it. And like, you know, there'll be some sort of practical reason why you can't do it in the like, theoretically best. Way or whatever. And so you work out solutions to that and you get everyone on board with the outcome measures and talk about like, oh, okay, well if we're measuring like, money spent on X, like then we, we are treating that as a good thing or a bad thing if it goes up or down.
Like just kind of making those decisions about what it all means. Um, where you can get the data, how you're gonna get the data, um, like what you're gonna do with it. All those things. Like you, there's a lot of work that goes into it before the trial is actually in the field, which I think is really helpful.
Benjamin James Kuper-Smith: Hmm. Yeah. So since you already. [00:35:00] Made the bridge to your preference? Uh, preference or your, your paper. Yeah. Uh, I, I think I would, I'll take that. Yeah.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: Um, actually, maybe we can, it's funny we're just doing the conversation Exactly. The, the other way around that I planned it.
Hanne Watkins: Yeah. Yeah.
Benjamin James Kuper-Smith: Sorry about
Hanne Watkins: that.
Benjamin James Kuper-Smith: So I, I think, no, no, I mean like in terms of me also starting with the government stuff.
Oh yeah. I thought we talked like about morality and war. Mm-hmm. Then get to your paper, then talk about pre, uh, registered report because I have a few questions about that. Yeah. And then talk about government stuff, but now we're just gonna completely flip that around it.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: Um, so then I'm also just in general, I haven't.
Um, I think you're the first person I'm talking to who I know has done a registered report. Mm-hmm. Um, it's like something, you know, you hear about a lot and that I've, I've talked, I talk to my supervisor a lot because we kind of occasionally about it because we kind of like the idea and want to do it, but, so it's the question like when it applies, when it makes sense to do it.
So for a few minutes I'd just like to get your like, practical experience from [00:36:00] writing one. Mm. Um, and I've actually, I have been, uh, involved in the peer review process of write report.
Hanne Watkins: Oh, great. Yeah.
Benjamin James Kuper-Smith: And that
Hanne Watkins: I haven't, so
Benjamin James Kuper-Smith: Okay. Yeah, that was kind of quite cool. Um, although it's also a lot more work regular.
I can imagine.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: So, yeah. Anyway, um, how. Maybe how did you decide that this should be a registered report or how did that decision come about?
Hanne Watkins: Um, so this
Benjamin James Kuper-Smith: paper, or sorry, just very briefly, what is a registered report?
Hanne Watkins: Oh, yeah. Okay. So a registered report is when you submit a paper to a journal with the introduction written.
Um, so the literature review, you've developed your hypothesis on the basis of theory, and you've decided kind of on your method. Um, and then you submitted to the journal before you collect the data and analyze and write up the results and conclusions. Um, and the journal, [00:37:00] um, sends the paper out to review, you know.
Without that whole second half. And then, um, if they decide to accept it, they commit to publishing it regardless of what the results actually come out, um, as. And so it helps, uh, reduce publication bias because often, um, null results, for example, just or unexpected sort of weird findings, just don't make it into journals.
But they definitely happen and end up in people's file draws. So without registered reports, you can end up with a, with a really skewed, uh, sort of a biased representation of, of the studies that are run because you only end up with a sort of very neat, positive, positive in the research, in the statistical sense findings, um, in the literature.
Yep. So that was, is that an okay summary? Anything you'd like to add?
Benjamin James Kuper-Smith: Um, yeah, not, not. Much. I mean, like, one thing I think is kind of interesting is that I think this is probably the model [00:38:00] that most non-scientists kind of assume almost.
Hanne Watkins: Yeah, yeah.
Benjamin James Kuper-Smith: How science works, uh, in the sense that you get critical feedback before you do it, rather than just complain afterwards what you didn't do.
Hanne Watkins: Yeah, it's
Benjamin James Kuper-Smith: true. Which is a standard model.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: Uh, no. So I think, yeah, that's, I think that's a good explanation. Uh, so how did you then, yeah. Why, why was this, I mean like one, also what I'm trying to get at is, um, not only like why did you decide to do it, but what kind of projects are most suitable and when should you do red, but when shouldn't you?
Hanne Watkins: Yeah, that's a good question. Um, I think, just to answer that last part first, I think one thing that made it sort of more straightforward in this case to do a registered report was that we were directly replicating a previous study and then building on it. Um, and so I think. That helped because, [00:39:00] um, hmm.
Why do I think that helped? I don't actually know. I, it's just like an intuition. I'll, I'll tell you how it came about and then I can just rethink like what it is about it. So, so I, um, had a blog. Uh, it still exists, but I, I read this. What is
Benjamin James Kuper-Smith: it? Do you wanna tell people about it or,
Hanne Watkins: uh, it's, I haven't, I haven't updated it since 2018, I think, but yeah, it's, it's my scholarly group.
It might be interesting to follow this story on, on the blog, it's called My Scholarly Goop, uh, dot wordpress.com, which is a anagram of moral psychology. Um, but it, uh, I'd written,
Benjamin James Kuper-Smith: so just let me just quickly my, oh yeah,
Hanne Watkins: I
Benjamin James Kuper-Smith: just, my scholarly goop. GOOP or,
Hanne Watkins: yeah.
Benjamin James Kuper-Smith: And then, sorry, what? My scholarly goop WordPress
Hanne Watkins: do wordpress.com.
Benjamin James Kuper-Smith: What, uh,
Hanne Watkins: [00:40:00] did
Benjamin James Kuper-Smith: you find it because I'm also, uh, I think so, yeah. Yeah. The description in, uh, that's also the first thing that it says. If I just Googled it, it's quite up updated.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: It says this webpage hasn't been updated since February, 2018.
Hanne Watkins: Yeah, that's right.
Benjamin James Kuper-Smith: And below is old news. Okay, cool. No, I just, I'm, I, I'll put this stuff in the description also.
Hanne Watkins: Oh, yeah, yeah. No
Benjamin James Kuper-Smith: worries. It's good that I've found it. Yeah. Okay.
Hanne Watkins: Yeah. So, um, I read this paper by Jonathan Phillips and Farry Cushman that was looking at the way that, um, when you're presented with someone doing something immoral, one of the reactions that you might have is like, oh, you can't do that.
Like, as if it's somehow impossible, just because it's immoral. And I wrote about, just sort of briefly summarized the paper on my blog, said, oh, this sounds really cool. Um. Just, uh, I wonder how it holds up in a war context because in, in war, like things that you can't do that in a peace context, you kind of can, uh, a lot of the time [00:41:00] in war if you're a soldier and you know, appropriately authorized and so on.
Yeah. Um, so that was just a, like a question at the, at the end of the post, like, oh, can we replicate this? And see also it does it work in a war context? And I think Mark Brandt, um, just responded on Twitter actually and said, oh yeah, I'd be, I'd be up for that. Like, let's do it. Um, oh, because he's the
Benjamin James Kuper-Smith: co-author.
Hanne Watkins: He's my co-author. Yeah. Um, and I guess the other, the other thing that I'd mentioned in the blog post, uh, is that, um, the original authors had made all their materials and code, um, and data files and everything available. Um, on GitHub. So it looked really easy, right? To just like, take their materials and the code and just Yeah.
Re redo it all. So, so Mark said he was on board. So then I was like, okay, cool, I'll try and do this. And then, and I started working on, on the project and then wrote it another blog post about the way that I felt like that[00:42:00] I had heard about registered reports. At that point, I'd never done one before. I thought it was a cool idea in theory, but I felt just kind of like, like apprehensive about it, about actually doing one.
And part of that apprehension was sort of about like. Um, the way it's really exciting is to, um, get the results of a study and like run the analysis and see like, oh, does, like, is it gonna come out? And part of the excitement has to do with like, you know, that if it comes out, how you expect it and if the results innate, like get a publication out of it.
Um, potentially, whereas like the disappointment is, oh no, this, like, I'm gonna have to go back to the drawing board. And obviously that that sort of motivation and that. The, the fact that like getting results is so closely tied to publications that, that you have that sort of excitement, it doesn't actually seem like a good thing for science.
So, so, um, I was just kinda reflecting on that and thinking I should [00:43:00] probably try and overcome it. And then in emails or something with Mark, I can't remember exactly how it came about. We said like, oh, maybe we should do this one as a registered, uh, report.
Benjamin James Kuper-Smith: Mm-hmm.
Hanne Watkins: And it was, again, fairly straightforward I guess.
Like, so this comes back to what you said about what makes a good registered report. I think it was at least like a good one for me to start with because I didn't have to do so much work to kind of justify all the methodological. Choices, um, because, uh, that was already justified by the fact that we were fairly directly replicating a previous study.
Um, and of course with the results and everything not being there, it's the methodological choices that get a lot of focus in the review process. Um, as it turned out, the reviews we got, um, I don't remember them in, there was just one round of reviews. We got desk rejected at, uh, nature Human Behavior and [00:44:00] then got a round of review at, um, journal of Experimental Social Psychology.
And the only thing that, that I remember from the review process is that they really made us think about, uh, what it would mean to get a null result. Like how were we actually going to interpret, um, these findings? And that was also really helpful because again, like sort of having to think all of that through in advance.
In this case it didn't make us change, uh, much, but it's sort of, um. We, we really thought through like there are potential three-way interactions here. There's two-way interactions. Like what, what do we really think is gonna happen and why is each of those findings important for the research question and the theory that we're sort of addressing?
Um, one of the reviewers was fairly skeptical that, uh, a null, null result would be interesting at all. Um, and, and said actually that they thought that a registered report only made sense if like a null result end a [00:45:00]finding, um, like a significant result would be worthwhile. Um. I guess we got around that.
I think we just, we just argued. I think that in this case, well, you just
Benjamin James Kuper-Smith: said no.
Hanne Watkins: Yeah. Interesting. I can't remember how we responded. Yeah, I think we tried to argue that. Yeah, we did. I mean, interest people vary in what they're interested in, right? But in this case, the Noel would've had to do with, um, differences between war and pace.
And I think if a result of a finding, unlike moral judgment is oh, there's no difference between war and pace. Like, that's kind of interesting. Um, because it's not what I would intuitively expect. Um,
Benjamin James Kuper-Smith: I'm, I'm wondering would it, uh, it's actually, I, I'm just thinking about that point now, whether no result would've been interesting.
Yeah. I mean, because often, you know, a no result can also just mean somehow your manipulation didn't work or whatever. I mean, in this case, your manipulation is, you just say in war context basically. Not in war. Um, [00:46:00] so it seems like, you know, it's easy to understand for the people, but. You know, just reading the words and actually fully, you know, like fully,
Hanne Watkins: yeah.
Benjamin James Kuper-Smith: One thing I want to talk about later is a bit more, is that the thing about how, um,
to what extent it makes sense to ask lay people about these things, especially people who never really think about it. Um, or at least I don't.
Hanne Watkins: Yeah, yeah.
Benjamin James Kuper-Smith: Um, so like in that sense, from that perspective, I do wonder like whether it could have just meant that people don't really think about these things or something like that.
Right. Just don't have an opinion.
Hanne Watkins: Yeah, absolutely. Um, and I guess from a statistical perspective perspective, what we did was, um, include a, uh, equivalence test. So like, could we even say that, you know, like trying to argue that a null result means no difference is a little bit, sort of statistically tricky.
Um, and as it turned out that the main, we did get a null result for one of the key tests, but we also didn't find that, um. [00:47:00] The effect, like with the equivalence test that they were statistically equivalent. So basically we're just like, in this no man's land of, there might be an effect there that we just didn't detect.
We can't with confidence say that these two things are the same. Um,
Benjamin James Kuper-Smith: uh, just, I can't remember the details. What was your, so I have equivalence testing is another thing that, that I've been wanting to do, but never, haven't gotten. I mean, I mean there's, well technically now in the third year, my PhD, but, um, so I guess it's not like I've been thinking about it for 10 years.
But, um, the one thing that I always found with equivalence testing is like, you know, you need your smallest effects, size of interest. Mm-hmm. Which to me seems often very arbitrary. I can't remember what was yours in the paper.
Hanne Watkins: Uh, we took the, um, effect size from the original paper.
Benjamin James Kuper-Smith: As the smallest. Okay.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: Did the account, or did they have a large effect size or,
Hanne Watkins: uh, it's, so, it was, it's a little bit hard to say, like, because it was, um, [00:48:00] error rates in response to a, uh, sort of a time manipulation. Like, how many more errors do people make when they have to make a decision quickly? It's a little bit hard for me to say whether it was a big or large effect.
Like, 'cause I don't, I don't really have a good sense of like, what, what's a baseline in that sense? Like, obviously going from zero errors to, no, sorry, from a hundred errors to zero errors would be a big effect, but I don't, yeah. Um,
Benjamin James Kuper-Smith: okay.
Hanne Watkins: Sorry. Sorry. That's not, that's not, yeah.
But
Benjamin James Kuper-Smith: okay. So you just, you check the original paper and then
Hanne Watkins: Yeah.
Yeah.
Benjamin James Kuper-Smith: But it still has this Yeah. Sense of,
Hanne Watkins: yeah, definitely.
Benjamin James Kuper-Smith: People say what, what the smallest effects, what
Hanne Watkins: it means. Yeah. Um, and again, like that was something that,
Benjamin James Kuper-Smith: sorry, just to get to the point, because if you'd chosen a larger, smallest effect has, then it would've been
Hanne Watkins: different. Right.
Benjamin James Kuper-Smith: So
Hanne Watkins: then, um, but then I guess what you can say is like, any difference between them is smaller than this effect and someone will come.
Yeah. Because you picked a bigger [00:49:00] one. Right. And so someone will come along and say, yeah, but you picked a really big effect. And I'm actually interested in like, even if a difference is smaller than that. So Yeah. It's, it's a, yeah. Just an interesting, um. I think we, it would've been hard to pick an effect size, uh, if we didn't have that sort of like arbitrary as you point out, but kind of obvious baseline to go with.
Benjamin James Kuper-Smith: Okay. Actually, uh, so I'm just curious now about the review process from the author's perspective and the other side. But actually first, uh, just quickly, what was the reason that Mad Maha rejected it? Did they give a specific reason or did you just say No, thanks, I can't
Hanne Watkins: remember. Yeah, I think they just said no thanks.
Maybe it was like too niche.
Benjamin James Kuper-Smith: Okay.
Hanne Watkins: Yeah,
Benjamin James Kuper-Smith: they just, okay. They said thank you for your email.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: That surprise somewhere else.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: Yeah. Good luck. Good luck with another channel.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: [00:50:00] Um, okay. But, uh, okay. So from actually. You have. So one thing I found kind of neat is that you have the on, well, page two is the big figure of your, you testing a central assumption.
Oh yeah. Um, and, uh, well, the figure is an important part. You tested an a essential assumption. Um, I was curious, was that before you did, before you handed in the register report, or was that part of the register report or,
Hanne Watkins: uh, that was part of the, uh, original submission. Yeah. Okay. So the stage one submission.
Okay. Um, yeah, I can't remember why we did that. I think we just, as we were like in developing the materials and stuff, we were like, oh, everything hinges on this assumption. We better just test it. Um, so yeah, it was part of the original.
Benjamin James Kuper-Smith: Okay. Because I was curious because, uh, like the reason I asked that question is because it seems to me that if that assumption would've fallen flat, then.
You wouldn't have needed to do the rest. So [00:51:00] what did you then specify that in the register report saying like, okay, the assumption just doesn't hold, so we won't do the rest? Or did you in the register report say, we're gonna do it anyway? Or,
Hanne Watkins: oh, sorry. No, so, so that, that data we did collect before we submitted
Benjamin James Kuper-Smith: Ah,
Hanne Watkins: okay.
Yeah, sorry, that's Yeah. So you, oh,
Benjamin James Kuper-Smith: sorry. It was okay. Yeah.
Hanne Watkins: Yeah. So it was part like the whole Yeah, the, the, that whole bit,
Benjamin James Kuper-Smith: the, the, okay. The collective
Hanne Watkins: results was part of mm-hmm. The collective data. Yeah.
Benjamin James Kuper-Smith: Sorry. So sorry. Okay. Yeah.
Hanne Watkins: Okay. So that's, that's something that you can do, right? Like, if you have a couple of ideas that, that sort of, you just wanna explore and you've already collected the data and analyzed it, and then you think that there's a, there's something there that you really wanna like commit to.
Um, that, that's one way to, I think also like dip your toe into registered reports is to actually submit, um, a stage one manuscript that already has like the sort of traditional methods and results for like maybe one or two. Uh, initial studies, but then you are registering the, the final [00:52:00] study in the kind of package that's more confirmatory.
Yeah,
Benjamin James Kuper-Smith: yeah, yeah. I mean that, that is specifically one example I'm thinking of because we have like something where it, in a way it's kind of like expanding results we already have or um, yeah, generalizing them a bit, but then I, I always wonder, what I find really tricky is this whole thing of, let's say you then have another idea that would fit really well into it.
Can you then like add that as another register report? Or, it's also for me the question like a register report, does it, does it make sense to call it a register report if like one outta five experiments is a register Right. You
Hanne Watkins: these kind of question, right. Yeah, that's a good question. Yeah. Yeah, yeah.
And I wouldn't, yeah, I agree. That sort of starts to seem a little bit strange, but, um. I guess as long as it's clear in, you know, people skim the abstracts, it's clear in the abstract that only the last [00:53:00] study was, you know, submitted for review before the data was collected or something. Um, that doesn't seem like such a problem.
What label you wanna put on it? I don't know.
Benjamin James Kuper-Smith: Yeah, it, it is more just how you caught it, but
Hanne Watkins: yeah, yeah, yeah. Um, I was thinking something about testing assumptions. Uh oh. Actually, uh, one of my other papers kind of ended up being almost like a registered report by accident. Um, okay. So in it, um, it was a lot, the two of my studies for my PhD had to do with the principle of discrimination, so how people make, um, judgements about when soldiers versus civilians are killed or do the killing in war.
Um, and.
Benjamin James Kuper-Smith: Yeah, this is not, the discrimination is a very moral morality of war context specifically.
Hanne Watkins: Yeah, yeah. Sorry. Yeah. So principle of discrimination describes that, that you're allowed to kill soldiers and war, you're not allowed to kill civilians. Um, so I just wanted to say [00:54:00] again, like what do lay people think?
Uh, what we can hold on the question of whether that even makes sense, that's a thing to ask. Um, but, but, um, in my PhD I done, like, I was just sort of learning, you know, like how to set up experiments, all these kind of things. And so there were a few things about these two studies that was just kind of weird.
And, uh, nonetheless, I packaged them up like after my PhD and submitted them to a journal with my supervisor Simon Lamb. And the reviews came back and they were like pointing out all these weirdnesses, like, oh, this is a bit odd. And like, why did you have. Response, this kind of response scale here, and things were out of order here and this, would this have influenced your effects?
And they were fairly minor things, but in the end I just like, and I totally agreed, like, oh yeah, that's not how I would do it now. So what I did before, uh, resubmitting the paper was just like, run the study again with all the, with all those things fixed up, but otherwise [00:55:00] like very, very similar and the same analysis and exactly the same results came out.
So it was like, you know, that was reassuring. Yeah. But, but, um, yeah, that was kind of a, that was an opportunity to, to in a way, like I had committed to a particular method because I'd been through the whole review process, you know, if I'd made additional changes at that point, the reviews would've been like.
Come on, like, what are you, what are you doing? But instead, the changes were all like to just tidy everything up like the reviewers had suggested and like I agreed and then to, uh, replicate my own work and it was published, so. Mm-hmm. That's good.
Benjamin James Kuper-Smith: Yeah. That's another thing that to me, that kind of format complicates the question of whether to register report, because you can of course submit something and then they sometimes say, can you do this other small study?
Hanne Watkins: Mm-hmm.
Benjamin James Kuper-Smith: So, and that was part of regular peer review. You, you can kind of also have a registered report in it anyway often,
Hanne Watkins: and then you just don't get the label right. Like,
Benjamin James Kuper-Smith: yeah, [00:56:00] exactly. Yeah. But what the label,
Hanne Watkins: yeah,
Benjamin James Kuper-Smith: that's all I care about.
Hanne Watkins: Mm-hmm. Of course.
Benjamin James Kuper-Smith: Yeah.
Hanne Watkins: Contribution to science path.
Benjamin James Kuper-Smith: Yeah. Okay. So then, um, from what perspective does it make sense to then ask lay people these questions?
I mean, there's a similar. Discussion I once had, um, just privately with a guy who did experimental philosophy. Mm. Where there's also the question like, you know, why does it make sense to ask people who don't usually think about this thing? Um, yeah. So, yeah, when, when does that make sense? And when doesn't it?
Hanne Watkins: I think in my case, um, it didn't really make sense. Uh, and I made, uh, it was mostly a pragmatic thing, like I just, uh, can't get access to soldiers. Um, and I kind of wish that I'd made some different [00:57:00] decisions because of that. But, uh, I think it can make sense because morality is so much about, uh, regulating other people's behavior in a social setting.
And, um. Third party observers, like people who just, you know,
Benjamin James Kuper-Smith: sorry, just quickly, I think your microphone's super close to your mouth so I can hear, I dunno, is that movable? Is that better? Maybe it's better. Yeah, I just, there was a lot of, uh, just breathing air. Yeah, yeah,
Hanne Watkins: yeah.
Benjamin James Kuper-Smith: Okay. Yeah. Now it's better. Okay.
Hanne Watkins: Yeah. So, so, um, morality, uh, seems to function a lot to regulate people's behavior in a social setting. And in those cases, it's not just about the, like the perpetrator and the victim, it's also about these third party observers and the kind of, sort of punishments and praise and help, uh, that they, um, provide.
And so those are people who are not [00:58:00] necessarily. Directly involved, but their judgments still matter because they form part of that whole sort of structure of moral judgments and norms that then influence us directly and indirectly through like our knowledge about those norms and so on. Um, when it comes to like, people who don't think about it that often, uh, that's slightly different from the category of people who are just, you know, outside of a given conflict.
Like an external observer might spend a lot of time like thinking about something or they might not, um, you know, like I agree that, um, it doesn't really, in some ways it doesn't really make sense, but when it comes to war, like. I also just have this feeling like you should be thinking about it actually.
Sorry. Um, yeah, so, so I agree. Like people aren't really thinking about it that much and, and that's part of like how we've set up society is so that there's this particular group [00:59:00] of people who go and have it as their profession, and the rest of us don't really have to think about it. Um, but I think that's like a kind of, um, unfortunate gap, I suppose, um, for lots of different reasons, um, including just, uh, sort of the disconnect, like if you, I was in the US at the time, and so there's this trope of like, thank you for your service, right?
Like, and, and, uh, military personnel get to board planes first, and you go to a, a basketball game at, and at halftime there's like a big, you know, like fanfare and, you know, thank you to the, the military men and women here
Benjamin James Kuper-Smith: really every game.
Hanne Watkins: Uh, maybe not. I've only been to a few days. Yeah,
Benjamin James Kuper-Smith: yeah. You didn't go to all the games,
Hanne Watkins: all the game.
No. Um, but there'll be these like little moments, right? Where, where the mil, the military is kind of, uh, highlighted, but it feels very, um, [01:00:00] superficial, sorry about the dog again. Um, and, and like not a very strong engagement with what you're actually asking them to do on your behalf as a citizen. Um, so, uh, there's this sort of, um, gap between the military and civilians that, that obviously my research didn't actually help with.
Maybe if I'd continued in academia, I would've gotten there eventually. But, but, um, yeah. So what we can. What we can learn from asking people to make these judgements when they're not used to thinking about it. Is there, uh, like what they come up with in that sort of unreflective, like, oh shit, you're asking me to think about what?
Uh, alright. Like, and if you do start to see particular patterns or biases in those kind of unreflective, I think you have learned something provided that, that um, you know, I guess it could just be noise and so just nothing ever replicates 'cause [01:01:00] it's just gonna be different for all these random reasons.
Um, but if you do start to see patterns, I think, uh, maybe you've learned something. Hopefully.
Benjamin James Kuper-Smith: I mean, this is also psychological research, right? It's not. Um, you know, you're trying to understand how people think about this, not make an argument for or against
Hanne Watkins: Yeah, no,
Benjamin James Kuper-Smith: exactly. More principles. I mean, that's, that's what we have.
I don't know. Lawyers for philosophers. Yeah.
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: Um, but not, not, not psychologists.
Hanne Watkins: No. Um, and part of the reason why I started studying it was because I had been reading a lot of philosophy of war and, uh, it touches like the, the philosophy of war. Um, sort of talks a lot about the legal systems surrounding war and vice versa, but it doesn't really seem to talk that much or at all about the sort of intuitions that everyday people have about it.
And sometimes some of the conclusions that they come to seem to [01:02:00] be pretty like, uh. Disconnected from, from how like the, the sort of uninformed person might just think that it, things should work. Um, so that's, and that's, that can be a pro, like you don't always want, uh, like, um, I think there's a role for philosophy in pushing people towards like a different way of thinking.
Like if you think of, for example, Peter Singer and the way that he draw takes like utilitarianism to the logical extremes. Um, but, but by like starting to think about suffering and animals and so on, you know, he's pulled a lot of people in the direction of caring about something they didn't previously care about.
Um, even if they haven't gone all the way to the, uh, extremes that he has. So, um, similarly, like there might be a reason why that there's a gap between what moral philosophers say about war and what regular people sort of think and say about war and that. It doesn't have [01:03:00] to be that the philosophers should move to be more similar to the everyday person.
It can also go the other way around. I'm not making any like particular arguments in that sense, but you've gotta find out what the gap is first, right? Before you can start to think about is it even a problem? Yep.
Benjamin James Kuper-Smith: Yeah. I mean for, for me, just personally reading, um. I think this was more about the Marat of war or review and research agenda, um, about that paper there.
What, what I kept thinking about was what, so I'd never, you know, I I, it's kind of funny that I've never really thought about war considering I grew up in the first Yeah. In the, in the first German city that was freed by the allies
Hanne Watkins: Oh wow.
Benjamin James Kuper-Smith: And was like, where I grew up, like when there's like the, I dunno what they're called.
There's like huge stones that tanks can't drive over.
Hanne Watkins: Oh, right.
Benjamin James Kuper-Smith: Um, that are like just, there's like li like rows of that through the countryside. And so they, they keep them there in Germany. Just so to remind you like, you know, this used to be a thing. Mm,
Hanne Watkins: mm-hmm.
Benjamin James Kuper-Smith: Uh, so [01:04:00] it was kind of like that. It's kind of weird that I grew up like that and I've never really thought about war, but one thing that really struck me is like, like it's obvious that there are different legal or different legal, I dunno, was it systems or structures or rules or kind?
Yeah. Yeah. For war and for peace context. And like, I just thought about it. I thought like. Why would it even make sense to have different legal systems for one, for peace? And I thought like, shouldn't killing someone. Well, I guess we, I mean, I know nothing about law either, but, um, you know, when I, from what I understand, when someone has killed someone, you take into account the reasons and all these kind of things, right?
Mm-hmm. Um, but it still seems kind of weird to me that you'd make this huge distinction between saying, well, the country's in war, so therefore you can shoot people.
Hanne Watkins: Yeah, yeah.
Benjamin James Kuper-Smith: Um, as a kind of categorical difference. But then for me, I mean, like, I also think about game theory and these things because of my own [01:05:00] research and like the what first then I kind of thought about that.
If you, okay, so let's say you have the same legal system across the board.
Hanne Watkins: Mm-hmm.
Benjamin James Kuper-Smith: Then that would probably just be a, um, how should we say that would not be. Uh, good for the only country that does it, right?
Hanne Watkins: Yeah. '
Benjamin James Kuper-Smith: cause they just get killed.
Hanne Watkins: Mm-hmm.
Benjamin James Kuper-Smith: Yeah. So it almost feels like this kind of thing where evolutionarily you can't really, whoever it's adopts
Hanne Watkins: it's massive, massive coordination problem.
Benjamin James Kuper-Smith: Yeah. Yeah, exactly. And as if one person or one not person, one country, yeah. You can't really go, can you go as a single person to war? Um,
Hanne Watkins: that's a good question.
Benjamin James Kuper-Smith: But if a single country were to adopt it, then they just would stop to exist.
Hanne Watkins: Mm
Benjamin James Kuper-Smith: mm
Hanne Watkins: Yeah.
Benjamin James Kuper-Smith: But do you, is that like, do you know why there are these two separate systems?
Is that part of it or is that completely different, like legal reason? Um,
Hanne Watkins: good [01:06:00] question. Uh, it hasn't always been separate. Um. You'd have to talk to, like a historian of the legal structures of war or something, to, to think about how it, how it came about. Um, but just war theory does have its roots in sort of theorizing by, um, Thomas and Thomas Aquinas and Augustine, um, where they were kind of trying to reconcile, um, like the Christian sort of prohibition on killing mm-hmm.
With the fact that of the king can just like, send his man out to slaughter and pillage and take, like, do whatever he wants basically. Um, and so I feel like possibly, and this is just, it's been a while since I've thought about this and it wasn't a huge part of my reading, but I feel like the seeds were probably sewn at that [01:07:00] point.
Right. And then so they, they were saying not, not. Like, yeah, this on the surface of it just seems kind of wrong. Like if we're saying it's wrong to kill another human, those people are being sent out to sort of kill others. Like err. Um, and then started to sort of think about like, oh, well, what might be the reasons that we can give to say like, oh, well that's, that's justified.
Um, and then you get into religion and, um, nation states and all these other, uh, things. Um, the, I guess I would say that one sort of interesting trend that I noticed, um, noticed just maybe in the last like two or three years, I'm sure it's been going on for longer before it popped up on my radar, but is this move towards it?
It sort of seemed to be at, uh, for a long time that, um, the. That you didn't really think about [01:08:00] soldiers as individuals, that they were, uh, a collective sort of extension of the state. Um, but that it, and, uh, that, I mean that in, in like the philosophy of war sense, but then over time it sort of became more of like, no, we do have to think about the individual responsibilities and individual rights of soldiers.
They, they are, um, sort of agents, uh, even as they are, they, they are like, uh, you know, autonomous agents as well as being agents of the state. Um, and so how do we reconcile these things? And yeah, I think that's a really interesting direction that things are going in. Um, but not working on it anymore. So I don't know what's happened Yeah.
In the years since I was there. Yeah.
Benjamin James Kuper-Smith: Yeah. I just find it so fascinating that there's these, I don't actually know what the. Decision is on these cases, but there's like some cases of, you know, [01:09:00] soldiers being in a country and they don't know war stopped. Mm. But they just keep killing people because they think they're still in a war, but they're actually not.
Yeah. And that's just really weird. I don't actually know what you do with these people because like technically they're committing a crime.
Hanne Watkins: Yeah, yeah.
Benjamin James Kuper-Smith: But,
Hanne Watkins: but,
Benjamin James Kuper-Smith: but yeah, it's really weird situation. Mm. Because you can always just claim it. You can just stay there, kill some people if you want to say, oh, sorry, I thought,
Hanne Watkins: I thought, thought
Benjamin James Kuper-Smith: it was,
Hanne Watkins: yeah.
Right. Yeah. Yeah. It's, it's definitely, it's fascinating. Um, and, uh, actually maybe this will be a nice, uh, uh, sort of positive note to end on or a question to end on. Oh, nice. Is, uh, did you realize that, uh, that um, nuclear weapons were banned, uh, on Friday yesterday?
Benjamin James Kuper-Smith: What? No.
Hanne Watkins: Yeah. So, uh, now if you use a nuclear use, develop store, whatever, uh, weapons you are, uh, let me see.
I'll, we can look it up.
Benjamin James Kuper-Smith: Feel like that's something I should [01:10:00] have heard of.
Hanne Watkins: Yeah, yeah.
Benjamin James Kuper-Smith: Although to be fair, it's Saturday morning right now, so I don't know. Um,
Hanne Watkins: oh yeah, yeah, that's true. I'll send you that. I mean, it's like, it's what the perennial question about what, what does it matter what the UN says if it can't enforce anything?
Um,
Benjamin James Kuper-Smith: yeah, yeah.
Hanne Watkins: But in terms of setting international norms and that coordination problem that you mentioned, uh, with game three theory, like what happens to the one country that that tries to, um, I think that it's important to, uh. Like, if you think of the ban on, um, cluster munitions and things, it's pushing people in the right direction.
Benjamin James Kuper-Smith: It also says the, the, the, the subtitle, major
Hanne Watkins: cows sign
Benjamin James Kuper-Smith: it. Yeah. Despite a lack of signatures. Maybe that's why I didn't hear about it, because they're like, nah, we're not gonna do that. We are not gonna let Kim Young be the only one.
Hanne Watkins: It'll be the only, yeah. Yeah. So maybe the rest of it. So just freeloading, who knows?
Benjamin James Kuper-Smith: Maybe. Okay, [01:11:00] cool.