Giuliana Spadaro is a postdoc in the Amsterdam Cooperation Lab, directed by Daniel Balliet. Her research focuses on cooperation and prosociality. In this conversation, we talk about Giuliana's recent work on the Cooperation Databank (https://cooperationdatabank.org/), a database that contains around 2,600 studies on cooperation, coded by experts to facilitate meta-analyses and other tasks about cooperation research.

BJKS Podcast is a podcast about neuroscience, psychology, and anything vaguely related, hosted by Benjamin James Kuper-Smith. New conversations every other Friday. You can find the podcast on all podcasting platforms (e.g., Spotify, Apple/Google Podcasts, etc.).

Timestamps
0:00:05: Giuliana's career before working on the Cooperation Databank (coda)
0:13:09: What is coda and what can it do?
0:18:58: Different payoff matrices in the Prisoner's Dilemma
0:24:25: The benefits of annotating hundreds of studies
0:28:57: Further uses of coda (e.g., search engine)
0:33:28: How can people add their own studies to coda (including unpublished studies)?
0:39:10: Coda in the long term
0:45:15: What if I want a new feature added to coda?
0:53:47: Learning to run and from a meta-analysis
1:02:49: Working on Coda
1:11:38: What's next for Giuliana?
1:15:03: Coda workshops

Podcast links

Website: https://bjks.buzzsprout.com/
Twitter: https://twitter.com/BjksPodcast

Giuliana's links

Website: https://amsterdamcooperationlab.com/giuliana_spadaro/
Google Scholar: https://scholar.google.de/citations?user=ZuzhtPEAAAAJ
Twitter: https://twitter.com/g_spadaro90

Ben's links

Website: www.bjks.page/
Google Scholar: https://scholar.google.co.uk/citations?user=-nWNfvcAAAAJ
Twitter: https://twitter.com/bjks_tweets

References
Kuper-Smith, B. J., Doppelhofer, L. M., Oganian, Y., Rosenblau, G., Korn, C. W. Risk perception and optimism during the early stages of the COVID-19 pandemic. PsyArXiv.
McShane, B. B., & Böckenholt, U. (2017). Single-paper meta-analysis: Benefits for study summary, theory testing, and replicability. Journal of Consumer Research.
Scaffidi Abbate, C., Boca, S., Spadaro, G., & Romano, A. (2014). Priming effects on commitment to help and on real helping behavior. Basic and Applied Social Psychology.
Spadaro, G., d’Elia, S. R., & Mosso, C. O. (2018). Menstrual knowledge and taboo TV commercials: effects on self-objectification among Italian and Swedish women. Sex Roles.
Spadaro, G., Tiddi, I., Columbus, S., Jin, S., ten Teije, A., & Balliet, D. (2020). The cooperation databank. PsyArXiv.
Thielmann, I., Spadaro, G., & Balliet, D. (2020). Personality and prosocial behavior: A theoretical framework and meta-analysis. Psychological Bulletin.

[This is an automated transcript with many errors]
Benjamin James Kuper-Smith: [00:00:00] So, you know, I'm interested in the corporation data bank and I am basically here is like how you got to work on that. Um, so if we can maybe talk a bit about like your career so far, whatever you wanna call it. Um, I can't remember, was your bachelor's in psychology or?

Giuliana Spadaro: Yes. So actually like, uh, my background, uh, led me very naturally to this, uh, this, uh, this new position as a postdoc here at the pool.

And, uh, for example, I did my bachelor in Palermo, that's a, uh, super South Italy. And I did my bachelor, like running an experiment of the, on the bystander effect. Actually a replication study was really something that, I'm sorry, which

Benjamin James Kuper-Smith: effect?

Giuliana Spadaro: Uh, the bystander effect.

Benjamin James Kuper-Smith: Oh, bystander. Sorry. Yeah.

Giuliana Spadaro: Yeah. So it's, uh, so it's all related to the prosociality and prosocial behavior.

So since the very beginning. Then I moved for my master, [00:01:00] uh, to focus a, a little bit more on research and get a master that was, uh, like focused on psychology, but also had some, um, computer scientist, uh, science, science notions or, uh, a lot of data analysis and then so on and so forth. That really, uh, brought me to my PhD.

Uh, I did my PhD in, uh, in Torino at University of Torrin in the north of Italy. And, uh, my PhD was, um, in social psychology, but I also attended some course on, uh, anthropology and education. So I'm, I can say I'm a psych, I have a PhD in psychology, anthropology education, very broadly, but let's say

Speaker 3: psychology then.

Benjamin James Kuper-Smith: Yeah. By the way, you are the, you'll, you'll be the second guest from who you studied in during.

Speaker 3: Oh,

Benjamin James Kuper-Smith: really? Uh, I have, yeah. The next episode is gonna be with Bianca. Tovo. I think she's from Turan. I can't remember. It's funny just because I was, I was just editing, just finished editing her episode. So I've, I hear a lot of, I'm not [00:02:00] gonna hear a lot of Italian accents right now.

It's just, um, yeah. But you're not from Turan or?

Giuliana Spadaro: No, no, no. I'm, uh, from Palermo.

Benjamin James Kuper-Smith: From Palermo, okay. Yeah. And wait, how does psychology, well, what is a psychology degree like in Italy? So, in the UK it's very, well, I did, I did psychology as, uh, for my bachelor's. It's pretty formalized. So if you do a, like, psychology bachelor's, you, like, the first two years is like, I think pretty much wherever you do it, it's the same thing.

You have like some methods and you have like one module on the different topics. You know, have like something about like clinical psychology. You have something on cognition, you know, like you, you go through like the whole thing. Is it.

Giuliana Spadaro: Yeah, similar there,

Benjamin James Kuper-Smith: or

Giuliana Spadaro: in Italy, the Bachelor is pretty much organized in the same way.

So it's pretty general and broad. And then for your master, that takes like two years. Uh, you decide, uh, how you want to specialize and what you want to focus on.

Benjamin James Kuper-Smith: And, uh, why did you go to CHI and do that degree? [00:03:00]

Giuliana Spadaro: Because they had a very unique program that was not, uh, of course you want to specialize at the master, but I, I, what I knew is that I wanted to do research and in Italy there is nothing like a research master.

So that was like the closest thing that was at that time out there because it was general, still general enough to allow you to, uh, pursue many different, uh, career options. And I like that.

Benjamin James Kuper-Smith: I think that's why a lot of people do psychology, right?

Giuliana Spadaro: Mm-hmm.

Benjamin James Kuper-Smith: Because it doesn't like, uh, it doesn't force you to, you know, do one thing or something.

Giuliana Spadaro: Yeah.

Benjamin James Kuper-Smith: At least that's why I took it. Like, you, you feel like you can do whatever you want with it. Um,

Giuliana Spadaro: yeah. Then you realize, uh, that, uh, it is not that easy. So

Benjamin James Kuper-Smith: the degree or the

Giuliana Spadaro: no, that you can do whatever

Benjamin James Kuper-Smith: you want with that. Yeah, yeah. Exactly. Yeah. Yeah, yeah. Yeah. But it seems that you were like pretty focused in that regard, right?

Like in [00:04:00] terms of working on cooperation and

Giuliana Spadaro: Yeah, yeah, yeah.

Benjamin James Kuper-Smith: Social interactions.

Giuliana Spadaro: Exactly. So also, like my master thesis, for example, uh, which is, so cooperation is not a research tradition at University of Terrain, but my master thesis was on, uh, uh, the effect of priming on cooperation, like priming of trust on cooperation.

And, uh, yeah. So I, I really started pretty early on with, uh, with these.

Benjamin James Kuper-Smith: Do you think that's a good thing or? I mean, for example, I, I almost took the opposite approach where it seems like every half year I'd work on a different research project. Like until, like I, when I started my PhD, which is on like cognitive neuroscience of social interactions, let's say.

Um, like I, I've worked on attention. I did a project there, I did something on, um, well, I, I, I did actually do one corporation thing before I did volition, uh, for one of my master's projects for another, I did like theoretical [00:05:00] neurobiology for another, I did body ownership, you know, like I, I went like all over the place basically.

And in some sense, I think like it's, you know, the nice thing there is that you have this like, breadth of overview of different topics. But of course, like when I then started my PhD, I had almost no specific knowledge about the topics. So I. How was, I dunno, like in hindsight, do you, are you glad you focused on one thing and stuck to it or would you prefer to, I don't know, do some other stuff or?

Giuliana Spadaro: Well, you know, it's, uh, of course it's a trade off, uh, because it's very efficient, uh, and it can allow you really to go, uh, in, in depth in that research topic. And, uh, for the PhD that's super convenient because you don't, you don't have most of the times like, uh, all the time in the world to, to perform your literature search and uh, and so on and so forth.

But okay, yeah, sure. I was focused on something, but, uh, during my PhD I also [00:06:00] did like other projects that were unrelated to my dissertation and, uh, they were pretty, yeah, they were, they were, they were not related on, on cooperation at all. So like, if you see now a paper letter I published with, um, with a student of mine.

We analyze, uh, women self-objectification in response of, uh, seeing advertisement in Sweden and Italy of menstrual product. So there is a, it's

Speaker 3: very, it's pretty far from, uh, cooperation, I can say.

Benjamin James Kuper-Smith: Yeah, it sounds like it. Yeah. Now, I mean, especially asking like the, the focus versus breadth thing because, you know, we once had, um, someone we supervised for their bachelor's project in our lab, and then there was a question whether they wanted to also do their master's project.

And I always felt like, almost, almost felt like telling them like, no, don't, don't do like your project in the same lab twice. Like go out, go somewhere else. Like, you know, experience what kind of research you can do. Don't just, you know, I mean, we're of course a [00:07:00] very good lab and very cool, but like go out and do other things.

Don't just yeah. Focus so early. But yeah, I don't know.

Giuliana Spadaro: Oh, makes sense. Uh, makes complete sense. So for example, what I did for, for my PhD to like, uh, to reach some, some, some kind of trade off also here was like to visit other departments as well. So like, uh, I spent some months at University of Vienna, the Department of Economic Psychology, and at the very end of my PhD I spent some months as well at dfu at the free university.

Uh, yeah. Okay. That was maybe more related to my dissertation, but, but still you could really experience, uh, different stuff.

Benjamin James Kuper-Smith: How did you end up in Vienna briefly, or what, what were you doing there?

Giuliana Spadaro: Um, I did, um, a series of studies, uh, that, uh, they end up in my dissertation eventually. And they, that specific research group that I was visiting was, uh, working on a model on the tax [00:08:00]compliant.

And, uh, that pretty much resembled another way to study public goods, uh, that was like way different from what I was used to do in the lab. So I thought that was pretty cool. And even if we didn't focus on taxes, eventually it turned out to be interesting.

Benjamin James Kuper-Smith: But how did you make that connection or get there?

Like, did you find them and say, this is cool, or Did, was there Yeah,

Giuliana Spadaro: exactly as you say, it's, uh, it was not a collection, uh, a connection that we had, uh, before. Uh, like I read some papers and I, I found them cool and I just contact the people. They were, they were pretty welcoming, I must say.

Benjamin James Kuper-Smith: Okay. And that was just easy to do as part of your PhD or did you have to take time out or

Giuliana Spadaro: No, no.

That was a part of my PhD and that was a pretty much encouraged at University of to do.

Benjamin James Kuper-Smith: Yeah. I'm still considering doing something like that, but in a way, I, I already, we, we did like a [00:09:00] study on COVID. Um, I'm gonna upload the second version of the Preprint today once we're finished. Um, yeah, I think

Giuliana Spadaro: it's crossed

Benjamin James Kuper-Smith: so in a way, like I feel like I've already done my, like, non PhD work, even though it didn't meant to spend as much time on it.

But yeah, it always depends on like how much time you have for your PhD, how much money you have, that kinda stuff. Um. But then, but the thing, so if I read it correctly, when you went to Amsterdam, that was already in the same lab you're in right now, right?

Giuliana Spadaro: Yeah, that's correct. Although when I went to Amsterdam, I was visiting, uh, professor Pul van because, and we were working on a, a, a different topic.

So we were working, uh, we were trying to come up with an experimental paradigm to study the effect of corruption on cooperation and trust. So that was what I was doing there. So I was not working directly with Dan, with Dan Bt with my Ah, I see, I see the PI in my lab right now. So

Benjamin James Kuper-Smith: [00:10:00] that was more coincidence that you were already in Amsterdam or

Giuliana Spadaro: completely

Benjamin James Kuper-Smith: Ah, I see.

It sounded like when I saw your severe whatever, it looked like you, you know, did like a brief internship with Daniel Bait and then afterwards went like, oh, can I do a postdoc? You know? Um, okay. Okay. So that was completely different. Okay. So let's move slightly closer towards the corporation Data Bank.

How did you get your, your current position as a postdoc?

Giuliana Spadaro: So actually the, the, the, the data bank project started way earlier than I joined the lab. So I think, uh, three years, uh, was, uh, was already out there for three years and people were working on it. So there was a transition time in which they opened a position to find another person to be in charge of the annotation of the literature and on the training of other people.

And, uh, as I said, I was, uh, there working on other topics on cooperation, and I applied for, uh, to the position. So that's how I [00:11:00] am. Uh, I transitioned to this new job, but at the beginning it looked, uh, very different from what it turned out eventually, because my responsibility was mostly to work very closely, uh, to the data.

Benjamin James Kuper-Smith: Rather than, or what did you expect to do?

Giuliana Spadaro: Yeah. Rather than building the entire infrastructure and, uh, bringing it to the next level, to the, yeah,

Benjamin James Kuper-Smith: because that had already been done, or

Giuliana Spadaro: No, because the, the entire project started with annotation of papers. So what, what what we did was to search and collect these studies on human cooperation using economic games and to annotate really in depth, uh, whatever was going on in there.

So in terms of how these experiments were structured, the variables that were in there and the findings, so this is how it started. There was no platform, no research interface, no logical and [00:12:00] representation.

Benjamin James Kuper-Smith: Mm-hmm. Um. I wanted to ask this later, but I guess because you just mentioned all the annotating.

Oh, okay. That sounds like a lot of very tedious work. Um, is, is it as, as, um, tedious, let's say as it sounds or, uh, you had lots of people, right? It wasn't just you doing all of those studies, but,

Giuliana Spadaro: um, I worked, uh, really, um, full-time on the annotation, at least for my very first year.

Benjamin James Kuper-Smith: Okay.

Giuliana Spadaro: So I can, uh, so I know what I'm talking about.

Um, yes. It's a very detail oriented, uh, kind of task and, uh, like to give you some kind of, yeah. To give an example to annotate, um, a relatively straightforward study, let's say a study that was, uh, conducted by psychologist and published in a psychology journal can take, um, uh, alpha hour. A single study. So

Benjamin James Kuper-Smith: that's like [00:13:00] a straightforward study where, you

Giuliana Spadaro: know, a straightforward study in which you know exactly what is going on and you know where to find and to look for the information.

Benjamin James Kuper-Smith: Yeah, yeah, yeah. Okay. Um, yeah, shall we maybe then introduce what we're actually talking about here? So what is the corporation data bank? What can it do, what problem does it solve?

Giuliana Spadaro: Okay. So we can, uh, I think, uh, we can define the corporation data bank as a machine readable and annotated body of data that we extracted and curated, uh, from studies on human cooperation that use economic games.

So when this project started, the main purpose was to, uh, give the possibility to everyone who is interested in cooperation to, uh, perform meta-analysis and to get a research synthesis. On the, on this specific topic, but then like, uh, it really got bigger than that. And, uh, it allowed to perform many more function and uh, and to accommodate more.[00:14:00]

Yeah, to accommodate more than this.

Benjamin James Kuper-Smith: Uh, what do you mean by more than this? Than

Giuliana Spadaro: because what we realize, so for us, like, uh, the, the original group started as a group of people who were mainly interested in meta-analysis. So we know how it works, like when you code a paper or annotate that, um, and so on and so forth.

Uh, so we're pretty much oriented towards that When we had this, uh, large body of annotated data and we had to fit it into like some formal data model. Structure this knowledge that for us was like really Excel files with a lot of data. We realized that nothing similar was already out there that was able to model really this kind of information.

So what the next step of the project was, was try to, uh, really find a formal way to represent this knowledge and this information because we were doing that in a very intuitive, um, meta-analysis oriented way like researchers would [00:15:00] do.

Benjamin James Kuper-Smith: You mentioned machine readable earlier. What exactly does that mean?

Giuliana Spadaro: Um, this comes from the idea that all the knowledge that is, uh, containing this studies is actually buried into these PDFs. So like if you open an article and you want to, uh, understand immediately what is going on, what are the findings and the characteristics, there is no way to, for machines to extract this information for you.

So what we wanted to create was a way to annotate the data and to. And to formalize this knowledge that machines could read and you could perform some kind of operations on.

Benjamin James Kuper-Smith: Mm-hmm. But that means then, for example, like you say, let's say you, you take one paper and you say, okay, here we had 24 people, uh, 50% male, um, you know, cooperation rate of whatever, um, that kind of stuff, right?

That's what you mean.

Giuliana Spadaro: Yeah, that's correct.

Benjamin James Kuper-Smith: Just take you, kind of pull out the relevant information or [00:16:00]

Giuliana Spadaro: Exactly. But even if, like, if you have, uh, these, uh, this PDF with the research article, let's say, and you want to look for the, um, for the percentage of males in the sample, how would you even look for it in, in your document you're gonna like, think of some keywords like gender, male, female, and just jump from results to result in your, in your document.

So this is not a, uh, an efficient way like it is, is not really read. But if you annotate this information and you formalize that, uh, there, it's, it becomes easier to extract the information directly.

Benjamin James Kuper-Smith: Do you mean by that like having a big exo spreadsheet and putting it like each row is a study or how should I imagine this?

Giuliana Spadaro: Yeah, that's, that's correct. So you can, uh, this is, this will, this is will happen if you download, for example, the data, uh, going, uh, into the cooperation data bank research platform. If you download the data, you can then [00:17:00] navigate them in a, an excel, uh, spreadsheet, uh, form. And, uh, of, so this, this kind of file is gonna have a lot of columns.

There are gonna be all the variables that are in there. So there's gonna be one column, uh, about gender. And, uh, you can directly check this information, but if you want, you can also retrieve the same information from, um, from the interface and the platform. Yeah.

Benjamin James Kuper-Smith: You mean like the website? The

Giuliana Spadaro: Yeah, yeah, yeah.

Benjamin James Kuper-Smith: Way of the thing. Yeah. Yeah. So, so is the idea then kind of the overall idea to allow other peoples, to do basic peoples, to allow other people to do meta-analysis just easily? Is that kind of the basic idea, or?

Giuliana Spadaro: That's correct. That was, uh, the vision that then Ball had at the beginning of, uh, of the project because him, he run many meta-analysis and published many meta analysis in his career.

And, uh, he was well aware that although meta-analysis is often like, uh, sold as a, as [00:18:00] a kind of an easy way to, to get a paper published and, uh, because you don't really need to run the study, you, it just like use a recycling of secondary data. It's not easy at all and it's very Italian because it takes a lot of time and it's not true that all the research groups and all the labs in the world, uh, actually have this time and resource.

So what he had in mind is really to build this kind of resource that would allow everyone to do that and everyone to have access to the same knowledge.

Benjamin James Kuper-Smith: Yeah. I have to admit, every time I see a meta analysis, I always assume like, ah, like when people do lots of meta analysis, I know this is wrong, but I always think like, ah, it's just easy way to get lots of citations.

But obviously it's, it's not like it's lots of work to actually do.

Speaker 3: Yeah. It, it takes, uh, it takes, uh, some time, let's say, but that's, that's true. That's really a common, uh, prejudice.

Benjamin James Kuper-Smith: It's, it's kind of like, I'm really glad that you made [00:19:00] the state bank because I so add this one idea that I tested experimentally in the lab and, you know, we had a few experiments and, or maybe I'll back up a bit.

Like I, I've always been interested in like what, like from a practical perspective, what payoff matrix do you use when you do a prison Cinema, for example, because there's no like guideline like. Here are the relevant factors for using a payoff matrix. Like it's, it's just everyone just uses a payoff matrix, and sometimes it's specific to the experiment.

Like there, there's specific reasons within the experiment, but often it's just kind of arbitrary. Um, anyway, so I was thinking about that. And then, so I started like writing down different payoff matrices from, from all the papers I came across. So now I, like, I have this like word document with a table in it that, you know, it started off just like very casually, me just going, oh, I'll just write down what they did, what they did, what they did.

And now I have this thing with like 200 payoff matrices or something stupid. Um, because every time I read one, [00:20:00] I just add it to it. And it's kind of nice that people like you are doing this in a more systematic way. So it's just not a word document on my laptop, but I'll say this. You don't have payoff matrix in there, right?

Giuliana Spadaro: Uh, yes, that's correct. You're right. We, we don't code for this specific payoff. And why not? That's the

Benjamin James Kuper-Smith: one thing I want.

Giuliana Spadaro: That's a, that's a very, yeah, that's a very good question. Although I'm very optimistic. These can, these, uh, will be integrated soon because some people are already working on annotating that for some studies.

Benjamin James Kuper-Smith: Yeah.

Giuliana Spadaro: So yeah, I'm optimistic it's gonna be included.

Benjamin James Kuper-Smith: Oh, that would be great. Yeah, because I mean, I mean, yeah, there's a few things you could be interested in there. Of course. Uh, the one, the one thing I'm interested in is part particular is the kind of gain loss of individual payoffs. So not whether everything is gain or loss, right?

You already have that in your study, the gain loss frame. What I'm more interested in is like you can, you know, because the, in a [00:21:00] prison dilemma, the order of the four payoffs is, um, well that's what defines the prison dilemma. But you can obviously shift that all of that relative to zero. So none or all of them, or some, or some of them are positive or negative.

Um, and what, I mean, the one thing that I found, for example, interesting in my. My very professional table in my Word document is that there's almost no payoff matrices that use negative numbers at all. Um, I found that really interesting, almost. I mean, you know, you give people money, it makes sense, uh, in these studies, but it's, there's almost like the proportion of what I wrote down is probably like 90% use only positive numbers or something.

Yeah. Uh, so like I wanted to see whether I could do metalized over this, but there just weren't enough studies I even knew of that have negative numbers to begin with. Um,

Giuliana Spadaro: yeah, that's a, that's a very good point. And, uh, it's true. I mean, if I have to come up with a proportion as well, I think that the, the positive payoff are around [00:22:00] 85% of what's out there, so it's really more like on the norm.

Benjamin James Kuper-Smith: I mean, as I said, like it makes sense from a practical perspective, right? Because you, you, you, you want to incentivize it so you give people more money. You know, like one problem we had is that you can't really, like let people give you money for taking part. Like if you do a real prison dilemma and the the outcome is negative, you can't really do a study where the participant owes you 20 euros afterwards.

Right? Like, that doesn't,

Giuliana Spadaro: but, uh, I can, uh, I can tell you from experience that there are several studies that, uh, for example, give a budget before to cover this kind of loss.

Benjamin James Kuper-Smith: Yeah. That's what we did, basically. Yeah.

Giuliana Spadaro: Yeah, exactly. Okay.

Benjamin James Kuper-Smith: Yeah. Yeah. That's like the only thing you can really do. Um, yeah, I mean, it just also, like, for me, not even like I know of.

Yeah, it, it just feels really unethical to have, you know, people to give you like an hour of their time and then you, they have to give you 10 euros afterwards or something like that. Just [00:23:00] sounds a bit wrong.

Speaker 3: Yeah, it is. And I think it's already hard enough to give them back because you are, you as a participant to think they're already in your pocket, you know?

So. Yeah.

Benjamin James Kuper-Smith: Yeah. I mean, we had to say like an our thing. We had, like, we did an experiment. There were like three factors and each of them you could win or gain money, that kind of thing. And one person just got really unlucky, you know, but, which is what you'd imagine if you test lots of people, right? At some point someone's gonna get very unlucky.

One person also got very lucky and took away a lot of money. Um, but you feel really bad for that person who gives you. That's the weird thing, right? Technically we paid them less than they should have been paid per hour, right? Because of the incentive structure and the whole thing. It's just this one, we didn't really expect someone to be that unlucky and get like the minimum in all three tasks.

Um, but I mean, that signed up for it. We told them like in the advertisement, like it's not like this was hidden, but yeah, it still always feels a bit bad. But yeah, so it's, um, okay. [00:24:00] I'm, I'm looking forward then to, to, to, once, once you add that to the data bank, because I mean like the, so I had to look at it, like briefly look through it and it seems like, you know, it's, it's set up the way you'd want it to be set up that you, you know, you do five clicks and then you've got your data.

Um, so that would make everything so much easier. But like, was your task literally for the last year to just code studies or what? Like,

Giuliana Spadaro: no, that, that was my task for the very first year. Um Okay. In which, like, I transitioned to this new job and to be honest, it's uh, okay. It's a, it's a tedious task and it's a, it's very detail oriented, but it, it gives a lot of benefit.

Like, uh, okay. Talk about the, it takes alpha hour. You need, you need to look a lot of, um, to a lot of detail. It's more than this really, because you get really [00:25:00] into the, the specifics of the, of the experiment in a way you, you never had a chance to, like before we were saying that, um, you know, maybe sometimes where when you do your PhD it's hard to read a lot of paper and to really, you really need to be efficient with that somehow.

Uh, but if you read a lot of paper just to annotate them and to understand what was going on, you really get, uh, some, a level of de detail you will never expect.

Benjamin James Kuper-Smith: It's funny. You must be like, when it comes to experimental details, you must be like a complete expert in terms of like, knowing not necessarily every detail, every study, I think, but like, just knowing what people have done before, eh, you

Giuliana Spadaro: must, uh, it's funny because you start, uh, knowing people from their design and, uh, you start, or for example, some, this is a funny, uh, funny story, but sometimes there are some papers that I really like to, uh, to code and to, to annotate, uh, like, uh, of some specific research group.

I always thought, yeah, one day I [00:26:00] really like to end up at work with them. That's the way it should be done. So I start really these, uh, these little fandom and these little connections. Totally unilateral though.

Benjamin James Kuper-Smith: Yeah. I mean, do you plan on. Like when you think about like the rest of your, not the rest of your career, that's very grand term. Uh, but like about like what you want to do kind of next is for you, like the whole, are you gonna use this kind of fairly detailed knowledge of experimental methods more, or is um, I dunno, like do you see yourself as like experimental mainly, or,

Giuliana Spadaro: um.

Yeah. So I definitely like experiments and, uh, I think, uh, I will continue to perform more in the future, although one, uh, drawback, let's call it in this way of working with such a project is that is so big and so long term [00:27:00] oriented that everything else really at some point feel like, uh, really minor. You know, like, uh, if you, if you, if you make a contribution, if you really make a contribution in, uh, in, in the field, uh, then uh, coming back and scale down to perform individual studies that maybe are not gonna make a revolution, the field, it's, uh, a bit strange.

But, but, but I really love experiments, so I'm gonna keep on doing more, but I also got this vision for long-term things and long-term projects. So basically all the projects I'm involved right now, uh, been like two or three years in the making are just super long things. Uh, yeah.

Benjamin James Kuper-Smith: Do you like that kind of work?

Um, like studies that really take their time or like, just as a, as a comparison, like, I really noticed like, uh, for what I consider doing like animal studies and that kind of thing, um, 'cause I'm also like really interested in neuroscience and, [00:28:00] but I realized, I mean, I never really applied for it, but I realize now that I'm just not suited for something where you, you know, you have to train the animals for months then you, you know, like these studies that just take like three years to do or something.

Like, I'm just not suited for that kind of pace. Uh,

Giuliana Spadaro: yeah, I think, uh, I think it takes a lot of patience and perseverance, so it really needs to fit with the way you approach, uh, work and with your personality too. So I realized I, I like that and uh, yeah, there is not much of a problem for me except maybe in some.

I mean, in some specific times in my life, I wish just I had more things in the pipe and, or they're just, uh, faster but still like, uh, for most of the time. Um, no, it's, it's okay. I, I like it. And I think, uh, they're gonna be rewarding in the future in terms of the contribution that they can make.

Benjamin James Kuper-Smith: Yeah, I mean, like if, you know, I dunno, I've, I [00:29:00] haven't used this really yet to like, do a proper analysis, but like, if it works well then, you know, you've, you've created a framework for not any metaanalysis you might want to do, but.

You've like facilitated for so many people how to, how to do these kind of analysis. And I mean, as a side point, I also realized it's a good search engine, right? Like when you go through a, you know, I usually look through Google Scholar because that's just by far what gives me the best results. Um, but like, for example, for this lost gain thing, where I really wanted to find studies that looked at this and because I thought someone must have done this.

And some have, but not really the way I intended to or I intended to, you know, I went through like all the database and it was like a huge effort and still like I just in two minutes looked through your gain loss thing and found like three studies I hadn't seen until then. Even though in theory I'd gone, I thought I'd gone through all this.

Of this databases. Well,

Giuliana Spadaro: that, that's very encouraging to know, but, uh, it's true. Like we, we al always give like a, a big fo a huge focus to the meta analytic side of things and, [00:30:00] uh, to possibility to perform research synthesis. But the search engine, and even just using Coda to search to screen, uh, for papers, it's super helpful.

Like there are so many things you can do, especially as a, as an early career person, as a person who really haven't planned thousands of studies in, in the career. I think Koda is super valuable. There are really many things. Uh,

Benjamin James Kuper-Smith: yeah, I mean, what I find interesting and I dunno, like whether something like Coda exists or something like that, um, but, but Coda being the corporation data bank.

Um, but the one thing I find interesting as an idea is that because you. Have already coded all these variables and you are, seems like you're adding more as you know, uh, in the future. It almost seems to me like it, it works as like a, you can almost, like for many things that you're studying, you could make meta-analysis a part of your standard paper.

Like, you know, you can almost, like for me, for example, let's say I [00:31:00] do this one study and then I can add a thing at the end to it. I dunno how much work it is. Maybe it's, maybe it's huge, huge amount of work still, but it kind of seems like, you know, back in, you know, without, without Coda you, it's a whole new project to do this thing.

Whereas now it becomes like something smaller on the side almost, it seems, I dunno whether

Giuliana Spadaro: it works that way. Yeah, it, it look that's, uh, it looks like, uh, pretty much like all the studies, like, uh, the experimental studies that also in the paper report, let's say some analysis of survey data. Because you can, uh, like a World world Value survey or European social survey, it is not something that was collected firsthand, but it still like, make, uh, make, uh, can make the arguments stronger and provide a cross validation of what you wanna Yeah.

Claim.

Benjamin James Kuper-Smith: Yeah. I mean, have you, I mean, are you still working mainly on Coda then, or are you doing experiments right now, or what are you

Giuliana Spadaro: Um, no, right now, like, I'm not really [00:32:00] focused on, uh, annotating, I'm not annotating any new data, data at the moment because it was, uh, like really, really intense. So I'm, now, I'm transitioning to some, uh, research project I can, uh, I can do with this data.

So for example, I'm working on my own meta-analysis on, uh, cross-cultural variation incorporation and, and, uh, and on some other, some other experiments, uh, they're not really related to. To code A per

Benjamin James Kuper-Smith: se. So you, you're not like, you know, doing that thing where you almost, I mean, you could almost use it as a, I don't know.

I was about to say like, you can almost use it as a pilot for the study you want to do, but then again, if you already can do a meta-analysis, I want, I wonder whether your new experiment's gonna add much if it's already coded in there. I don't know.

Giuliana Spadaro: Um, what do you mean exactly?

Benjamin James Kuper-Smith: Well, I was thinking like, if there's already so much data out there that you can do new meta analysis, do you really need your new study?

Like, maybe, maybe. No, I guess you can. Yeah. Like for, for, for [00:33:00] my study, the data is already in your database, but no one's actually looked at the question I'm interested in, so it would still work. Do you see what I mean? Or,

Giuliana Spadaro: I think it, it still has value. It really depends on, uh, on, on how you exactly want to frame a reporting, a meta analytic results in your pa in your paper.

It really depends, but I don't see them as a ex a more exclusive, uh, things, to be honest.

Benjamin James Kuper-Smith: Yeah, no, I mean, I'll, once you've got your payoff matrix coded in there, I'll definitely see whether I can do, make something of it.

Giuliana Spadaro: But

Speaker 3: you're, you're very welcome, uh, to join, uh, our task force and, uh, and, and annotate some type of metrices, uh, with us.

Benjamin James Kuper-Smith: Uh, let's say this, I'll say this. I am, uh, I'm very happy to add our studies to it.

Speaker 3: Okay. Okay.

Benjamin James Kuper-Smith: If that's, I mean, so this is an next point I wanted to ask actually, is that you [00:34:00] mentioned, I think in your, in your pre-print briefly somewhere that the idea is that. New papers are added to it. Um, I'm assuming from your perspective, ideally by the authors, so you don't have to do it.

Um, but no, this is, I think this is a great idea, right? You have something where you, you create the framework that hopefully is easy enough for researchers to use, so they can just add new studies. I mean, that would be fantastic for you, right? You'd have hundreds of new studies every year just being added.

Giuliana Spadaro: I can, uh, I can already anticipate something about that. We're working already on, um, the implementation of such a feature in the data bank. So we're working on including a new tab in the, in the code, uh, research platform in which people can add their own studies. Of course, following a little bit, uh, the, the treatment structure and the data model that we, that is underlying our, uh, computations.

But of course we're gonna provide instruction for that. And, uh, this is already in better version, so we're testing that now and we plan to release, uh, pretty soon. And, okay, [00:35:00] as I said, of course this is gonna alleviate, uh, a lot the work of, of, of the people who are gonna annotate all of them. But the, the very cool thing is that if the authors can, uh, annotate themselves, the study they perform, they're gonna provide the most accurate information.

Like really the, the, the best information. And yeah, and I think it has a lot of value. And another thing is that the studies that we were able to include were publish studies that we were able to find online through the popular search search engines. But there are many more, uh, like out there in, uh, yeah, there are, there are, they remain unpublished somehow, and, uh, they deserve to be represented.

So that's a way to do that. So you can all, we are, we're planning to welcome really unpublished studies and file draw studies and uh, thesis and really to be able to represent as much as possible.

Benjamin James Kuper-Smith: Yeah, that's a really cool idea. Like even if [00:36:00] so, but like if you have a file draw thing, how do you guarantee that people are putting in the accurate information?

Like if it's not, you know, because if you have a paper you can just check the paper and see like, oh yeah, they did it correctly. But let's say you have someone who hasn't used code yet and they say, oh, I have this file draw study. I put it in, and you have no way of verifying it from a paper or something, or you just have to trust them or,

Giuliana Spadaro: so everything, we're planning to review every, every entry and every new record, um, by our.

Our editorial board of the of Coda. Uh, but of course there are some details that we can't, uh, we should rely on authors. So let's say if the, if the people report the, the participants in the study, were playing a prisoner dilemma game. Even if we review this content, we have no way to say that in the study, whether we never, that we never read anywhere they were actually playing another game.

So these information is gonna [00:37:00] be, uh, we should trust the authors to report like accurate information, but uh, from our side, we're gonna, um, make, um, the process of adding the information in terms of defining the treatments, defining the variables, or fitting our data model as much as possible. So we're really gonna help for that.

Benjamin James Kuper-Smith: Yeah, I'm just like thinking out loud now about like how to make this easier for you. Because yeah, as you said, like you probably have to like check papers so people don't just put in nonsense and just ruin your entire database through poor data quality. But would you like have, I dunno, give certain authors if they've already done it two or three times and they seem to know what they're doing, like giving them like a, not a certificate, but like a green card to just like put in their data because you know, they know what they're doing?

Or are you still gonna intend on actually checking through everything?

Giuliana Spadaro: You mean like introducing some sort of reputation system?

Benjamin James Kuper-Smith: [00:38:00] Not exactly. I mean, but you know what I mean, right. Like the kind of, you say like, okay, this person has put it in like three times correctly or something like. They seem to, we don't, you know, it's kind of worth wasting it.

I,

Giuliana Spadaro: yeah, I understand what you mean. No, we're gonna basically give the, uh, uh, ask parti, ask authors to also leave their, like, contact information. So if we see something that is, that does not add up, or if there's something that we would like, uh, to follow up with authors, we're gonna contact them and, uh, just, uh, have a one to one conversation to, to clarify.

But yeah. Yeah, no green card. I'm so sorry. The incentive structure.

Benjamin James Kuper-Smith: Yeah, you could put it like in your cv, I got like, you know, I've got a, I've got a bachelor's, a master's, a PhD, and I got a certificate from the coder

Speaker 3: or some, uh, cool stickers, uh, in your laptop.

Benjamin James Kuper-Smith: Oh yeah, that would be very good. Um, I'm not [00:39:00] joking.

That would probably like increase like, the amount of studies people put in, but like 50% if they could have a cool sticker. Um, yeah, but is, but how does, do you know how like Daniel, so like, you know, it runs through him, right? Because he had the ERC grant or whatever it was to, to do this thing. Like how does he plan on doing this long term?

Because it sounds like even, you know, if you want to add new studies, it's gonna be at least some effort involved for. Well, however long he wants to keep this up. But, you know, grants are not unlimited. So like how does he plan on ensuring like the longevity of the project?

Giuliana Spadaro: Okay, so that's a, that's a very good question.

So even, uh, so as you said, the, the project was originally funded by the ERC starting grant. It was awarded to them B and, uh, the people, many people worked to make that possible and to annotate data. And, uh, many of those people were actually volunteer and they really like, saw some, [00:40:00] uh, kind of benefit into the project.

Uh, they saw that they could make a difference and some of them, uh, uh, really wanted to get closer to the data, to, to do their own project with that. Like, uh, even annotating more data that didn't end up in the cooperation data bank, eventually so many people, uh, really, uh, contributed to that. And we have some people that are starting to reach out.

To keep on co on contributing more in some capacity. So like all those people, they didn't really have to annotate thousands of papers, was maybe what, what they had the time for was like a summer, uh, summer pro, um, project or something like this. So we are trying to acquire some other external funding and, uh, we're applying to some grants to, to have it, to keep that running.

So we're gonna base on the money from these grants that hopefully will come and from, uh, some other work from people who believe in the project right now. But we're also trying to invest to, [00:41:00] um, in, uh, some ways to automatize some of the processes because like right now, at the moment, there is nothing that is, uh, comparable, uh, in terms of data quality to the information that an an domain expert annotator can annotate.

So if the, if the two of us read a paper, we're, we're always gonna perform better than some machine in extracting the same information and understanding what is going on in the experimental game. But there are some tasks like the, uh, systematic search for studies that can be automatized or at least semi automatized through some methods like, uh, machine learning, let's say, that can learn from our previous work to, uh, and, and support us to retrieve like a helpful papers or a relevant record, let's say.

So we're investing in that and in trying to understand how much we can, uh, how can I say? Um. How ma um, whether machines can help us like to [00:42:00] increase the efficiency of the entire process.

Benjamin James Kuper-Smith: Yeah, I mean, I guess like relying on other researchers, I know like, to some extent, of course, relying, you know, relying on other people's help is maybe not the greatest idea.

Should only on it. Right. But in a way also it does seem very, I could imagine it working from the sense, you know, like I, I have like, you know, with the payoff metrics thing, I have all these payoff matrices so I can, you know, like I can just like take the stuff I already have and make it publicly available, right?

That's kind of what it is. And that's of course like. I mean, that's the whole thing behind, like open science, right? That you, you have your methods and made them available to others. So I think that motivation might get you pretty far, actually.

Giuliana Spadaro: I mean, it's a, it's a public good, uh, management problem eventually.

Oh, there we go.

Benjamin James Kuper-Smith: Yeah,

Giuliana Spadaro: there we go. I mean, if you see some benefit, for example, in, in the data and learning something from the [00:43:00] data, uh, that was already annotated there, but you need something else like, let's say the payoff matrix. Uh, nothing prevents you to like, to go through some, uh, some mattresses and to code them and maybe reach out for some further support.

Like, uh, you can, um, uh, share this data with us. We can, uh, provide some, some support with some further notation, and eventually this is gonna result for, uh, in a better outcome for you because they're gonna get more data. But for the collective too, I mean, everybody benefits at the end and, uh. That's a bit of the model that we had in mind.

We think people are gonna benefit from that. Maybe they want to add to this knowledge. Uh, yeah.

Benjamin James Kuper-Smith: Okay. So if we, I find it so annoying almost that when you do something with like cooperation, social interaction, everything becomes like a game theoretical problem. Um, but let's, let's go there for like a minute.

So how do you create the incentive structures though, that it doesn't become a prisoner dilemma, but that it becomes something more [00:44:00] cooperative, right? Because like in setting up Coda, you could set it up so that it's, you know, deadlock or whatever, or, um, you know, in terms of like people helping you, um, have you, have you thought from, from a game of theoretical perspective, how you can ensure that it becomes something where mutual cooperation is the, the Nash equilibrium or something like that?

Or. Uh,

Giuliana Spadaro: I mean, uh, we had a, we had a lot of conversation about that and, uh, to what extent, uh, we wanted to introduce, let's say, some kind of institutional control and, uh, uh, and eventually, like this leads to a lot of cascading decisions about how much you want to control. And we decided that, uh, we can't control that.

And, uh, that eventually, like, it really, uh, and, uh, it, it really depends on individual, uh, on the individual motivation. And, uh, there is not much you can, uh, you can control about that unless you start really controlling every single step and really constraining [00:45:00] people action. And one, what we wanted to give was like full freedom and full ownership of the data and, uh, full flexibility.

Uh, because when you start, uh, controlling and constraining, uh, it, uh, it really goes a, yeah.

Benjamin James Kuper-Smith: Um, so I mean, you mentioned, you know, if people have questions, whether they can contact you. So is that also, let's say you want something, a new factor to be added to coder, you know, like for example, me with the payoff matrix or something.

Um, and imagine you don't have a podcast where you can just talk to the authors. Um, do you just send you an email and say like, Hey, I, I'm interested in the payoff matrix. Are you planning on adding it or whatever, or how does

Giuliana Spadaro: Yeah, so. I mean, we're not like a company. So even if you use, uh, the contact form that you find in, uh, Dakota, the Corporation Data Bank website, uh, then and I are are gonna read the email.

So if you're gonna send an email through the form or you're directly con in [00:46:00] contact with N and i, we're very approachable. And, uh, and we're gonna figure out something together and see how the specific request, uh, can meet or, uh, whatever is going on, like, uh, from, uh, from our side and, uh, if we can meet in the middle and if we can help each other and start a collaboration.

So it's all very spontaneous. Uh, yeah.

Benjamin James Kuper-Smith: Yeah. Yeah. Okay. Then I have my second request after Payoff matrices is, um, are you also planning on adding other two by two games? So you have Prison Salama and then some of, you know, you have some studies where you people use a Prison Sal Dilemma and maybe chicken or Stack on or something, but you haven't, uh, you know, coded it for the, I mean you don't want all two by two games, but let's say the symmetric ones or something that you, you know, also include studies that only use Stack Hunt or something like that.

I dunno, it seems to me like if you're mainly using PRIs Cinema, right. And then if those studies also include other games, you have them too, but you're [00:47:00] not as far as I can tell, at least. No.

Giuliana Spadaro: Yeah, yeah. Your intuition is correct. So if, like, let's say a study included multiple games, we only annotated those that, uh, were relevant according to the inclusion criteria.

So of data bank, which are like, uh, the game being a prisoner dilemma, a public good dilemma, research dilemma, and there is a dilemma or some slight variation of those. I mean, so in this sense, like a chicken game wouldn't be annotated by our collaborators, and so you, you're not gonna gonna find it in there.

Hmm.

Benjamin James Kuper-Smith: I'm pretty sure I did earlier, like I looked as one of the variables was like game type. Game type. And then it was prisoner, but it was also chicken to select.

Giuliana Spadaro: Yeah. Yeah. So you, you can select, uh, you this, if you found a chicken dilemma in the, a chicken game in the, in our platform, you couldn't find it, uh, while filtering out based on game type.

So if you, if you are gonna filtering like what's [00:48:00] the, what's the game that has been used, you're, you're not gonna find that only gonna find these games. But if the article report and effect size, let's say, uh, that com, which summarize the, the difference between cooperation in a a prisoner dilemma setting and in a chicken game, we have a, a variable to Okay to, yeah, conceptualize a chicken game in our ontology.

So you're gonna find these effect size, but you, you can't really select studies based on employing a chicken game because there are simply not there. I don't know if I explained it. Yeah. And you're

Benjamin James Kuper-Smith: not planning on adding that or

Giuliana Spadaro: No, it's actually like a way, um, a point on our, on our agenda and we really want to expand the sort, the sort of data you can find in there.

And adding more games is definitely one of the, of the next, uh, challenge. And yeah, if we're still seeking for some collaborators who want to get on board and, uh, [00:49:00] and, and support with that,

Benjamin James Kuper-Smith: by the way, is it really annoying that like you have this like great platform where you have like all this data available and, and like the first thing people do is they say like, can you also do this thing?

Speaker 3: No.

Benjamin James Kuper-Smith: Like, why don't you have that thing?

Speaker 3: No, we, we do, we take it as a sign of enthusiasm. We like it.

Benjamin James Kuper-Smith: Yeah, that's true. It is enthusia. Yeah. Yeah, exactly. I wouldn't ask you if, if I, if I thought it was a waste of time. Um, obviously I only asked because it seems really cool. But when you say looking for collaborators, for example, let's say for other game types or something, what does that exactly, what would a collaborator do there?

Giuliana Spadaro: So the way we organized the collaboration and uh, people who worked, uh, who worked with us in the past was that they all went through a training session, uh, with me, um, in which like they were really get acquainted with the, with the data and how the data look like and how our annotation system look like to be able to be independent in, um, [00:50:00] in, uh, annotating new information.

And, uh, this is really a structured procedure. It takes, uh, it takes some time, but it really makes people comfortable in saying, okay, uh, I can do that. So based on then, what, what was their availability? Um, they were assigned, uh, a proof of, of studies of papers to annotate and, uh, they were always in contact with me if they had any questions or something.

So, yeah.

Benjamin James Kuper-Smith: Do the people require like a certain level of, um, education or can it be like, I don't know, let's say you have someone who wants to do a master's project or whatever, can, is that also fine or do they have to like already know the field or have experience in it also?

Giuliana Spadaro: No, we had, uh, um, of course, uh, it's, it's required to have some, uh, familiarity with, uh, with the field and with economic games in general.

But we also had, uh, some, uh, collaborators from, um, that were doing their master. And uh, some of them were, uh, actually interested in, uh, doing a meta analysis on cooperation eventually [00:51:00] and super motivated to learn. So yeah, nothing really prevent anybody to learn about that. Even with, uh, very little familiarity, we have our reference and, uh, they go through this training and there is a constant, uh, communication.

It's very open and uh, I find it, uh, especially beneficial for people, uh, early in their career because, yeah, because of the benefits I was mentioning before, but also because. It really gives you, um, also a perspective on writing and or on what you want and how you want to communicate your results in a, in a, in a research paper, let's say.

Because you know where to, uh, that when you read some results sections, they're very hard. It's very hard to locate, uh, the information and it's a bit annoying. So you, you really want to be clear when you write your own research section and, uh, yeah.

Benjamin James Kuper-Smith: So yeah. At, in the worst case scenario is that you gain an appreciation of well written [00:52:00] methods section Oh

Giuliana Spadaro: yeah.

Benjamin James Kuper-Smith: Methods. I mean, that's. If every method section was well written, that would be, that would be pretty cool.

Giuliana Spadaro: Yeah, I mean, uh, I think, uh, there are like a couple of, uh, of things that in, in which Coda can help for study reporting. So one thing would be that if you get familiar with the variables that we annotated that, for example, that are related to the study characteristics, so all the features of the experimental design and you really know, uh, what they are, uh, then you can check whether in your own reporting, uh, of an experimental, uh, method section, you're, you're really covering all those points because, uh, authors give many things for granted and keep a lot of this knowledge implicit.

Like, for example, for me, it's very hard, uh, sometimes to find, uh, information about the incentive in the game. Sometimes people don't really report whether, uh, participants were paid for that or not. So it's [00:53:00] very common in econ uh, econ papers and in psychology papers, let's say run in online setting, but in the lab is really not, uh, not super common.

And it should be.

Benjamin James Kuper-Smith: So you mean like whether people were paid for the responses in the game or whether it

Giuliana Spadaro: was hypothe? Uh, no, even if they, if they received like a show up fee, uh, for participation like a flood, uh, because people think maybe that's not interesting. Maybe that's not relevant, but it's actually good to be explicit, you know?

Benjamin James Kuper-Smith: Yeah. So like studying whether, oh, okay. I thought you literally meant, do you, sorry. Do you also have the variable, whether the reward, the pay, the payoff was hypothetical or real? Yeah. Yeah. So.

Um, I wanna say, oh yeah. So the, do you, I, and I know you have tutorials, um, on the website, but beyond that, for example, one thing that, [00:54:00] something I'm particularly thinking of right now is that I don't know how to do imagine analysis. Like even if I want to do something, I'd act actually know, like what the steps are, how it works.

Is that also something you guys assist with or collaborate with or is it really, I mean it sounds like you already have a lot to do. Um, so, but you know what I mean, like, is that also something where you, I don't know, act as a platform for connecting meta analysis cysts with people just wanna run one or,

Giuliana Spadaro: um, I dunno.

So what, what you can find in our website is really how best to navigate the platform to get like the, the, the analytic outcomes, but also like to navigate all the other functions. And we also had, uh, um, a dilemma about, uh, okay, but what kind of information about meta-analysis per se we want to compare.

And, uh, eventually we end up, uh, deciding that a lot of people are, are doing that and are focusing their research on [00:55:00] developing meta analytic methods and meta analytic software. So, uh, this was not like our. Uh, there are like a lot of people that can do a better job, let's say, training in the meta analytic methods.

So, as an example, like all the, all our software on our, on our research platform runs on are shiny and, uh, and the competition that are specific to meta-analysis are implemented using the metaphor pa our package, uh, that that is developed by, um, amp power. So if you need to learn something about meta-analysis and, uh, the competition behind that, it's better just to be redirect to the people who work with that.

Like, uh, okay.

Benjamin James Kuper-Smith: Yeah. Yeah, I was just curious particularly because Daniel has, you know, he does have the expertise right? In running meta analysis.

Giuliana Spadaro: Yeah, of course. But yeah. But, uh, I mean, what, what we can really, uh. All we can do is, is talk about like meta-analysis, about [00:56:00] cooperation and the specific, uh, um, the specific decisions that underlie like a meta-analysis on cooperation because then, okay.

There is nothing like the general meta analytic method. There is nothing like performing a general meta-analysis. There are a lot of choices one, uh, should make and we can, we can be knowledgeable about that specific part, but I think there are like a lot of amazing resource out there on the general, uh, yeah.

Benjamin James Kuper-Smith: Yeah, no, it makes sense. I mean, like, I, I had that question when, you know, I wrote that question down before, like whether you guys help it, when what, and it made sense, but when I was just saying it, I realized like, wait, you already, you already have so much to do. Like you can't, like, it doesn't sound like it's even like just physically possible for you guys to also do all of that whilst you're trying to set up and maintain this state of Hank.

Yeah, no, no. So I, I completely understand that you, that you can't do everything

Giuliana Spadaro: well, but what we, what we, what we could do, and what we do have in our website is a list [00:57:00] of useful research. We redirect people who are interested in more specific, um, function, like a meta-analysis or power analysis or other things that you can do.

Benjamin James Kuper-Smith: Yeah, I mean, it's also like, you know, because part of what I was thinking, like in the, in the thing I was thinking of is that. It would be really cool to do to like just see whether this effect also persists in, you know, across all the other literature. But in a way, I did do a few controlled experiments and am I really gonna put in all the effort just to see whether, you know, it works in all the other contexts when I feel like I've, you know, I feel like I, my study should in a way already be pretty confident that the effect is there, right?

So it's a bit of a, like, am I willing to put in the effort to learn how to do run a meta-analysis just for something that I, I think should already, you know, I should already have the evidence for it, but.

Giuliana Spadaro: But I, I mean, uh, [00:58:00] you can learn, uh, some other stuff with the meta analysis. You can get like, let's say an estimate of, uh, how much publication bias is out there.

And this is not something that you can learn, uh, if you perform an individual study. So you get a sense of what's the effect in your specific, uh, set of circumstance that led you to that specific result. But you can't really assess like, uh, how much like the, the literature that is already out there is reliable and is gonna replicate, uh, in the future, uh, with your own study.

So there are some benefits also in the, in having a analytic approach instead.

Benjamin James Kuper-Smith: Yeah. Yeah. I mean, I'm actually also, I mean, this is the kind of slightly weird thing. Have you, so this is paper by, I dunno who it's by. Um, I can't remember, but I'll, by the way, I always put all the papers and stuff we talk about as a description in the, uh, in the description.

So I'll put it there. But there's a. Something that someone did recently, which is about like single paper meta-analysis [00:59:00] where they basically say like, if you have several experiments on the same thing, they have like, also like a probably also shiny apps thing where you can put in the, you know, you do a meta-analysis over let's say your three or four studies in your experiments.

So in a way I, I kind of plan on doing that already, but yeah, some are running a full metaanalysis just sounds like, as you said, it's work.

Giuliana Spadaro: I mean, that's a different, that's a, that's still a different thing because if you run an internal meta-analysis, meaning like a meta-analysis of the set of studies that you conduct, you can get like a good sense like of the average effect that you get.

Like, it is like, if you want to summarize, uh, the, the finding on your paper of your paper in a single, uh, metrics, uh, and here you go, but still it doesn't, uh, add much to comparing that to the, to what's to the existing, uh.

Benjamin James Kuper-Smith: Yeah. Yeah. No, it would be somewhere, I mean, and in this case in particular because we did a few experiments and one of them, for example, wasn't, didn't show any differences that we [01:00:00] expected.

So that's the question, like, okay, you know, what does the, the cumulatively, what do the studies say? Yeah. Yeah. No, you're right. There's two different things.

See, I just keep thinking of new things you guys should add, which is very rude. But, um, again, you said you think of it as, uh, enthusiasm, but this is a genuine question. So like with, you know, I mean, like what you're doing is already a huge amount of work and already very cool. But for example, like with our study, we plan on putting, you know, all the data public, so you could actually have the trials, you know, every single trial of every person you could have.

I, is there any way of like incorporating that in the future into your thing, or is that just a completely different level of. Detail or, you know, rather than just having like a, an effect for a study, rather actually being able to include all the trials and that kind of stuff.

Giuliana Spadaro: Yeah. So are the, um, it's a, it's a different thing because, uh, what are [01:01:00] like the observations and the, the individual observation in a study that you can, that you collect are like, let's say the can be the participants or can be, uh, a group or a trial or whatever, but what we use as an observation in our data model is an effect.

So the way the data are, let's say even structured in an Excel file that can come up from your study and from the data bank is different. So we would still need like some kind of intermediate step to translate, uh, the former into the ladder.

Benjamin James Kuper-Smith: I mean, that would actually be almost very beneficial for the field, right?

If you guys just set a standard, like column one is this, column two, is that column three is this. Um, I dunno whether that makes any sense for very, like, you know, studies that can differ so much. But, um,

Giuliana Spadaro: I mean there is like a, right now a movement in science about how best to, let's say, to create codebooks and to, uh, share the data in a way that is standardized as possible.[01:02:00]

So there is some work that is moving in that direction and I think, uh, I think it's pretty cool and, uh, I think we can use that as well. But right now, what we can give as a, as a, as a guideline maybe could be like the definitions, uh, of the, of the concept. Like do we have in our ontology for example, uh, like that, that redefine some kind of rough structure of all the variables that have been studied in relation with cooperation.

But regarding like how really to store the data and uh, the data model, uh, yeah, that's a different entire different world.

Benjamin James Kuper-Smith: Yeah, no, as I said, like it sounds like you've got enough on your, on your plate as it is. Um. By the way, do you have like specific people who just do the programming of the website or the programming of the database or whatever?

Or is it literally just, I know you and I don't know exactly who's involved. [01:03:00] I mean, it's, it's on the website, but it's,

Giuliana Spadaro: no, we have a big team. And, uh, indeed you can, uh, like check all the names and see all the faces in on our website. But I was, our team is, uh, I can describe it as, uh, roughly dividing domain experts who are like, really the people who work with cooperation and, uh, who are knowledgeable of, uh, cooperation studies and, uh, computer scientists were the people who supported us with, uh, um, and everything to relate and pertains to the knowledge representation and to the actual implementation of the infrastructure.

And then there are some people in the middle of the, of, of those, uh, fields like, uh, my colleague, uh, Simon Columbus, uh, which was, uh, really super, super helpful in developing the shiny app and, uh, and the r script that is behind the, the research interface. So not all of us are involved with all the tasks.

And that makes sense. That makes complete sense.

Benjamin James Kuper-Smith: Yeah. [01:04:00] Yeah. By the way, what, as a kind of. Maybe final question about the data bank. Um, like how was, so first you were hired to work on that project, right? Like you knew you were gonna work on this. Okay. Yeah. So how was that, how did it turn out differently than you expected?

Like, you know, you went in with a certain ex assuming a certain expectation of what the work was gonna be like, and then, yeah. I'm just curious, like working on this kind of big, like creating a framework kind of project like. I dunno, how was it?

Giuliana Spadaro: To be honest, when I transitioned from my PhD, there was like the time, uh, in which you ask what do you wanna do next?

And, uh, how you want to bring, uh, your research or career to the next level? And, uh, there, there came this opportunity and, uh, this was, uh, described me very honestly by then at the beginning as an appointment that was super focusing on, on the adaptation and on the data correction and on the coordination of this big group of people.

Uh, so I knew exactly what, [01:05:00] what I was, uh, about to face and I actually was excited. And I also thought that's a nice way to, uh, keep thinking about what, what you wanna do in the, in the meantime. But then it really, um, became something bigger because. If you like, uh, turn your attention out of the Excel file you're working on and inot taking, um, number after number, you see what is, uh, what it's growing.

So I started thinking about my own meta analytic project and my own vision and how, and I, I learned that I could make a difference and I could shape the way, the future line of research somehow and, and influence, uh, influence the field in some way, in some positive way. I thought. So that was, uh, that was unexpected and was super rewarding.

And to be honest, at the end of the day when, uh, when I was tired from all the super detailed annotation, uh, that, uh, really kept my motivation up, but my [01:06:00] expectations were like, the job was described in a very realistic way. And, uh, but it was much more than that.

Benjamin James Kuper-Smith: How do you think about, uh, like the kind of work you're doing because.

So one thing I was struck by when I thought about like what it would be like to kind of set up this project is that, how should I say? Like, obviously it's a very, you know, I agree. Like this is a really, can be a very impactful kind of project, right? Where you can really help lots of people do lots of really cool things.

But in a way, it sounds to me that the, it doesn't have this kind of almost scientific kind of, should we say like it's almost not doing science, but creating a framework for other people to do science. Is that something that, uh, if I, uh, if I got that correctly, um, then I don't know. I'm not really sure what my question exactly here is.

Um,[01:07:00]

and is that, how do you think about it that way? Or do you just think about in terms of, in terms of the contribution or something?

Giuliana Spadaro: So at, at the end, uh, at the end, this, this, this entire project turned out more, uh, way more meta sciencey than, uh, what I expected it to be. But, uh, during like my, my work, I had, uh, uh, the possibility to develop some ideas.

They were really like, about cooperation and that could help to answer some questions related to, to cooperation, such as, for example, whether cooperation varies across societies or which are like the cross cultural factors that can account for the variation that we can, that we observe. So I start thinking about some project to answer to some question, and this data for me, uh, were like the way to achieve that goal.

So it, it was just that I didn't conduct, let's say, experimental research per se, but it really helped me. I. I think, uh, at the end of the day, I could also give [01:08:00] like, uh, some kind of theoretical contribution and not only like a meta scientific contribution. And so all the other people who worked in the project, like we, our lab has many meta-analysis, um, going on at the moment.

And many of these people were involved with the notation of these studies, but this really didn't prevent to ask interesting question, um, about cooperation per se.

Benjamin James Kuper-Smith: The lab must have like a lot of meta-analysis coming out soon, right? Like, you're the first people to be able to use this database, you know how to use it.

You've got your questions, you can just run with it, right? I

Giuliana Spadaro: mean, that's a little bit of a, of a competitive advantage, but still like, uh, the, the project we're working on that are not, um, all of them out yet, uh, are gonna be based on the same data that is out right now. So it's not really competitive in this way?

Benjamin James Kuper-Smith: No. I mean, I didn't even mean it in that sense. I meant it more like in terms of like. I imagine, like for someone like Daniel, who as [01:09:00] far as I can tell has, you know, largely been doing meta analysis for most of his career, this must be just like fantastic, right? Because he now has this tool that I, I, I assume he wish he had this 10 years ago.

Right?

Giuliana Spadaro: And you're assuming, right? And we, when we were testing, uh, the interface, like when it was, uh, developed first, like he was so excited. He run like hundreds of meta-analysis in like one hour and he was like, oh, I'm running one meta after the other. So like, that was very fun, but I could see enthusiasm and uh, yeah, it was a bit

Speaker 3: wild with, uh, with the meta-analysis.

Benjamin James Kuper-Smith: Yeah. Yeah, he is probably gonna have like a phase after that, like when he has to adjust, like, you know, when he did like all the meta-analysis in like one year and then he has to like calm down a bit when like he's already like done everything he wants to do.

Speaker 3: Yeah. Well, uh, from what I learned there is always gonna be more, so I can't wait to see [01:10:00] what's coming next.

Benjamin James Kuper-Smith: Yeah, definitely. By the way, how, how long are you still working on that project? Like I'm assuming your contract is not unlimited.

Giuliana Spadaro: Uh, I started working, uh, in, um, at the end of 2017 and, uh, and then I had another appointment and I'm, and now I'm back again, uh, at Devo in, uh, in the, in the cooperation, uh, in the am cooperation lab.

Uh, but although I had, I still have some responsibilities with the data bank. My new appointment is, uh, yeah, it's not entirely oriented to that. I'm also doing other research, but of course, as a, as a co-director, I'm still, uh, I'm still involved.

Benjamin James Kuper-Smith: Yeah. The, the, the co-director still have to do something, but is, um, okay, so now it's kind of, you're working mainly on other stuff, and this is, you know, using your knowledge of the field and the.

Database itself. [01:11:00] Um, your, sorry. The data bank, um, is, so it's more like a supervisory role now rather than

Giuliana Spadaro: Yeah, I'm, I'm, um, I'm less hands on with, uh, with the data. Uh, I'm not annotating any new stuff, uh, but I'm coordinating and supervising, so that's the role right now. But I'm sure that during the next round of annotation and, uh, when it, when it's gonna start, this can probably change.

Uh, I also see some benefit in keep on annotating stuff. Maybe not full-time, but, uh, yeah.

Benjamin James Kuper-Smith: Yeah. So what, so what are you working on now or what's kind of next for you if you know? Obviously, you don't have to say all your secret ideas, but, um, in, in the broad strokes,

Giuliana Spadaro: no, I'm, I'm really focusing on the meta-analysis right now because it has been like, uh, two years in the making.

So for me it is a really time to wrap it up and, uh, send it [01:12:00] out there. And really I can't wait, uh, to share the findings with the community too, and not just staring at them myself alone, but as other, we have some other projects that are related to the new grant that, uh, Dan, uh, was awarded because he got, um, um, another year CC grant recently.

So we're starting, yeah. Super well done. And we're starting, uh, to really work on, uh, interdependence and, uh, using experience I method and bringing that to a more global, let's say, uh, level.

Benjamin James Kuper-Smith: Can you say something about that? I'm curious what, like the experience sample or you don't have to if you don't want to.

Giuliana Spadaro: Um, no, uh, it, it is still in the making, so there is not much I can, uh, I can, um, I should be able, I, I'm not, I'm not able to share more right now because we haven't started yet. So.

Benjamin James Kuper-Smith: I see, I see.

Giuliana Spadaro: Simple as, as that,

Benjamin James Kuper-Smith: uh, you mentioned [01:13:00] earlier a few times kind of that you like the contribution to the field that you're doing with Coda on a kind of long term career skill, do you have like a contribution you want to make, uh, a specific thing or is it kind of just, I dunno, like how do you think about that kind of like long term career and work you do?

Giuliana Spadaro: Um, I. Uh, to be honest, I haven't been so strategic to think about, uh, like the long-term horizon with that. Like, not at all. But one thing that for me is very rewarding already is, uh, that my reputation changed a little bit. Not, not like changed, but if some people like read my name or anything, like out there in the web, like they think, oh, those are truly cooperative people who are working, that are working for the common good.

And, uh, to me that's, that's amazing. That's just the best thing they can learn. That can happen. Yeah. Yeah. From a career [01:14:00] perspective. But then, uh, I don't know. I think, uh, the main long-term, uh, benefit that I see is gonna be really changing the mindset related to the project and what you wanna do. Like, uh, now I really feel like with the next grant I will, uh, apply to, I want to make, uh, a different.

And also I want my work to be helpful for others and not for myself. That, that's like a take almost that I got for the, from this project.

Benjamin James Kuper-Smith: That's, that's pretty good. I mean, no, I mean, like, I guess it is like an inherently kind of community, community thing, right? Where you, you take all the studies from other people, you make it available for everyone.

It's, it's, if I is very much, seems like a kind of, you know, from the people, for the people kind of thing, um, without wanting to, you know, be too grande. But yeah, no, I mean, I'm looking forward to kind of trying it out, seeing whether I can, you know, do the things I wanna try out with it. [01:15:00] Um, so you mentioned briefly like the, the workshops.

So as I said, this one won't be. Uh, public for that one. But are you, is that kind of something you're doing quite a lot

Giuliana Spadaro: or? So we're planning to, to do like a series of workshop, like, uh, at the beginning. Uh, there may mainly gonna revolve around navigating the function of Coda and really like, uh, showing firsthand and working with people to understand how to use the, uh, the platform in a way that can serve the research purpose.

Uh, but in the future, once like people are gonna be acquainted with the platform and the possibility of the data, we really want to start like some, um, some kind of meetings to think about research ideas and uh, to create, uh, research networks and even, um, to think about possible, uh, avenues for these, uh, for the data bank, what we can add and, uh.

And, uh, yeah, because people have always like, uh, good ideas [01:16:00] and, uh, thoughts like, Hey, why don't you do that? And maybe it's something we never thought of because we were already refocused on what was on the table. But yeah, so we're gonna have this opportunities and we, every time we, we tried, we, we plan to disseminate them in like, um, channel, like Twitter or some mailing list, uh, uh, like, uh, yeah, some, uh, relevant mailing list.

But if you check like our website, we have a new section. Uh, so just checking out like every now and then, because some events can be on the mailing.

Benjamin James Kuper-Smith: Yeah. But you, do you have a mailing list or not?

Giuliana Spadaro: No, we don't have, um, a quarter mailing list. Uh, because we really didn't want to spam people and to be intrusive with that.

But I mean, if you're interested and you really want to. Uh, you don't wanna miss, like the next opportunity, just send us an email and, uh, we're just gonna keep your, uh, keep you in the loop for every, um, future [01:17:00]communication.

Benjamin James Kuper-Smith: Yeah, sounds good. So I'm assuming right now everything's still dig like online and then maybe at some point once, uh, once COVID is over, then at some point maybe in person, but I'm assuming right now it's online.

Giuliana Spadaro: Yes, right now it's online, but, uh, I very much look, look forward, uh, to transition to a, a face-to-face, uh, setting thing.

Benjamin James Kuper-Smith: Yeah. Are you gonna put those like online, the workshops, something like a, you know, for people who, uh,

Giuliana Spadaro: don't have time

Benjamin James Kuper-Smith: during the time putting it on YouTube or something? Is that

Giuliana Spadaro: more Um, I don't know about the next one, but definitely also because it's a bit harder to find the time in the day there is inclusive, uh, for all people, like in, uh, all the time zone.

So like, uh, with the time that we usually, um, we usually work with, like, people in Japan are not really, it's not really easy for them to join, so it's a good idea to, [01:18:00] to publish them. Yeah. So, yeah,

Benjamin James Kuper-Smith: that's good.

21. Giuliana Spadaro: Cooperation Databank, payoff matrices, and meta-analyses

Featured Posts

Read Full Transcript