Bitcoin Network Monitoring

Hi everyone, welcome back to Stephan Livera podcast. Today we're gonna be doing an interesting one on the Bitcoin network and observing what's happening. So joining me today is the, the famous, 0xB10C, or I believe now you've rebranded to B10C. So, welcome to the show, B10C.

Alright, thanks for having me. Yeah, B10C is, is a bit shorter and, rolls off the tongue a bit better, I think.

Right, yeah, yeah. And I was looking this Now you're just kind of, just for simplicity, you just,

you're just

known as B10C.

So somebody came up to me and said, "Hey, you're an Ethereum guy," and I was like, "Why?" And he said, "You, you got the zero X before it," and I was like, "Ah, I, I have to rebrand."

yeah, yeah, but yeah, I, it, I mean, it's, it's shorter. Yeah, okay.

Well, let's, let's start just for listeners who have no idea who you

can you just, I mean, obviously only what you're willing to share, tell us a little bit about yourself. What do you see as your role in, you know, in this Bitcoin ecosystem?

Alright, so, so I've been working on, on Bitcoin open source projects for, for a few years, I think s-since twenty twenty-one full time, and I'm supported by different grant organizations, doing, doing my work. And I built a couple of monito-monitoring tools, for, for different parts of the Bitcoin network, and I think that's the, the main, the main thing I'm kno-known for. but I also enjoy like doing data analysis on some of the data I extract from this, or in general, some Either on the blockchain or you can find elsewhere. so for example, I looked at, what mining pools include in their, in their blocks and, what they leave out, and we actually found out that, for example, some mining pools, leave out, OFAC sanctions transactions, which we weren't aware of before, and that's something as a community, I think, that's important to be re-aware of. Obviously, mining pools can, can choose what they in-include in their blocks. but, I, I think it's for, for a community, it's, it's, good to be aware of since, one of the key points of Bitcoin is the censorship resistance, and if we found out like all pools are including some kind of, of transactions, then maybe the censorship resistance, part of the Bitcoin, idea wouldn't hold up anymore. yeah, and also lo- yeah, also looked at, for example, what, what the, Stratum jobs of the different mining pools, what they send out, and what they contain, and, found out there that, different mining pools, actually send out the same, Stratum work, so they actually send out the same block template or build on, on, on the same block template, and, or, this is mainly Antpool and, and some of their friends, I call them They might appear as different, mining pools based on the, the name they put in the, in the Coinbase transaction, but in, in reality, they are actually the, the same, the same mining pool or the, the same node that's created for, for the, for the block template. yeah, alright, and this is for, for, for the data analysis part, but I've also contributed to Bitcoin Core, build their, tracing framework and, yeah, diff-different, different smaller side projects here and there.

Now, I think one of the earlier things that Perhaps you became known for was, as you mentioned, the OFAC thing. So for listeners who aren't aware, it's like, this relates to like what's called sanctions laws, and basically governments around the world have sanctions regimes, and they might have a list. So I think in the US, it's called the SDN or something like sanction designated, whatever. Anyway, the point is, they've put some Bitcoin addresses on there, and as I'm understanding, this is some of your early work where you were trying to under-- like point out, are there mining pools who are not in Including transactions that are sent to the addresses listed in this sanctions list. So can you just ex-elaborate a little bit on some of your research there?

yeah, correct. So, so you look at, if, if a pool is sending or if a pool is filtering out transactions sending to these addresses or spending from these addresses, so, both ways. we, we don't, I actually don't know what's, what's like really forbidden there, and I, I'm not sure if it's hundred percent clear, but people, especially like people in the US tend to be careful, not to touch any addresses involved in like OFAC stuff, and, for them not to do it. And, if you want to detect, someone not, not, or like a mining pool not interacting with these addresses, you build a block template yourself, and if a new block comes in, you compare the block template you built to the template of the pool, and, you, you, you're going to get some mismatches here and there, because, mempools and, and different, different transaction relay and, and so on. but you, you're going to see See if, for example, one pool is repeatedly not including one transaction, or if one transaction is repeatedly not included by different pools. And if you, if it turns out this is like an OFAC sanctioned transaction, and you see this maybe ten times, then you have a high likelihood of, of this pool, or you can say with a high likelihood this pool is probably filtering out these transactions. but it's, it's not only for OFAC transactions, it's in general like interesting, what do many pools mine, which could- Also tell you, for example, on, on which version, of Bitcoin Core they are on and, and, and different, different other stuff, or maybe you, it can also tell you what, a mining pool includes, so, a mining pool might include zero fee transactions, which payouts, their mining pool customers, in, in these transactions. So, these wouldn't normally be included, but, they, they, they, they are including them.

I see. Yeah, and maybe there's kind of just special cases where they'll do this kind of thing where, you know, I think there's an infamous case where, I think it was, the F2Pool founder, I think, actually at the time, I think it was Luke Dash or was asking for like a donation, and I think- He paid him out of a, a Coinbase output, right? Like it was like a baller move that he had paid out of his own pool kind of thing.

Yeah, it could be, yeah, yeah. I also asked, the FPool, F2Pool founder while ago to include like a non-standard transaction. That's something else you, you're going to see, pools include. and I, I asked the, the, the founder to include a non-standard transaction, and he actually did it, so,

was a fun, fun experiment. Yeah Taproot discussion as well, because I mean, maybe even that term might be the wrong-- I don't know, maybe it's not the correct term to apply, but I, I think, I think it's filtering,

it's filtering, right? Because censorship would imply that's all, that, or that there is no way of this transaction getting through. There, there, there are the maybe smaller pools or solo miners or wh- whatever, who are going to pick up this transaction by, because they're not, for example, in the US or, they're, they think they aren't affected by the sanctions, so they- Or they are, quote unquote, black market hash rate or black

market pool, whatever, yeah, yeah. Yeah. So I think it's,

it's, it's filtering is the right term and not, not sanctioning or, yeah, sorry, not, not

censor- Not censorship,

yeah. And it's Time, right? It was maybe three or four years ago, Mara had this thing where like it would say like, you know, OFAC compliance, something like this, and then basically a lot of people were kind of social media campaigning them to like stop doing that because that was seen, at least at that time, people weren't using the term filtering, they were saying, "No, but you're censoring," and like the whole point of Bitcoin is to be censorship resistant and so on. Do you have any comment on that? W- you were around then, right?

Yeah, yeah. This was speculating of what will is going to happen if, if Maripoon Pool launches, and is, is OFAC compliant. And this was the main, like, motivation for building out the mining pool observer tool, which actually looks at the blockchains, the templates, and compares it. And I, I just finished it up when Maripoon Pool, just mined their first block. I was ready and, could say, okay, for, for-- They say it's, it's OFAC compliant, but there weren't any, any Transactions around at the time and, so, so they actually didn't filter anything for at least the f- So it didn't even matter at that time? no, because, like, like the people rarely, like, if, if, if it, if a address makes it onto the, the list, it's, it's probably dormant at, at the point, at the time. So there is rarely any activity on these, anymore. some times, sometimes there is a new one coming up and there is still some activity, but at some point it dies down. Super easy to switch to a just a new address that's not on the list. yeah. And it only matters for like

address reuse, which- Yeah. And that, that does track with at least historically, now again, I'm talking in tendencies and things like this, but it's definitely a thing that like hacked coins or, you know, coins that were stolen. You know, whatever, rumored to be by Lazarus group or whatever, it's like, it's been a thing that, you know, some big exchange would get hacked and those coins just wouldn't move. They would just like sit there for ages. Now of course, other coins, they go through mixers and whatever, and there's other, you know, techniques that they're applying or maybe they're using mules to try to get out through KYC exchanges and things like this. But it, it, it would say, yeah, that's at least-- it, that tracks at least this idea

so be it. Yeah,

actually, actually let, let me look, look it up. Okay, there is, there is currently, six hundred and eighty UTXOs, with a total value of nine thousand three hundred five BTC, on sanctioned addresses, which is a lot more than I was expecting. I think, when I last looked at it, it was like sixty-three. So it, it must be like a new address on there or so. So, quite a few addresses,

quite a few UTX On these sanctioned addresses right now. Interesting. Okay,

yeah, I mean, I, you know better than me, I'm just saying colloquially from what you've sort of heard. Okay. And then, I'm curious in your research, did you see a difference in the treatment between, let's say, a Chinese mining pool and an American mining pool or a- Non-US, non-China mining pool, like did you notice differences in their treatment of the, of the so-called sanctioned transactions?

so, so I only detected four, F2Pool that they are filtering this out and, actually we, we only know if, if, if many people send or spend from, from these, off-exchange transactions, and I'm, I haven't tracked if there have been, any recently. So I, I can't really comment on,

on that. Sure. Yeah. And I guess, 'cause the argument might also go, well, just because it's sanctioned by the US doesn't mean some other country cares. And of course, yeah. I guess the general Bitcoin argument would be, the whole point is it's censorship resistant, and so just because one particular mining pool doesn't wanna take that transaction or include it in their block, someone else will, and the idea is that you-- the people, the, the person who is hypothetically trying to get their transaction through, they can just keep Until such time that there is a mining pool who's willing to take that. Now again, yep, not defending-- Now to be clear, for listeners, obviously not defending, you know, actual criminals, terrorists, anything like this, but just talking, let's say, more academically about the network under- You know, under the condition of people trying to make it censorship resistant.

Yeah, it's to- totally agreed. That's, that's my goal with it as well. like, I, I'm not based in the US, I have no ties to the US, so I, I, I personally don't care too much about OFAC, but i-if there is some, some other country coming out with, ad-addresses that are sanctioned and, miners based in these, in these, for-for whatever reason sanctioned, could, could be like, Just getting sanctioned or whatever, like, we just, we want to be aware of, if, if there is censorship going on and or filtering going on and if it's, if it's turning into censorship and it's really get hard to get these transactions mined, just to be, just to be aware of if, if like, does Bitcoin hold up to this censorship resistant property, we hope it has.

Yeah, and, I guess So to your knowledge then, it's, it's only the US that has Bitcoin sanctioned addresses? Now I know other, I believe the EU does have sanctions, like in fiat, fiat sanctions, and other countries have their own sanctions regimes, but it seems that, it's only the US that explicitly adds these Bitcoin addresses to their SDN list. Is that correct? Is that your understanding?

Yeah, as far as I know, yeah. Yeah. If, if someone else knows something, different, please let me know. I, I- I'd, I'd love to track these as well, but, as far as I understand, yeah.

Great. Okay. and then I guess the other big one you mentioned just earlier was this concept of multiple mining pools. Using the same block template. So, do you mind just elaborating a little bit on that? And I guess what we're getting to here is this kind of "ant pool and friends" kind of argument. So, what's your thought there?

Yeah, sure, sure. So you can, if you, if you connect to, to a mining pool, and say, "Hey, give me, give me work for my, my ASIC," they're going to send you, not the full block template, but they're going to send you like parts of the block template hash together, so the Merkle branches, so it can build up to the Merkle root, not, not going deeper than this here, but, you can compare these, these Merkle branches that the, the pool sends you and, They match across pools, then there is a very high likelihood, and if they match across like a longer time span, there is a very high likelihood of the pool being actually like only like a proxy pool, so they issue the same work, the work is based on the same block template. So we know there is probably one node running and it's queried for like a block template and this block template is then sent out through a different Yeah, names of mining pools, but in, in the end or underlying is actually the same, the same mining pool, the same, same person that, is, is like, the owner of the company and make, can make decisions on what transactions to include and, exclude, from, from this block template. So this goes in the same, yeah, filtering direction, but also goes in the, centralization direction where, if, if there's now like a mining pool that has like Under the hood, in reality, like more than fifty percent of the hash rate, it's, it's really easy for them to make, for example, the, the famous fifty-one percent attack. but you, you might only see it if you look at the Schattenberg, because on the blockchain, they all put like a different name in their block. So in mempool space, if you go, go there, it might tell you, "Oh, well, this is from, from Brains, and this is from, Antpool, and this is from Foundry." but they won't tell you, "Oh, this is from actually Antpool and Brains could be the same entity based on the, on the block template we saw."

just To this, and I'm sure you're aware of this, is it's not just the Stratum work component of it, is I think some people were sort of doing chain analysis and basically looking at where the payouts were going and at least, you know, a couple years ago, this is true, I don't know if it still is, that a lot of the mining pool payouts were going to the same, I believe some entity called Kobo in China, because apparently they were like an OTC desk, so then it was sort of like, not only was the Stratum aspect of it, being a proxy pool, the payouts were also going the same

place. Multiple, like things pointing or multiple data points pointing at the, at the existence of like some bigger consortium. Of miners colluding in some way. We, we, we obviously don't know, we don't have, we don't have insights into their, like,

their, what they have written up or what, what their agreements are, but we can only

speculate there. Okay. Now, this does throw into question around all the arguments around, you know, Bitcoin mining decentralization. Of course, we want it to be decentralized. I think most people can agree with that, but maybe where people might disagree is where they might say, "Well, Paul, enter..." country is still relatively open or the fact that these, they might say, "Hang on, these pools are just an intermediary. The underlying miners or hashers, if you wanna use that different parlance, those hashers can just point their hash rate somewhere else. That if they see something wrong, now to be clear and to be fair to you, some of your work might help them understand if there is some kind of filtering happening at the pool level, but theoretically those hashers can..." You know, with, without a huge amount of effort, although even that's a debated point maybe, so those hashes can point their hash rate to some other pool, and in that way, the, you know, the argument could be, well, it's still central-- decentralized in that sense. what's your thought on that?

Yeah, totally, totally agree. So, some, some mining or big, bigger mining farms might have agreements, with a big mining pool, to, to say, okay, you have to point your hash rate here because we helped you finance They're also like if there's a small operation, it's, it's maybe like self-bootstrapped or whatever, they obviously can switch, freely switch pools, and, and they should do it, obviously they need to have some insights into, okay, here's something wrong, and people need to raise awareness for, for them to get aware of it, But, yeah, definitely, like the underlying hard hardware distribution, where the hardware actually sits and who owns the hardware is probably quite de-centralized. But then again, the next, next point is like, where is actually all the hardware coming from?

Right. And that's again, then again,

it's mostly one manufacturer or the biggest manufacturer has a big market share. yeah.

Yeah, I see. So, yeah, so 'cause there's different ways you can slice and dice that aspect, as you, as you mentioned, it's mainly Bitmain, and I think the, MicroBT, the creators of WhatsMiner. but then I know there are some competitors, you know, competi- there is some competition there, like, yeah, there is Proto, the one from Doty's Block, yeah. And, maybe some other, some other smaller ones. I know there's, this, was it called Auradine or Auradine? I don't know how to pronounce it, but I believe they have an SV2,

compatible or it's like built into the mining, machine.

Yeah, that's a new one. I Coming up or, or similar? Yeah.

Yeah, I guess then, do you, I guess, do you have any further, thought there or any, point you wanna make there about mining cent- Oh, actually, that's the other point I was gonna ask you around FPBS and PPLNS or PPLNS style payout models. so this has been another, let's say, big debate and talking point, which is that FPF-PPS is sort of like an insurance product for some of these, mining pools. so, and so you need a large balance sheet because you are like fronting the capital. You're paying them out before you've actually received the incoming. Of course, there's been arguments there from, Some of the other pool operators saying that's, you know, maybe that's the wrong thing or that's not the good thing for de-- Bitcoin decentralization. Obviously here, like Ocean is one example with their Dartum, and then another it would be, the Demand Pool with their, they have full SV2, and so they would be probably in the camp of They see them, they arguably see themselves as going against FPBS. do you have any comment on the payout structure for mining pools?

yeah, so, so two things come to mind. So one is, I, I, I think, I think there is one big entity or one insurer behind some of the FPS pools. And that's why some of them need to-- like, this is speculation, I, I, I, I can't confirm this, I don't have any insights into contracts here, but, speculation is that there's one issuer that is, helping like smaller pools or like little pools, with some of the capital, and helping them if, if they don't find a block, helping them with their FPS. And the second comment is, of course You see, I think with FPPS, you don't as a, as a, as a, as an ASIC miner, you, you sell your hash rate for the, the block subsidy, but I, I don't think you're actually getting much of the, the fees. And, we saw this during fee spikes, the, the miners at, at, non-FPPS pools, they're actually making a lot more, in revenue, compared to, to others who, who mine at, at FPPS I suppose because maybe the pool keeps the FAP, the, the fee, the fee part or has to give it to the insurer or I don't know what, what the ideal there, there is, but, yeah, that, that's the two things that come to mind for me there.

Okay, understood. And then while we're here, I'm not sure how close you are with this, but Bitcoin Core v30 has this new thing, I think it's called, I was just looking it up, it's called IPC, Inter Process Communication Mining Interface, and the idea is it was a way for SV2, To connect with Bitcoin Core, I'm curious if any of your work touches in this area or not

really? not at the moment, but, I'm looking, forward to exploring this interface a bit as well. I, I, I think, it's, it's, a good next step of, of like having, real time, getting real time data out of Bitcoin Core, especially like for, for the mining side, but also there for, for other sides. so I definitely want to play around with it, In, in, in future projects and also like, the, the mining, mining pool observer, project I, I built a while ago, it used the RPC way of getting a block template, but I think, actually like investing some time into it and, and getting it, to get the block template via RPC would be, would be very helpful, in some, in some future like small hack project on, on it.

I see, yeah. And I, as I understand, the SV2 guys, the Strato V2 guys, see that as, you know, they've been trying to get something like this into core, and I think this was their way of making it, You know, 'cause, 'cause their goal obviously would be to have ecosystem support, right? Like the pools to support it, the mining machines to support it, and Bitcoin Core to support it, and of course, people to demand it themselves, like to want to use it. And so I guess this is one step to sort of- In that direction, and so they're also trying to show various things about SV2 being better, things like, okay, the communication layer is encrypted and obviously the, the block template, creation can be decentralized, that's probably the big headline thing, but there are these other benefits that, some of those guys are talking about.

Yeah, additionally for like the IPC interface, it's, it's great because you can now, well, this was the o-original motivation for it was that you can now run a different process for your node and your wallet and your GUI, and you have these separate and not, not the same process. So s-some attacker can't go via like, the P2P messages and in some way crash your node, and now your, your wallet and GUI shuts down as well. or maybe he can, but, if, if, if some, if, if for If the wallet crashes or the GUI crashes, your node won't crash, because they're, because they're not in the same process anymore, because they're using this inter-process communication, this IPC interface. And let's

now talk a bit, we, you were touching on it there, but let's talk about your broader project, which is this Peer Observer project. I see this is one of the newer ones that you are working on. can you give us an overview what is the Peer Observer project?

Cool, yeah. so Peer, Peer Observer is, is It was a tool and infrastructure, and the idea is to detect, attacks on, on nodes in the P2P network, but also like anomalies that these nodes, show in some way. and the motivation for it was, I was reading in, I think twenty twenty-one or so, I was reading on, on some attack that happened on the P2P network, and, after thinking about for a while, I, I noticed, well, this was only detected by coincidence. different people saw it, but only because they were, for example, reviewing a PR and were looking at, at logs, or some, some, some guy building a custom node implementation saw that his node implementation couldn't handle the attack and, and crashed in some way. And I, I thought, "Well, wait, with a trillion dollar network, not at the time, but now we have a trillion dollar network, we want to have some monitoring in place so we can detect these attacks a bit more professionally and say, 'Okay, well, there is something going on. do we need to react in any kind of way? Does it, does this need to fix?' And, the question I always ask is like What do other two trillion dollar network or, companies, I mean, have in monitoring and incident response teams, and how many people do they have on call? Like, think about like a company like Meta, how many people do you think they have on call right now? and how many people does Bitcoin have on call right now, to, to respond to incidents on the network and, and, and fix stuff? Obviously, obviously that's not, not an apples to apples comparison, but definitely like a motivation to To say, okay, we're, we should be a bit more professional about monitoring i-i-in general, not only P2P monitoring. This, this network we have is a very val- valuable asset, and if, if, if, if something goes wrong there, yeah, nobody can send transactions and nobody can mine blocks, and everything building on, on top of it is, is going to, break in, in some way.

And as I recall from the earlier days, there was-- there used to be this kind of network alert key thing. I don't know the Oh yeah, yeah, yeah, yeah, yeah. because that is also kind of a centralization vector. So at the same time, it's like, this thing has grown so much compared to early days of Bitcoin, you want a way for people to sort of obviously track and stay up to date on what's going on, and then also if the fix is needed, how do you get the word out, right?

Yeah, yeah. Right. And, and the, the roadmap for the, for the Pure Observer project is to have passive monitoring. So, you, you can also have Is you connect to all nodes on the network that are reachable, like,

like a spy node kind of thing. Yeah, yeah.

For example, or could also be like a research node. It's not, not only like spies, like spies always has this negative touch to it. Right. but, I, I choose, I choose the passive route, so I have like well-behaving Bitcoin Core nodes and nodes and whatever, like, on, in, in different, different locations around the world, they all are well-behaving, they all behave like very,

yeah And I call them honeypot nodes. so I have the hope that an attacker or a network-wide attacker also happens to attack one of, one of my nodes, and I have just attached a bunch of monitoring tools, and infrastructure to it, so we can actually know what, what happened to this node. we have logs from it, we have data, we have metrics from it, and, we can get an idea of, okay, what, what happened to this node, and,

Why, why is the five thousand per crashing or why is it slowing down or what else? Yeah. Interesting. And as

you mentioned in your, I believe this is in your blog, you mentioned that you run them with different configurations, right? Some of them are on Tor and I2P and CJDNS and one has got like, you said Bloom filters, known to be doccable. Talk us through some of the different configurations that you have. Yeah, yeah. on this.

Yeah. So, so, so the goal is two fourths, so we wanted to have In fact, if someone comes from the outside and sends up something, we end up, slowing down on, so it doesn't end up service, or if the node internally has a problem, like for example, a bug in some feature, maybe not even like a, like a much used feature, like a feature that's only used by, like a selected few, percent on the network. So, that's why I said, okay, to have good coverage there, we want to run these, these nodes, with different configurations. So for example, I have nodes running on, on Tor, but also nodes running on, on I2P. So Tor and I-I2P are both the privacy networks, and there's a third network, that's called CJDNS. it's not really a privacy network, it's more like an overlay network. I have some, some nodes running there. There's not much activity go-- like, no activity on, on this network, right now, but maybe we're going to see more of it in the future. And, yeah, as you mentioned, I have some nodes with Bloom filters, and Bloom filters are, the predecessor of the, compact block fil- compact block filters, that we have now, and they're, they leak some privacy information, and they're also known to be doxable. So if, if someone connects to your and actually, requests a bunch of these, these, filters from you, your node will slow down, and that's known as Doxvector, and they're disabled By default, very few people run them. yeah, but I, I actually like on this node which I, which I run this on, I actually have seen it, it, it getting dusted, ju-just because there are still some old wallets out there, the old xPub wallets, that haven't implemented, the copic block filters, they still rely on these bloom filter nodes to be available. so if a new bloom filter node comes on, com-come online, they all reach out to it And, yeah, i-in some way does it, they overload it, the requests for, for these routers. And, yeah, I, I also run, for example, one node with a ban list just to see, what happens, to this node if I exclude these known or maliciously known, IP addresses from connecting to it, and what happens to the other nodes where they're not banned. I'm also testing the ASMAP project on it, on some of the nodes, some of the nodes Nodes are pruned, some of them aren't, aren't pruned, some of them have like very many inbound connection slots available for, for peers. So what happens if, if one, like, if, if five hundred, inbound connections are on this node compared, like, to the, to the normal one hundred and fifteen, and one node is just running in, in blocks only mode, so it doesn't have mempool, yeah? You get a feeling just to cover as many grounds here as, as possible.

I see, yeah.

And so,

are there any overall insights, that you have to share from what you've done so far on the Peer Observer project?

So generally, it, it produces more data than, than one person can handle and analyze, I noticed. And also like not everything can be done with passive monitoring. so if, if you want to look at, for example, block, block propagation, you need some active- Monitoring as well, because, because these passive nodes don't connect to, to all of the network, you, you don't get like a good sample of, of what, what happens on the net-network. You only get the local view of what happens with your peers and your node, but you don't get the global view of the network. And for, for some of the data that is really helpful for research, y-you want to have this, this global view of the network, if you, if you can get it. But I, I think it's generally good Very positive. my, my, my hope is that there's other people looking at other stuff as well for to monitor. So I, I'm currently focusing on, on P2P stuff, especially on the passive stuff, but there's so much other things to look at in, in, in Bitcoin data and, and things to lo- monitor for, and people are doing it, like there is, there is people now doing, the, the monitoring, the, the Stratum works that pool sent out and so on, but yeah. So definitely good to have a P2P monitoring, but there is so much more, if we, if we want to do, good, good have good coverage of, of general Bitcoin network monitoring. So

what other non-P2P things that you think need to be monitored, or what else, what, what else would be interesting to monitor?

Yeah, good question. So, so for some, some other thing that has been people, or people have been monitoring is like, in general, like forks and re-forks and, and stale blocks and so on. Oh, okay, yeah, yeah. Yeah. So, so, Bitmax Research is running the, fork monitor, like a fork monitor thing. Right. And I ha-- I have a similar tool called Fork Observer. We have the observer top domain, yeah, and, and I, I guess that's one, one thing, and, and there is the Stratum dot work thing, and, MemPoolSpace also has this, in, in some capacity that they monitor like what, what, pool sent out, but, yeah, I, I, I think there is, there is plenty of, of, of things, i-if you look, if you look at, at some stuff, there is plenty of, of low-hanging fruits to detect,

and, and, I, yeah. But I, I don't have like, like an, like a list of, of, of things, for people right now. In terms of

cost, what does it cost to do this? Like, how do you, f- like, is it, does, is it a big cost to do this or is it a cheap cost?

so, since I initially built this out myself and, I have grant funding, but I, I don't have any other funding for it or had any other funding for it, I, I I, I think if you, if running one node is like five, five USD per month or so, and then, like scaling this up, I, I think I currently run eighteen nodes or so, that, that's, so it, it's not, it's not too expensive, and, a lot of people have stepped up and, and said, "Hey, we're going to sponsor some of these, these nodes." For example, Brain has been sponsoring some of them, MIT DCI, as well. They, they initially, Sponsoring Mewes nodes, and, yeah, the local host research people are, are also sponsoring a node. Chaincode is not sponsoring a node, and even some ind-individuals, stepped up and said, "Hey, I have, I have capacity, I can sponsor like a node or so." but I think currently, it's not, it's not the, the bottleneck is not spinning up new nodes and running new nodes and maintaining them, but it's, it's more, actually getting people on board, Well, b-because what, what does analyze the logs and things like this? Yeah, and, and, and thinking about new metrics to track and maybe even working on some anomaly detection frameworks or so, because, what, what does more data help you if you, if you can't process it right now? I think, I think the next step would be to, to look more at, more, more looking at the data and not producing more data.

Understood. Okay. so talk to us a little bit about How you're getting that, or like, what kinds of data are we talking about here in from a, now, let's say, in the world of P2P? Like, are we talking things like the INV message and, you know, that, like, kind of that version negotiation between nodes, like peer, like how peers find each other, this kind of, this kind of thing? This is the realm that we're talking about, right?

Yes. So the, the peer observer tooling. So a peer observer, I split it in, in three components. So tool Infrastructure and data analysis, and the, the tooling is, is like what hooks into different interfaces of a node, and one of the interfaces is the tracing, tracing interface, of, of the Hikore, and from there we can get A very good view of like what P2P messages are exchanged. So for example, when the connection is opened, the in-- the, the, the version message and the, the verack messages being exchanged and so on, but also over time, like addresses being gossip, blocks being, blocks being propagated and transactions, being shared. but it could also be like connections being opened, and if you, if you, for example, observe like a node getting a very high rate of, of connections. Being open to it could also be like an indication that something is wrong. so, definitely, definitely these, but also like just simply like a transaction getting added to mempool or a transaction getting replaced in the mempool is, is something, that, that we can't track or is interesting to track, because like for example, a network-wide attacker could do like a, attack where he replaces a lot of transactions over and over and over again, which costs a lot of bandwidth for all nodes on the network. Network, and that's something we want to see. So definitely like transactions getting replaced as well. So just to make this,

I guess, practical for listeners, and especially people who aren't, let's say, as deep into the weeds, let's talk about What is the expected behavior for a Bitcoin node and what would be unexpected behavior or typically like spy node behavior or like aggressively trying to connect? So maybe you could talk us through like what is like a standard expected node, what are the normal behaviors you should see? first, and then we can talk about what is like a more malicious or maybe questionable node might do. Alright,

so, if you run a home, a node at home, and it's behind a firewall, it's, it's going to open a few outbound connections, so it, it's going to reach out to a few peers on the network, and it's going to ask, them for, for transactions and blocks, and it's going to share these with the connected peers. now compare that to a node you run in, for example, For a cloud provider, and it doesn't have a firewall, so the port eighty-eight sixty-three is, is, is, is open and people can connect to it. it, like, additional to these outbound connections, it's going to make some of the outbound connections are going to reach it as well, so it, it's going to be inbound connections on this node. Right. You're

reachable, that, the word is, right? Yeah.

There, it, it's going to accept by, by default, I think, one hundred and fourteen or one hundred and fifteen To fill up over time, the, the IP address of this node, this cloud node, is going to, to spread out ac-cross the network, and maybe one of the, the no-nodes at home is going to say, "Hey, I'm, I'm missing one of my, one of my eight or ten or eleven, outbound connections. I'm going to open a connection to this cloud node." And, over time, the cloud node will fill up, all, all the, the inbound connection slots. So, and one, one thing that InTech could do is, it- The attacker could go ahead and say, "Okay, I want to open all, or I want to open many connections to, to this cloud node, many, as many as possible, and fill up many, many of these inbound slots there as possible, and just don't do anything with it, just, sit there and don't do anything." And like a grief, a griefing method sort of thing. Yeah. that's, that's one attack an attacker could do, but it could also do, it, it could connect to, to many, outbound Any of these cloud nodes and just say, okay, hey, send me, send me data about like transactions, you know? And the node will happily do it. but then again, it might reveal while doing so, that it, reci- or that it knows about one transaction earlier than the rest of the network does. and this would, for example, be a privacy leak where the, the attacker could tell, "Hey, because I'm spying on all nodes in the network, and I saw this node..." Broadcast the transaction first, it's probably this node that, or this IP address that initially created this transaction, and now I can't tie the UTXO to this transa- to this IP address. And yeah, as I recall, this is

like part of the motivation for like the whole dandelion protocol thing. I don't think the dandelion thing made it into core, but the idea was, yeah, if you had like some super nodes like all around the world, and you were trying to aggressively, aggressively connect to everybody so you would find out where it first appeared, and then Oh, it was very likely it came from this location, and if it is a home IP, then boom, they already are able to figure out with a, you know, with other information like they could potentially figure out your address, and, you know, that's also obviously a doxing. Or if it's, if

it's like a, like a big, like a big node from like an exchange or so, and they see all, but these, all these transaction first originate at this, at this big node, and they know, oh wait, this transaction is actually from exchange. Change XYZ, then they can say, okay, well, we can actually tag this, this, this address, right? This is the, this is the Coinbase coins, this is Binance coins or whatever,

right? Yeah. Interesting. Yeah. And of course, the, even now, nowadays in the age of, you know, let's say when treasury companies come out and announce, "Oh, we just bought this many coin," like people are trying to like trace on chain to sort of understand, okay, where, where are these coins coming from? What's their OTC provider? What

In the background, kind of under the hood. So I guess in the realm of expected behaviors, you should see nodes just, you know, syncing up the chain, transac-- you know, forwarding transactions and forwarding blocks and receiving transactions where they need to, but then in the malicious side of things, let's say the malicious or spy node might be kind of aggressively connecting to everybody or aggressively like griefing your- Network slots or kind of aggressively asking for information even if it doesn't need it, this kind of thing, right?

Yeah. For example, but I, I, I think we're looking for, while, while doing so, we're looking for a lot of unknown unknowns. There are so many, so many things that, that, that someone can do, to your node, so while, while monitoring it, you, you have to look for unknown unknowns. You, you don't know what, what, you can't map all possible attack vectors beforehand. So gotcha. Yeah, so it's more just

like you have some rough ideas of what they-- what attackers have done before, right, but there's always something new they could try.

Or you know what, like, what normal behavior is, and if you see something that's way out of, way out, way out of the, out of the baseline, yeah,

okay, that's something I need to research or analyze. Okay.

And this is usually always the flow, going into the data analysis part of it, if I see something, That's not as, as you say, not in the, in the baseline. I, I, I go into it and say, "Hey, is this actually worth analyzing? Is, is this something, that, that could be, could be in, in some way malicious or dangerous, or should we, should we dive, dive deeper into it?" And maybe I, I write up some, some, blog posts about it or so. And then the next step is also like, "Should we react to it? Is, is there something we can do to, to mitigate this?

About the, the spynode behavior and there's actually like a, a pull request that came out of one of these, analyzes I did. So Vasil is, is working on, a pull request where we don't, we don't do, dandelion, but we do, send our transactions over Tor or I2P, peers first, and then see if it comes back to us in, into our mempool through, through, through a different peer. And then say, okay, this transaction is broadcast. We don't send it out to, IPv4 and IPv6, peers first. so that's a bit of a privacy-p preserving way of doing transaction broadcast, which I think originated from some of the data analysis, we did with, with the Peer Observer project.

Gotcha. I'm curious how that impacts or is impacted by the V2 peer-to-peer encryption? Because don't we already have that now, as of like a few versions back? So i-is that, you know, not enough or you see it as like, that's just like some orthogonal thing?

I, I think it's completely orthogonal. the P2P V2 thing is, you want to hide from an I-ISP or like a, a global attacker that can read all network traffic. so be-before V2 tr-uh, transport like the, the initial, the, the messages being exchanged where- They're private and afterwards, sorry, before we do the, the, message exchange, they're unencrypted, and then we do they're encrypted, so this hides from, from an attacker sitting on the wire, like a passive attacker, just recording all network traffic, sitting on the wire. This hides when a transaction got broadcast here and there. But if you're, if you're, if you're a node and you connect to someone else, via IPv4, for example, they're going to see, hey, this IP address, arrived from this other node, from my home node, for example, to, to this other node. and the other node is, is, is then able to tell, hey, okay, this is the first time I saw this transaction. it must be, it must be from me. B10C. So the different, different attackers, in, in a sense. So one is the, the spy node attacker and one is the global attacker, and the global attacker is, it's, it's gotten a bit harder for the global attacker or for the ISP attacker to spy on, on, on nodes, because they're, yeah, because the, the, B2 encryption of the, of the protocol messages is, is there now.

So while we're talking about spy nodes and other entities, I know you did a post- About this entity you named as Linking Lion. So tell us the story here. Wh-who or what do we know about Linking Lion?

So, yeah, I, I, I f- I first noticed Linking Lion be-probably because they're, they're opening, like a lot of connections to all nodes on, on the network. So I, I think at times I have like three or four or five or six or seven connections from them, from different IP addresses, but all from the same IP ad-IP address block, All reaching multiple of my nodes, so on, on, on all the nodes I see the same, IP address blocks being connected to me. and then I started looking into, okay, what are they doing? Why are they opening these many connections? And, they just sit there and they listen for transactions, my node tells them. And then looking at the IP addresses, I found out, wait, actually these IP addresses all belong to the same, AS. So they all registered with the same, with the same, with the same party. this party happens to be called, Linkline Networks. I, I don't think actually these are like the people doing it. I think these are the people, just- Providing infrastructure because probably the infrastructure is rented from them. They're like an ISP for whoever's doing this. Yeah, f-for example. And then it turns out these IP addresses are actually are on like a, a ban list Craig Maxwell published in like 2018. So they have been around for a while. and then it turns out these IP addresses also appear on like a Monero ban list. And at this point, it was like, okay, if they appear on Bitcoin for ages, and they appear, the appear, appear, appear, appear on- On Monero for ages, this must be some kind of Genesis company, spying on Monero and Bitcoin, and trying, trying to- I see. Like a chain analysis

or, one of these other ones, elliptic, elliptic or whatever.

Specifically like this Genesis company, but, like, like some company trying to improve their- The point is, it fits

the pattern. It fits the pattern of a chain surveillance company.

Yeah. So in, in some way trying to improve their products by, by having this additional data, by linking like this And, IP address, data, for transactions by linking them together. And I, I, I guess you can, you can create like a, a good model of who owns which coins or which IP address interacted with, with, with, with coins, by doing so.

This episode is brought to you by CoinKite, the makers of my favorite Bitcoin hardware wallet, the Coldcard Q. Now, some people think self-custody is too hard, but it's really about taking responsibility for your Bitcoin wealth and understanding that self-custody gives you a true feeling of liberty. The Coldcard Q has a full keyboard and big screen, it's got two secure elements and a true air gap, allowing you to go fully air-gapped using QR codes from seed generation to transaction signing. You can power the device using three triple-A batteries, so you don't even have to plug Sparrow Wallet for PC or Nunchok on mobile, and you can dial it into the right level of security and complexity that you choose. If you want a simple setup, just use twelve words and single signature. If you want passphrases, easy. If you want multi-sig or co-signing features, you've got those too. So go to coinkite dot com, use code livera to get ten percent off on your cold card or other devices and level up your self-custody today. Yeah, fascinating. Because, and the other thing, as I'm sure you're well You know, in the past, and probably nowadays, a bunch of the public Electrum servers might be run by some of these chain surveillance companies too, because that's another way that they can easily find out, "Oh, okay, you're, you are coming from this IP address, and you're asking me about these particular coins, I'm gonna assume you control those coins, right?" And so that's like a way for them, for unsuspecting users, to dox their coins to the chain surveillance firms.

Yeah, so, so if you, if you connect to, if you connect to like an However, you're going to tell them, hey, I have funds on these addresses, tell me, tell me if anything moved here, and I'm going, in the future, I'm going to use these addresses, and you send us in the same message so they can cluster them together. In the future, I'm going to use these addresses to receive funds, have you seen anything on, have I received anything here? So it, it, it would be really stupid of like a Chainalysis company of not, not running like custom Electrum server, infrastructure that just locks everything. And, yeah, I, I actually wrote up like a small like project idea if, if someone wants to hack on this, maybe you can link it in the show notes, but I, I wrote up like a small project ideas of actually like finding out if, if there's like, Electrum servers that all like behave, very similar So for example, they switch to a new block at exactly the same time. so from there on, we could actually tell if it's like a, like a proxy or like a final node. Right. Trying to surveil them back and figure out which one,

which, which of the public Electrum servers are likely to be chain surveillance ones.

Correct. Yeah, yeah, correct. so we, we, I think with data analysis and monitoring, we can actually turn it around and make this more public and, and, yeah, I haven't gotten the time to it, for

Research report on it or like a blog post or so.

Interesting. And I guess this is also part of the broader thing of like why, you know, typical like Bitcoiner, you know, advocates will say, "Hey, run your own Bitcoin node because you get more privacy that way." And maybe run your own Electrum

server if you're, if you're using Electrum, yeah, yeah.

Of course, of course. And so maybe for that, let's say that user who isn't willing or not able to run their own full Bitcoin node yet, that's where some of this compact block filter Whereas the project is gonna be interesting to get more people able to quickly, do their own validation without doxing to the public Electrum servers and things like this.

Yeah,

definitely,

yeah.

Yeah. So let's talk a little bit about compact block relay because I know this is another area you've done some research and some writing on this, a-and this kinda comes into like block propagation and my understanding here is the general idea is for the decentralization of the network, you want this block propagation to- You know, to be, you know, you want it to be very low latency and, such that, nodes aren't having to go out and manually ask for transactions that they can just, you know, do it on their own. Can you just elaborate and explain this?

Yeah, okay. Let me step back one, one, one step because we just talked about, block filters or compact block filters, and now we're talking about compact block relay, and I, I think these are very different things, but they're named the same way. So compact block filters, that's BIP one five eight five

seven and one five se- Yeah, one five eight is it? Yeah.

Yeah, one and one five two. So very confusing, same names, same in the same number range of for the BIPs, but this is for- Block relay, not block filters. Block filters is something where we get like data from about like outputs and coins we have, and, block relay is where we, like, a-actually distribute the blocks when they're mined across the network. so, for, for compared block relay, the, one of these, one of the ideas is that while a block is mined, it's being mined, you receive a bunch of the, the transactions already, you already have them in your node's mempool, and now So if, if someone is, is finding a block, they send the full block, all of these transactions included, and these would arrive twice, because you already have them in your mempool, you're, you're spending like the, the, the same bandwidth twice, you're sending the same data twice, so this is a problem for bandwidth and also like sending the full block, like, one megabyte, two megabytes, maybe even like four megabytes if there's a big inscription in it, it, it just, it's slower. Just, sending this around the globe is way slower than just sending like a, like a wall, small sketch of it saying, "Hey, you already have the transactions, they're are in your mempool here, here and here. Just pick them up, and try to construct the block from that." So you, you do, you, you send a blueprint, say, "Hey, you have the, the transactions probably have them already in your mempool. Try to reconstruct the block from, from these transactions you have in your mempool." And, if, if you-- this, this normally works fine, this normally works quite well. Sometimes you, you don't have a transaction, then you need to request it, but that's also fine. the hope is that if, if, like the mempools across the, the network are very similar, then, most of these blocks don't need like an extra request. If we do need to do like an extra request, this is like an whole additional round trip because you receive You receive like the message, "Hey, here's a new, compact block reconstructed with this, with these, transactions." And On the, and if, if you don't have a transaction, you need to say, "Hey, I'm missing this transaction, please send it to me." And while doing so, you can't, you can't reconstruct the block, you can't forward the block again. And well, this-- depending on how far away your, your peer is, it needs like, an extra like hundred milliseconds or so, and we want to minimize this, this delay.

Okay,

yeah. So just let me

try to recapitulate in simple terms, just, both myself and also for the listeners. So the idea is, when you're running a Bitcoin node, you are also sending and receiving transactions that you are relaying, and the idea is that instead of having to download a full new block, the idea is- Is actually you might already have these transactions that have been relayed to you anyway, so just use that to help reconstruct or, create the block without needing to manually go and request, "Oh, I'm missing the ninety-ninth transaction in the, whatever, the twenty-seventh transaction or whatever." and the idea is, by saving, by doing this compact block relay, you're saving that extra round trip of having to go and manually ask for these transactions that you don't know about. Is that Broadly right, or yeah. Yeah,

you're saving, you're saving on, on, on bandwidth and you're, you're saving also like on, in general on, on latency. So you're making, you're making block propagation faster, b-because you can actually send out like,

send out like these small me-data messages and not these big, big few megabyte messages. additional, there are a few additional things. So for example, so some connections they not might, might send even the, the combat block before it's, it's validated. so you, you might receive, like a, a, a, a combat block of, of an invalid block that could happen. but this makes, like, before validation, sending it out before validation makes It's so much faster than it's actually worthwhile doing, and, yeah, yeah. We, we have,

like, the point you're getting at there is mining has become such a brutal competition that mining pools are doing this because they don't wanna lose out, because even these small, delays in timing can really cost them in their bottom line, right?

Yeah, and this is especially true for like small miners. so a big, a big miner has a higher chance of finding a block so, if, if, if they, if they, broadcast it, and it takes, let's say hypothetically five seconds to reach a small miner, the, the small miner would have mined on the old block, which is, isn't going to, to get any reward from for like five seconds. so the big miners have, have like a, a, a, a good, a, a, a good, or a higher rate of finding blocks Blocks, they find more blocks, so that's less of a problem for them, but for like a really small miner that, where it takes a long while for, for them to reach or to, to receive the new block, that's, that's the bigger problem. And this causes mining center division in, in some ways, because obviously like, the big miners then have an even higher chance of finding the new block because they actually have the work, of, on, on, like, they're, they're building on the new block way earlier than, than the smaller mining pool.

Coming now to how much of a, I guess, how much of a factor do you think that is? Like, is, is it meaningful to the point that small mining pools can no longer start up? Or like, I guess what I'm trying to get, I'm trying to understand from you is, is this just like, you know, quote unquote, nerds arguing about technical minutia, or how relevant is this to the actual real world business model of mining pools?

Yeah, I, I think it's somewhere in between, and, yeah, I think we, we also revisited a-as a few, a few of the P2P engineers, invol-involved in this, we sat down and we actually spoke, talked through this and pulled, pulled out some statistics on this. and I, I think it's not hundred percent clear if, if these like worst reconstruction rates or reconstruction rates is if I have to re-cast a transaction, and it, it's all, everything is, is getting slower, i-if, if these actually make a meaningful or a big impact on, the actual blockchain Propagation time and also on like stale rates we see, just because we have this sending out the block before we validate, we validated, which, which is a huge, gain in, in, in timings. So, yeah, I think the, the bottom line there was we need to dig a bit deeper and we actually need to, look at which blocks propagate fast and which propagate slow, to get an idea for, or a, a bit of Better database and not intuition and, and, and, assumption-based, model around this. So I, I think this is even like a still an, an open, a somewhat open research problem,

Okay.

but yeah, it's, it's also some, some people arguing about like, oh, this, this, could be higher and this could be lower, and but also like having a lot of stale blocks, is, is not nice. So, yeah.

And I guess just to clarify there, the, the point also there If you're a big mining pool, obviously you wanna have a very low or zero stale rate because the more stale rate you have, that's just like directly you were point, you were contributing or you were putting hash rate into a block that was Not viable, right? Yeah. And so basically they want to minimize that or avoid it completely, eliminate it if possible.

I, I think it's, it's, it's not possible to elim-eliminate it completely, but, there's always going to be some delay, but you, you want to minimize it, yeah, definitely.

Okay. And so can you just-- obviously I know you've written like blog posts and put up whole diagrams about it, but do you have any broad trends you can share with listeners on? You know, the compa- the reconstruction rates and the success rates and the failure rates there?

Yes. The compaction reconstruction rates, I just looked at for a given day, how many of the blocks that were relayed to my nodes as compact blocks, how many of them needed, like an extra round trip, how many of them needed the transaction to be requested. Just plotting this over time, you see, on some, in some months it's, it's really good because everybody has- Agrees on what, what their mempool policy is, and obviously doing, for example, like one Satoshi per byte summer, where some mining pools include transactions, others aren't including or, even notes, the, the majority of notes aren't including, y-y-you're going to need to re-create them. So these, reconstruction rates, they, they're going to be, deep, deep red, so they're, they're going to tank. there's, there's always a transaction At least one transaction or a few transactions you have to request, which does the extra round trip, which makes, propagation just a tiny bit slower.

Okay.

Do you have a sense of how much of a delay that was causing? I, I, I don't, I don't really have any like data on this, but this is like, an a-active like research topic. gotcha.

But you would say fair to say that events on the network, such as, quote unquote, SubSat Summer, which for listeners, there was like, let's say, a social phenomenon of Okay, so I guess I'm just gonna quickly explain it, and you correct me if I'm getting something wrong here. But for a long time in Bitcoin, it was kind of just assumed that one sat per vbyte was the floor, and so this is kind of hardcoded into a bunch of wallets and everything. But then I think someone tweeted it or posted about this, and it, it was like a social phenomenon that someone said, "Hey, actually, there are other peers on the network who are willing to relay transactions below one sat per vbyte." And then this kind of kicked off a trend where a lot of some, a bunch of pools all like lowered their min relay TX fee, such that it was now below one sat, and some of them were going to point one, some of them like went down to point one, but back up to point five. Anyway, the point is This was like a social phenomenon that then r-resulted in many, many transactions where previously one sat was just kind of unquestionably the floor for most, for most things. It was like now there were actually a lot of transactions going through at below one sat. Have I recapitulated it correctly there or anything you wanna add there?

No, so, so just because like the nodes all had the, the same default policy in them, and if you don't change the default policy, you, you, you, you stick with the one sat. We buy, but you, you could have changed, or you can manu-manually change, the, the, to point one,

the config, right? So like just for listeners, this is like if you go into your bitcoin dot conf file and you type, I think it's min relay txv equals, you know, whatever, and if you manually s-change that, that's what we're talking about here, just so you know. Yeah. Yeah.

And, and there is like, this, this doesn't need to be like, this, this general So there is also the, the operator and policy change, and that was the, full RBF policy change, and for all of them we see it in some capacity. So, for example, if for, for the full RBF, the mining pools were all, or many of the mining pools, like I think like more than ninety-five percent of the hash rate were all mining full RBF transactions, so and thus deviated from the default policy in, in, in, for example, Bitcoin Core and, It, it turns out, if, if, if, if you change, the policy on your node, you're actually better aligning with what the mining pools do, and you get a better under-- or you get better compact block reconstruction rate. Obviously, that's not the only thing we want to optimize for, that's not the holy grail, but it's, it's important to keep in mind that, compact block propagation is, is fast. our block propagation

is fast. So just to explain that a little, it's, what you're getting In the case of full RBF, that was arguably kind of mi-mining led. It wasn't actually core who kind of came out and changed the default, it was that, you know, the miners had Either realized or maybe, maybe Peter Todd or someone was kind of, pushing that idea of, full RBF, and then because it was already in practice on the network, then at that time Core changed the default. And yes, at that time, off the top of my head, I think this is like late twenty twenty-two-ish, around there, there were a few people in opposition of that, so people like John Carvalho or Sergey from Bitrefill were against that, And I think the Moon Wallet guys also didn't like that, but I think it eventually was seen like, the network is just going this way, we're just gonna have to deal with it.

do you wanna add anything there or on the kind of, I guess, the context around that?

I think, I think, yeah, I, I think it makes sense to switch to a default if, if like, if it's part of the, of the network does it, or switch, or update the de-up, update the default if, if, if it's part of network, it, makes it-- there, there's always, definitely like friction there because some, some people maybe relied on this for their infrastructure or maybe some people don't like having multiple operations in, in, in their, in their transactions or like bigger operations. Returns in the transaction zone, there's always going to be some, some friction, doing these changes, but yeah, a different, different things to optimize for,

yeah. But I guess from an observation and analytical standpoint, did you notice that? When it was, like, so let's say in the full RBF case or even, well, were you, were you doing some of these stats even back then in the full RBF days? Yeah, I, I think I have some of them, yeah, yeah. Okay, so I'm curious then, would you say, would you say it's fair to say that- When the miners were just doing it on their side, not at the core default level, compact block, reconstruction rates were lower, and then after core changed the default, they went up. Is that, yeah, a fair statement?

So, so I, I did, I, I ran different nodes, with different configurations there. So I ran one node with, for example, full RBF, enabled, and this node, over the same time span had a much better rate at, at reconstructing the- These blocks. So this clearly indicated that, the full RBF was the, was the, problem here, but you can also see this from just looking at what's, what transactions need to be requested, from, from the peer, by, if you, if you can, if you can tell this transaction actually is like a full RBF replacement, and I, I haven't seen it before because I rejected it, or I have seen it and, and rejected it from my mempool because I wasn't allowing full RBFs. then you can also, you can also get it from there. Having multiple nodes and being able to run them in different con-configuration makes us, makes us much easier.

okay. And so bringing it to the more recent, let's say, debates or arguments, I guess the subset summer is an interesting one, and also the- Operate on limit or a default change. so what pattern did you notice there on the, let's say, on the SubSat summer? I, as, as, as I'm recalling, you were saying there was that same dynamic, right? Like Mining pools had shifted first, block reconstruction rates went down in terms of, you know, the good like what we want, and then after it became, or at least after more and more people started changing their own setting, would you say the block reconstruction rate came back up or what are you seeing?

I, I saw it when I switched my monitoring nodes, to a different configuration. So if I, when I turned on or when I included, DPR for, the lower default, for example, or when I switched my node's configuration to a lower default, then I saw it immediately maybe on the next day or so, when I had like the lower fee rate transactions i-in my mempool, I was able to reconstruct these blocks, with a way higher success rate without needing to request like a additional additional transaction.

Interesting. So, yeah, so you're saying it really did make a difference on block propagation and, the block re- block reconstruction rates? but I guess it

definitely, it definitely did, it improved the, the reconstruction rates. And then again, we're not hundred percent sure this is the only and the biggest factor of block propagation. So we think there, there's, there's different, different things that affect the speed. Yeah, yeah. Yeah.

Okay, fair, fair enough. so I'm just trying to think through the implications. So then, y- now in the, you know, op return case, did you notice anything there, or did you think it was just too, like, not-- there weren't enough op returns that it made a difference?

I think it was the same thing. I, I don't recall how many of, of, I, I, I don't, I don't think we saw many of these, really big, of returns. and even not many of the multiple of returns in the same transaction. but just having one is enough for, for needing the extra round trip. so switching it, switching it over also showed that it improved.

yeah. It, it- Interesting. It, it generally, if, if there is, if there's, If you have like a broader relay policy, active on the network, and you have a stricter policy, you reject some stuff on your node, it's, it's always going to make a difference, because you're always going to have to request, stuff that your stricter policy rejected, but the, the broader net-- the, the broader network policy al- allows. So, or yeah,

okay, yeah. So, and I guess bringing it to some of the kind of arguments online about, you know, even nowadays, the people in the no'ts camp would see that as Maybe they, maybe they, they would argue, now, I don't know, I'm, I'm not personally in that camp, but let's say they're in the camp of, you know, it's not a big deal, it's not, it's not changing block propagation by that much, whereas maybe if you're in the core camp or other camps, you might sort of see it like, no, that is important. Is that how you think the, the different, the different camps are seeing this or how are you seeing this?

yeah, definitely. So, so I Differently, I, I, I think, I think only like looking at block progression isn't the biggest, biggest concern. It's more, about how, like, how is, how is the additional up-returned space going to be used? and how is, the multiple up-returned outputs in a transaction that we have now, how is this going to be used? I think this is where the biggest divide is, and, well, that, that this affects, sorry, a different point of view This is effect, compact block relay or compact block reconstruction. That's just, just a secondary effect. And obviously that's, that it gets tied into the disc discussion as well, because on one side it's a, it's a pro and saying, okay, this actually makes Block, relay better, and the other side says, "Hey, doesn't matter too much, but I, I, I guess it's not the main point, to argue about. It's more the, the point is more to argue about,

yeah, does this, does this change make sense or does it change, change, doesn't make sense for, based on my individual like, preferences for the network. and I, I, I guess I try to be, most Firstly, in some way neutral on this, saying, "Hey, I have the data and the data showed this. do whatever you think, is, is, is right with this data." Obviously, I can't be hundred, hundred percent neutral on, neutral on this because I definitely have my own preferences and, and, and ideas, about this, but, yeah. Another argument

that probably would be interesting to get your perspective on is, I think an argument I hear and see in the Knots camp is, "Oh, it's a good thing to punish spam miners, and therefore if their block propagation gets worse, then that's a good thing from a perspective of stopping spam." What do you, how do you see that? Do you agree or disagree, or how are you seeing that?

maybe let's circle back to the, to the, topic we started on initially, like is our all fake transaction, transactions are they spam?

Probably not, but, I guess who, who defines what is spam and, who says this transaction is allowed and this transaction isn't, isn't allowed? Obviously, each mining pool or each block template producer, can define this for, for themselves. but, I, I guess as long as it's, it's, it's a valid Bitcoin transaction, you can try to censor it on, on like policy level, but it, it's going to make its, its- Straight into, into a block eventually, at least hopefully, and, and that's, that's perfectly cycling back to the, to the, topic we had initially. So you, you're filtering it, but you're not censoring it from e- ever getting confirmed in, in, in, in a block. Interesting,

yeah, because I, I can imagine people might- Yeah. Yeah. And I, I've seen people talk about this concept of a toler-tolerant minority, right? This idea that- As long as there's even a few nodes that are more, let's say, liberal or permissive in what they will relay, it's just gonna be very difficult to stop that and eventually getting to a miner, and if that miner wants to mine it into a block, or the mining pool, or the block template creator wants to put it into a, a block Then there's very little that can be done to stop that. But I think maybe where some people would disagree is they would say, "No, actually there are certain forms of spam. Maybe the, you know, the argument is more like, you can't encode ways to actually stop this spam. I don't know, maybe that's, yeah, maybe that's kind of how, maybe that's how I'm seeing it at least. I, I see it as like, yeah, some of these things are spam, but I just don't think there's a good way to actually stop this spam."

Yeah Not going to sit name names, but any of these transactions that people call spam, but I don't see a way of, of stopping, stopping them from being included without building like the infrastructure for filtering, other transactions as well,

I see, yeah. Yeah, I think those are the key points probably to cover there. What about the Bitcoin Network Operations Collective? I know that's a new, idea that you are- Floating or putting out there. Can you tell us what is the Bitcoin Network Operations Collective? What is this idea?

Yeah, so it, it goes all back to the same, topic I touched on before. I, I think there is way more monitoring work, that can be done for the P2P net-- for, for the Bitcoin network, not only the P2P network, for, for Bitcoin in general, that might be forks, it might be looking at, Electrum servers that are run by Genesis and, and others, other Interesting, topics to monitor or lo- to look into in, in, in terms of data. And what a big company would do, they would spin up or would hire people to work in like a network operations center for, for their company or maybe like even, a SOC, like a, security operations center for, for their company. but, yeah, the Bitcoin network isn't really a company, we can't really hire people to do it and, people need to be f- like freely join, such a, such a no what, what, like, what I, I call it, I call it like a, like a Bitcoin network, operation collective, so the, the B-noc. it's, it's not the noc in, in center, it's, it's, and the C isn't a center, the C is collective. so, different, different people, maybe from a uni-university or different grantees or different, Bitcoin core engineers or, what else, who, whoever is interested in, in, The monitoring efforts, can, can join and discuss there, what, what I set up is like a forum for it, so, people can go there and post, for example, their observations, even if it's just a simple, "Hey, I saw this, did anyone else see this?" Just post a few screenshots or IP addresses or maybe you look, from, from your node and you say, "Hey, I saw this," "Uh, anyone else noticed this?" And then, maybe, maybe someone will come along and say, "Hey, I saw, I saw this as well. I was wondering, I, I have time, I can dig into it." and yeah, I think in, in general, like have a place for people who are interested in monitoring, the network, and, and providing data or requesting data or so for them Find themselves and, and, and communicate and so on. Excellent. So,

where can people go if they wanna, you know, find, information or contribute in some way with the Bitcoin Network Operations Collective?

So, so the, the website is pnoct dot x y z. That's, that's a forum very similar to Delving. It's, it's not Delving, because it's, it's meant for, these raw posts, these raw observations, and if, if you in the end turn out,

to, to Maybe post it to Delving, but first go to bnocht dot x y z and there is a couple of people there, there's a couple of topics. I spun up a couple of topics that other people spun up, on just general network observations or their papers, their reading that tie into this or maybe data they have, they can share or maybe data they can request, or they want to request from others. So if you're into data in the Bitcoin network or monitoring and Bitcoin network, I think this is going to be Yeah, a treasure trove for you, to explore.

Excellent. Well, I think that's a great spot to leave it there. So listeners, first off, make sure listeners you share this episode with, especially with any Bitcoin, Bitcoin developer friends. Go to b10c dot me for B10C's blog, and as, you mentioned, b n o c dot x y z for the Bitcoin Network Operations Collective. B10C, thank you for joining me today and, sharing your insights for the listeners.

Yes, thank you for having me, Stephan.