NVK Bitcoin Hardware Wallets vs Air Gapped Computers

Hi and welcome to the Stephan Livera podcast, a show about Bitcoin and Austrian economics. Today we're talking about Bitcoin hardware wallets versus air-gapped computers with my guest, NVK, the CEO and founder of CoinKite. So in this episode, we get into the concepts around locking down security and what attacks are possible against air-gapped computers or also- So hardware wallets, and we also get into multi-signature vulnerabilities and standards, as well as helping new coiners in the space and helping them balance trade-offs. This show brought to you by swanbitcoin dot com. Bitcoin has emerged as a major player on the global stage, it's been significantly de-risked, you've seen all these recent investments and announcements, at this point everyone should own at least a little. A common way people get started is establishing their initial position with a one-time buy, and then start dollar-cost averaging with automatic recurring buys. Swan Bitcoin was built To do just this, with Swan, you create a recurring purchase plan like a hundred dollars a week or twenty dollars a day, and you can also make one time buys. So you can use bank wires for large amounts or ACH transfers for smaller one time buys. Swan are available in all states and territories of the US, including New York, so they are a great place to send your friends and family when they are ready to start buying Bitcoin. Send them to swanbitcoin dot com slash livera, and they will get ten dollars of Bitcoin dropped in their account when they sign up. NVKite dot com are the creators of my favorite Bitcoin hardware wallet, the Coldcard. This is one of the most recommended hardware wallets by Bitcoiners. I think it has some incredible features, such as the ability to use it airgapped. You never have to directly plug it to the computer. You can use a micro SD card. You can also use Dice to add entropy, or you can bring your own entropy. It's got a whole range of features. It's really a leading wallet in the market. They offer PSBT, partially signed Bitcoin transactions. And it can work great as part of a multi-signature quorum. Also, there's all sorts of features, go and check them out. Get yours at coinkite dot com and use the code livera for a discount. Compos is an online marketplace which makes it easier for everyone to mine Bitcoin and enhance the Bitcoin network's security. The anti-cloud mining option, Compos helps you buy your own ASIC and secure hosting at great facilities around the world. For years, we've all heard that mining is only profitable if you're investing tons of money, but now with Compos, everyone is able to tap into economies of scale and access reasonably priced hardware and cheap industrial power rates. If you're unsure about how to get started with mining Bitcoin, Compu offers hardware and hosting bundles which eliminates the need for advanced technical knowledge and allows you to quickly get started mining Bitcoin with hardware you own. Visit them at mine with compass dot com and start mining Bitcoin today. Onto the show. N V K, welcome back to the show. Thanks for having me. Mate, things are getting really crazy now, and, it's obviously there are a lot of new people coming into the space, and they're all asking a lot of these questions around how to secure their bitcoins. now, and a really interesting topic that has seemed to come up over the years is this question of air-gapped computers versus hardware wallets. So maybe high level, can you just tell us how are you thinking about that question?

Yeah, so surprisingly enough, there's been a lot of discussions even sort of like this last couple weeks about this stuff. Every time the price goes up, right, everybody revisits their security and they should. so like air gapping, computers in general is one of the best practices of keeping things secure, right? Like this is, this isn't a new, thing just because of Bitcoin, it's just Bitcoin leverages this technique, right, of- what it means is that you don't have the device connected to the network, right? So it's like, think of it as a mold of air around it, there is no cables, right, aside from power. And what that means is that, it makes a lot harder for an attacker to have a synchronous, means of retrieval of an attack, right? 'Cause imagine that like there is no perfect device, there is no device that's like doesn't have any bugs, that doesn't have vulnerabilities, that doesn't have weak spots. Everything is hackable, right? Things just have sort of degrees in which they defend against which attacks, right? Now What happens is, say you have an unknown bug on a device, right? If the device is not connected to the internet, even if the attacker knows that vulnerability exists, he's not able to get in there remotely and retrieve whatever information he's seeking, right? That's sort of like the goal of airgapping.

So we could think of it, and I think one analogy I've heard is imagine you've got two houses, one house has ten doors and twenty windows, and the other house has only one door, and that's the only way in and out. That's perhaps a good analogy to what we're trying to achieve here. And so the question then is, there are some people in the community who believe that, you know, you should not use a hardware wallet because you can air gap a computer, and then there, I think probably the mainstream view is probably more like, "Look, a A specially designed Bitcoin computer, it is specially locked down. I guess that's probably one high level way to summarize what's going on, but, maybe you, if, like, if someone was to try to air gap a computer, what is actually involved in that process? And, you know, what are some of the pitfalls in trying to achieve that?

So, so the idea is, so, so let's just talk first about like general purpose computer versus hardware wallet, right? So what are they offering? So a general purpose computer Computer is gonna, is gonna protect you against, say, a, specific purpose device vendor vulnerability, right? So nobody knows what you're buying the laptop for, right? If you go to, say, Apple or Walmart, whatever, and you buy the laptop for cash, right? Nobody even knows you have the laptop. So, nobody is gonna be targeting you directly for Bitcoin. So that's one thing, right? Now, with hardware wallets, well, those are Bitcoin hardware wallets, right? So, the attacker knows what is going for, right? So, that, that may be a rogue person on the vendor side, that may be, carrier, shipping the device, So what happened in that sense is like, then, okay, great, so these are sort of like the, the more-- And I'm being very simplistic, right? so, so those are the two sort of like, advantages and disadvantages of this general purpose versus single purpose devices, right? So then you have to look, right? what steps does the vendor of the hardware wallet takes against those attack scenarios, right? and, well, that wouldn't count On the airgapping, laptop or the laptop itself, because, you know, nobody knows you have that. But, with hardware wallets, say like Coldcard, I can just talk about my keys. you know, we have the tamper evident bags, we have the firmware that you can reload, we have the, the, the anti-fishing words. So, like, we have like a slew of protections, none of which is perfect, but combined they, they offer a great level of, of such fat to worry security, right? So- That you're protected against sort of like vendor, specific attacks or supply chain attacks, right? Now, with the laptop, on the other hand, well, anytime you buy a laptop of Walmart, it's gonna come with like a ton of garbage, right? Like all kinds of essentially like, you could almost say it's malware by the vendor, right? you know, right? Like when you buy a laptop, you have to uninstall a bunch of stuff, right? Yeah. So yeah. And, and, and like, you know, an average- person is only gonna be able to one remove so much of that, right? And two, even if you wipe it clean and you install something from scratch, an average person isn't gonna understand really how far you have to go to lock it down. Right? While I understand why some very advanced people, totally wanna go that route because they are completely avoiding the trust in hardware wallets and they also are very capable of locking down a laptop, but even in their case, honestly,

A laptop has a football field size of attack surface, right? Because it's a general purpose device, right? There is a microphone, DSP chip in there, right? Are you drilling that out, right? And, and every single-- even when you talk about open source, like, Linux on top of a laptop, all the chips in that laptop, like, not all, but like most of it will be closed source, right? There is even the, the, the boot ROM, right? Like every- The thing in there is essentially an attack vector, right? While a hardware wallet, not being perfect, it is a very simplistic device, right? Like single purpose, oftentimes, at least in our case, there's just two chips in there, the rest is just sort of like electric, And, what it does is it just gives you a lot less stuff to have to worry about being attacked, and, and, and it's a honeypot as well, right? So there's a lot of security research gone into trying to break them. I mean, that's, that's like all that, like, is done to them all the time, right? Like, it's just people trying to get in and then show that they can get in into one of those, right? so, so that's sort of like a, a, a big difference between the two.

Right, and so just hypothetically, let's say somebody wanted to do airgapping a computer, what would that process look like? I guess off the top of my head, it would be things like, okay, trying to strip out the Wi-Fi, strip out the microphone, strip out the Bluetooth, like physically, trying to take, trying to physically remove those things from the laptop device. And then you might need to use some kind of live boot system, for example, Tails OS, or maybe you, you know, you're flashing something else on On there to actually do your Bitcoin things on there. Can you just outline a little bit what would that process look like?

So in my opinion, if you're gonna do a, a, a laptop for Bitcoin, you wanna start with, like, I, I wouldn't even use most distros of Linux anyways, especially not Ubuntu, because it was meant for people to do like spreadsheeting in Word on a Linux version, right? You would probably pick a security, minded distribution that has- Last cruff for you to clean out and last things for you to, to lock. and personally, I would go for FreeBSD, right? It's a, it's a very secure, platform, And, it starts essentially with nothing in it, and you add only the stuff that you need, right? As opposed to starting with a lot of stuff like Ubuntu, that then you take stuff out, right? I-, i-, it's sort of like it's a lot easier to make something safe when you don't have to take stuff out, right? and then of course, you know, you, you try to sanitize a lot of the hardware by taking stuff out. in most modern laptops, that became very difficult because a lot of things are single chip,

and, Delicate electronics, so it's not that simple. I, I think it's, it's, it's very unrealistic for the average person to do this at a satisfactory level, right? a-and then there is the concern about, for example, and, and this why I like freeBSD versus, you know, whatever distro you're gonna use of Linux, is that like you're gonna have to choose, like, exactly the library and build from source and check each library for, say, the disk encryption, right? Like, are Just trust the disk encryption that comes with the distro, right? these things are not simple. Like, imagine the millions and millions and millions of like little libraries and packages and things that, that like You know, people in the Bitcoin community would say like, "Yeah, you know, build your hardware wallet's firmware from source, right?" Well, you know, that's a pretty tiny little piece of software comparing to a massive OS, right? So it is just a lot less stuff to go over.

And so I think the other point is using it in practice. So again, it depends what the person is using for. Obviously, we're speaking mostly in the cold storage case. However, if that person actually needs to periodically spend out of that storage, let's say they're doing a big, it event they need to spend some coins, well, now they've actually got to facilitate that process of, you know, getting a transaction in and out of that air-gapped computer. And, you know, the problem then is how do you do that in a, in a user-friendly way? Because, well, theoretically, this is an air-gapped computer, you don't want it colle-connected to the internet. So now, how do you get something in and out of there? Well, maybe you do a USB, but then what software do you use? So it's kind of, I mean, maybe

To spell out here is there's some practical difficulties involved with using these, I guess laptops as a-- or air-gapped laptops as a hardware wallet. Now, I guess historically, and I've, I've used some kind of setup similar to this, when I was doing an Armory wallet, this is, you know, years and years ago, but do you have any thoughts around the practical components of using a, an air-gapped laptop?

Yeah. So I, I think just recently Bitcoin Core added some support for, for air-gapping, The node so that you can transfer data, on and off, right? 'Cause you also have to keep, blocks up to date, right? And normally you keep that computer connected to the internet, right? so, so it's kind of complicated that way. I, I find it personally, I find it impractical. and then what happens when you remove some of this practicality, right? Is that like it adds more room for people to mess up, right? It means that like you, you are gonna have to, to do things that are a lot less natural and a lot less sort of like,

a single purpose, right? So with the hardware wallet, the device was designed for that, is simple, so you, you know what to do, right? Like you, you, in Coldcard's case, you know you can do a gapped, so you're gonna do the, the sneaker net with the Micro SD to shuffle PSBTs in and shuffle transactions out, right? While with the laptop, you're gonna have to, to choose, right? Like, are USB, USB is a monumental cluster, and, yeah, i-it's just, it's just impractical, right? And, and I think that opens people to mistake, and people, a-as I see this market grow and like for quite some years,

most people screw themselves out of their coins more than they, get robbed, right? And one thing that was very common back in the day when there weren't no hardware wallets was for people to get robbed. So, I, I think, I think it's, it's kind of interesting that way.

So yeah, you make a lot of great points there, NVK. now, I think a really cool website, now this is your website, but listeners, go and check out airgapcomputer dot com and NVK, do you want to tell us a little bit about this website and maybe tell us about some of the attacks possible?

Yeah, so, you know, people love to talk about like air gap, computers, but I, I think a lot of the people don't understand like how much, you can do against a laptop, even side channel attacks too, right? Like where the laptop's not even connected. Mind you, a lot of these attacks will require like physical access to the device, right? but like you can do, for example, Right? Because the, the, what happens is when you're running a laptop, the, like all your, your keys normally go into, go into memory, right? So what happens is, let's say, you turn off the laptop what you can do is you can freeze the RAM, right? with like actually very cold, and then you can actually read the RAM. So, so you have access to some of the data. There is, of course, there's a lot of software that tries to, to, to do better against that, right? But, but there's a lot of those little sort of like, accesses. So, for example, another one, they can read the RAM, w-with the, the Wi-Fi card, so they were able To, to, to see, via RF, right, radio frequency, the, the, the data being calculated inside the RAM. So they were able to take data out. another cool one was, and this is actually quite s-it, it, it's simple, but it's not. It's simple in this context. Is, I love this one. so they, you know, there is this, this very, long distance, sensitive microphones that essentially it's a laser that they point at, say, your window because the glass, resonates right with sound, and then they can actually listen to your keyboard And then as you type, they can figure out which key you're typing to get your password, for example.

These are so scary, man. These are just crazy in terms of like what kinds of things are possible, and now side channel attacks are- comment, like they're an attack that people, a lot of researchers talk about them, and in fairness, that these, there are side channel attacks versus hardware wallets too. So maybe if you could outline a little bit about the possibilities there in terms of side channel attacks versus hardware wallets versus air gap computers.

So, a, a great attack for, for side channel was against the Trezor, this was many years ago. somebody, managed to do a power differential, power differential, attack on Trezor where they- We're able to read from the power, port, the device doing, cryptography on the MCU. There wasn't a lot of protection at that time on that, but the, you know, that was fixed. another one is, there is this, this, this, this thing called, chip shouter. So essentially, you use, very, very specific, very targeted, RF, pushed, so like radio frequency pushed into a little area of the chip, and if you do it just right, you can actually, on a, on a treasure dump the whole memory of the chip, so you get the seed essentially, right? I guess Coldcard, there was a good one, the guys from Ledger, essentially removed the, our, secure- Element, our old secure element out of a cold card, and they, no, they didn't remove from cold card, but they, they used the same type of SC that we used. and they found a flaw on it that this SC is designed so that you can't peel the top, to try to read the data inside. So they turned it upside down and they dug the bottom of the SC, they pointed laser, very expensive machinery here, talk about quarter million dollar equipment, pointed like a laser to a specific gate of The SE and send a very specific signal and we're able to trick the, the PIN, counter of the SE. so, you know, it's kind of theoretical in a way because it wasn't sort of demonstrated the actual device per se, and we keep the seed, encrypted in the other chip, but still, like, you know, it just, it just shows, right, like that, like, given infinite resources, nothing is unhackable. a good one on actual computers that happens quite a bit now is, you have malware on the, say, the firmware of a hard drive. So you wipe your laptop, you do all this stuff, but There is the, the firmware on the, on the hard drive itself that you think is doing encryption or doing something else, like there's, there's malware in there, right? And it, it's not like the kind of firmware you even have access to update yourself, right? There's a lot of stuff going on, that's why like ultimately one of the best defenses you have is to simply never plug a device to any other device. So on, on, on the hardware case, it's like you use the micro SD to shuffle the information and on the laptop, that's ergap, you essentially never connect the laptop to anything.

And of course, there are still trade-offs associated with that, but it's kind of like the idea is that's the most kind of reasonable balance usability-wise to still give you- Some level of security, but yeah, certainly some of those side channel attacks that you're outlining there are quite scary in some ways, like, the whole differential power analysis thing where, you know, people can try to assess based on the power levels what the device is doing, because remember When you're-- when you are using a hardware wallet, what you want it to do, obviously, it is storing the private keys, and then when you a-- when you go to actually sign a transaction, it's signing, it's doing some operation to sign a Bitcoin transaction so that you can then spend. So if they are able to monitor you in that Vulnerable moment, that is potentially where there's some leakage, right? Exactly. Of information.

Exactly. And, and then like, and then I, I like to like also sort of- Bring this all back down to sort of reality, right? There's a lot of these attacks either require, physical access or extremely targeted, attacks against you, right? And, you know, you're not Snowden jumping hotels, right? Running from the NSA. So chances of a lot of these attacks are much lower. And, y-you, you know, so I, I think that like, you know, it's been like over a decade now of like Bitcoin and protecting Bitcoin, so- I think that the space is still early, but it's like matured enough that people aren't losing money, right? So, the way I like to say it is kind of like, you have, for example, we need to stop scaring noobs into self-- like from self-custody, right? I, I feel like, There is just too much FUD, around, the, the, the solutions for security right now, especially from extremely sophisticated people, right? Like, I understand why, like, a person who's like a core dev or something is extremely sophisticated, understand computers very well, would say, "Hey, you know, of course, I can see all these attack vectors on a hardware wallet, right? Or, or on a laptop." So this person understands how to- Remediate these things, right? but we can't expect an average person, to do any of this stuff realistically, right? So what we need to do is sort of like, we need to teach them, not, not make them dumb, right? We don't wanna send them to centralized wallets or to wallets where the validation is done by the vendor, but, but we want solutions that are sort of a little bit more self-contained, right? So, you know, say you just sort of got some Bitcoin on Coinbase, right? Or, no, let's say Kraken. you, you know, send those, those few, like a few hundred dollars, couple thousand dollars to your phone wallet, right? And but make sure you're using one that's like, non-custodial, right? So you have a backup of the seed in metal, right? So that's a nice way of you just getting your feet wet with self custody, right? And then sort of like, whenever you're ready, you know, you, you can sort of upgrade, right? And then like the next thing is like, you know, do the Bitcoin Security Guide I'm ready. I'm gonna like take some, some self custody to the next level because I have a little bit more money, right? So, you know, you get a hardware wallet, you get a metal plate, a-and you get that set up, right? and then you learn more, and then you add a passphrase, and then you learn more, maybe you go multisig, right? But let people It's like follow a path that is like realistic and it's, and it's, and it's incremental as opposed to like, "No, you have a hundred dollars? No, you have to do five out of seven multisig with ten laptops that are gapped." It's, it's, it's completely-- You know what I mean? Like people go, "Like, no, Coinbase is better," and I can't blame

them. Yeah. And now, I guess one other point that maybe some, people out there might recommend is they might literally say, "Okay, fine, we can't-- even if you can't air gap the computer, what if you literally bought a computer only to run Bitcoin Core on it? And if you just, like, say you connected to the internet and, you know, you just ran Bitcoin Core and nothing else on this laptop?" What's, what's wrong with that?

Yeah, so I, I, I really, I, I think it's, it's a great idea to have a computer that you use for Bitcoin things, including that's the computer that you use to talk to your hardware wallet, right? it's nice to have a computer, you know, like the, the old adage goes, you know, don't shit where you eat. So like, don't mix in your everyday, you know, like, like, chatting apps or your day, everyday browsing with the computer where you do your Bitcoin accounting, where you do, like, you know, your banking, for example, right? Because that computer have less chance of having viruses, right? Viruses are still a concern even when you have a hardware wallet, because the, the desktop wallet could be pwned by the virus, and then lie to your hardware wallet. But at the end of the day, the last line of defense is you, right? At least with hardware wallet, you can look on the screen, right, and verify the transaction is going where it needs to go. Right? And then you approve it. So that's sort of like the last line of defense there.

Also, there is some discussion and one of the concerns people share around hardware wallets is, "Oh, the crypto library, is, maybe it's not using the Bitcoin's crypto library, cryptography library this is, and maybe there's not as many people reviewing the code on those hardware wallets as opposed to Bitcoin's cryptography and Bitcoin's libraries." How would you respond to that kind of concern?

so, okay, so I think there's a few things first here. So, LibSAC, two fifty-six K one, right? The, this, this new great library that's part of core, wasn't around or, or, or mature enough when some of the main hardware wallets came to the market. Okay, so that's one thing. It just wasn't around. and then I guess the other thing is, y-y'know, Monacrypto, the one that we use, y-y'know, does have a lot of eyes on it. it's, it's, it's, you know, it's one of the most used libraries for hardware wallets. I, I can't, I, I can't, I don't know which one Ledger uses, but, you know, but at least they get it audited Are a different set of trade-offs, right? but they, they, they also have secure elements. So like, you know, each of these, these sort of solutions will give you different sets of trade-offs. Now, I do agree that this, this library, the, the core library is much better. we are actually like transitioning to that, about eighty percent done, a-and, and it's a great improvement, especially in speed. But, but I think, I, I think it's a bit of a fudge Right? Because reality is there is like hundreds of millions of dollars in hardware wallets and, attacking the crypto library hasn't been, at least I haven't seen a demonstration that that has been a concern, right? it's the same with sort of BIP thirty-nine. BIP thirty-nine isn't as safe, let's put it this way, as, maybe other ways of making private keys, but it's just beyond good enough, right? and, and, you know, and so much so that like it became the de facto- The standard, right? Sometimes, actually most of the time, standards aren't the best solution. They're the solution that got the most buy-in.

Yeah, that's unfortunate, but, that's, that's the way, that's the way of the world. So for better or worse, this is, probably what we're stuck with. Also, thoughts around how this market changes as it grows, right? So for example, in the early days, you kind of had to know who to go to to get these hardware wallets, right? At, at, at an extreme, like ten x or hundred x in the number of people coming in. Does, at what point does the market change and, you know, other options become economically viable? I mean, just as a, you know, maybe a silly idea, but do-- does it ever get to the point where people buy hardware wallets from the supermarket or from like a department store or that kind of thing?

Yeah, so I'm not a huge fan of resellers and third party sellers because you do introduce one more hand in between, so supply chain attacks are more reasonable. Right to happen. So for example, e-even buying hardware wallets on Amazon, I think is a terrible idea, right? Because, say, I'm a bad guy, I buy a thousand hardware wallets on, on Amazon, I, I put very low grade, sort of replace the firmware, stuff that, like, you know, it's, steals money, or even I just preload them with seeds, right? 'Cause a lot of the people buy on Amazon don't know better. And then, and then I return them all, right? and then Amazon resells them again. So, so now you, you, you just open a whole new sort of like low grade, but also low hanging fruit, attack on noobs, right? So I, I really, I, I think the future and sort of like the best way of handling security devices is to like go to the source to buy it. I think the verticalized sort of space is, is growing, so like manufacturer to consumer, like you look at like Tesla, you look at Apple, right? Like you have all these companies that are skipping the middleman. Probably losing some sales in the process, but they have a tighter sort of like quality control, tighter security, right? On the, on the, on the process. you know, then you, you know, for the more advanced people, you can also build your own hardware wallet kind of deal. and then I think in the future we might see also, you know, better chips or better, like actual hardware solutions that provide better sets of trade-offs that, like, maybe make it possible to have less cons- About supply chain attacks or, or vendor attacks.

Back to the show after a word from the sponsors. Lend at HoddleHoddle is a global Bitcoin backed lending platform, so you can lend and borrow anonymously on your own terms. This is a peer-to-peer solution, there is a unique multi-signature escrow for each deal, so if you have, if you have stablecoins, you can put them up and get interest on those stablecoins by creating an offer. On the other hand, if you hold bitcoins and you need some liquidity or you want to lever up and get- Get more bitcoins, you can borrow stablecoins and keep on hodling by paying interest. With HoddleHoddle's Lend platform, you set your own terms and put up offers depending on how long you wanna borrow or lend and the interest rate. Go to lend.hoddlehoddle.com. Cyphersafe dot io are producing metal backup seed products like the Cypher wheel, and they've also got a Bitcoin recovery tag product which specifically helps with recovery. This is an extra stainless steel tag with information like the original wallet, gap limits, derivation types, scripts used, and all the major hardware wallets have their own type of recovery tag specifying the data for that type. So you attach this to your seed word backup with a stainless steel cable inc-included. There's even a web Website link so you can help your heirs or yourself recover the coins using Electrum, so it adds the value of helping you recover in practice. So the recovery tag works with any seed word backup device, you can go and buy yours at cyphersafe dot io and use the code livera for a discount. Unchained Capital are providing Bitcoin native financial services, and they're using multi-sig technology, so if you need a way to improve your security, you can bring your hardware wallets over and use them. Use them as part of your multi-signature vault. They have a two-of-three multisig setup, and if you need help, you can actually purchase the concierge service, so you can get calls and have the hardware wallets shipped to you. They'll teach you and walk you through that process and deposit a thousand dollars of Bitcoin in the vault. So normally it's fifteen hundred dollars, but you'll get fifty dollars off if you use the code Livera. Unchained offer all sorts of features, such as business accounts if you are a business, and you can also find a whole- whole range of educational content on their blog, so go and find them at unchained-dash-capital dot com. Now, back to the show. Of course, you know, I'm sure yelling through all of this is my friend Michael Flaxman being like, "Guys, what about multi-sig? We've got to have multi-sig." And I think maybe that become, that space becomes easier as well, although we wouldn't recommend that for beginners and say, "Hey, you've got to go to this level." Like, it's one of those things where hopefully over time, it A really good point now to talk about this recent, multisig vulnerability on the cold card. So, do you wanna just give an overview on what happened there?

Yeah. So, so multisig is, is sort of like fairly, like, complex and full of, like, edge cases, right? So, And then that's part of the reason why, like, as much as it's great to have multi-vendor, multi-sig, it's also not great to have multi-sig, multi-vendor as well, because like you're not testing against each other's sort of like bugs and things, But, so in our case, this specific bug, so most hardware wallets don't, don't keep state, right? So they don't keep,

like information about the cosigner on a multisig, right? So they wouldn't even be defended against this. in our case, we do, but, we had a bug that essentially wasn't checking it correctly. So, you know, you have the sort of theoretical attack where like somebody would build a virus, would say pound, Electrum or, or, or, Specter, and then they would build A, a rogue transaction that would send the money to either a Griffing address or, like, like a hosted addre-address or, or even maybe like a, a, a, a payout address, right, to the attacker. And then, if you had received this PSBT to sign in your cold card, the cold card would say it's okay, however, it would still ask you to check The transaction, right? For then, for you to sign. So you were essentially the point of last defense there. You could have said, "No, this is wrong," right? So, so that's sort of like what the problem was. We weren't, we weren't trusting the XFP, so the, the ID of some of the wallets that could be sort of trickery there. So essentially, we just sort of-- we, we've redone all the checks. It's pretty awesome now. and, and I, I found it interesting because, like, as much as I, I love, like, the fact that it was found and it was fixed and stuff, like how most wallets don't even do this checking, so they wouldn't even have a possibility to have this bug because they just sort of yellow, right? They just s- essentially sign everything.

Yeah. Okay. So let me just, I guess this is probably getting a little bit technical, let me try and explain it in simpler terms just for anyone who's listening. So I guess we can think of it like your hardware wallet is kind of, it, in some ways, it's dumb. It doesn't know what's going on out there in the world. And so that's why we have Bitcoin Core and some kind of coordinating application like Electrum or Specter. And so in this example, Electrum or Specter or one of those wallets, or Blue Wallet, is trying to Feed the hardware wallet to say, "Hey, hey man, I know you've got this private key, can you sign for this transaction?"

And only multisig. So this bug was multisig specific.

Yes, yes. So, in this example, it's a, let's say you d- you did a two of three multisig setup. And so in this example, what you're, what we're talking about here is coldcard registering the other members of the quorum. So let's say you had a coldcard and a cobo and, what's like another And each of those hardware wallet devices has their own, I guess, contribution into the, into the, into the quorum, the set of multi-signature, devices, and That, the bug was essentially that coldcard was not checking that its own, it was not a member of that quorum that it was setting up, correct?

Yeah. So, so essentially we weren't just trusting- heuristic that was not good. There was essentially a bug there.

Yeah. And then what's being fixed now is essentially that it does check that it is a member in that quorum. And also, I think when you're doing that multi-signature setup, it, you can sort of scroll through and if you're just manually looking at, okay, just kind of checking the first few letters and the last few letters of the three xpubs in that example, that would have also given that user some level of protection there, correct?

Yes, yes. I- Again, right? It's one of those things, it's like, y-you know, is this attack possible? Yes. Is this practical, realistic in any sort of time frame between being reported and being fixed? No, right? So, you know, this is the kind of stuff that's like, you know, you, you, you, you, you receive all these sort of like, disclosures, you fix them, you release a fix. More often than not, you don't tip the bad guys, right? So you release a fix with a firmware quietly, right? You, sorry, you, you add that to a, a feature, a, a update? It depends on the case. On this one, it makes that sense, right? And then what you do is you wait for the disclosure period with this firmware out there, and then, and then on the day of the disclosure, you tell everybody to update, but that means that most people have already updated. So that you don't tell all the bad guys, "Hey, listen, there is this open bug out there."

Gotcha. Okay. And so in terms of, you know, going forward, I know there was, a couple response blog posts. I know, for example, Hugo, did, Hugo, of the Nunchuck project, did, a blog post, and of course, CoinKite had a blog post as well, kind of just talking through some of the basics and what's going on here. But in A multi-sig standard. So do you wanna outline a little bit about that? What is that?

Yeah, so, so the, the problem is inherent to, to the current state of things, right? There was, like, we're sort of like going above and beyond here to try to To create some trust between the devices on the quorum, right, of the multisig. And because there is-- there, there is no secure way of doing that, we end up doing all these convoluted ways that just open yourself to more possible bugs, right, or possible exploits. So, Hugo had this, this great idea of creating a standard for creating a quorum, right? Coldcard already does that, well, did something like that between cold cards So if you're just using cold card to cold card for multisig, you're a lot more protected. But the, the challenge here is when you have multiple vendors, right? Creating a multisig between each other, right? Like we all have different ways in which we wanna do things. as, as everybody who works in project knows, everybody thinks that their way is the best way. So, so, so this standard is, is great because it's-- we're calling it BSMs, Bitcoin Security Secure Multi- Sig setup. So what it does is it creates just a little standard, it's, it's actually quite small, for the, the wallets to create a secure session with each other. So essentially they have a handshake between each other at the time of, of creating the multisig wallet. And, and then once they have a secure session between them, a secure channel, let's put it this way, between them, they can then trade the secrets and important information, right? So they can trade the xPub, they can trade all other meta information of the, of the multisig, like the redeem script, right? And then there is no room for somebody else to get in there and try to lie about what's set up with what.

And so this would just be part of the new setup? Up ritual, if you will, or set up ceremony, and essentially it could be conveyed the same way normal information is conveyed. So let's go back to that example, let's say it's Coldcard plus a, Kobo Vault plus a Specter DIY hardware wallet doing a two of three, and so in the case of the Coldcard, you would still ferry or shuttle that information from, say, Specter wallet into your Coldcard using the SD card, and then the other guys, you would do probably the QR code, scanning, and that's how you would, they Ingest the info. So maybe, for example, Specter Wallet would show a QR code, and then the DIY and the Kobo Vault would scan that and then read it, and then the user at that point can, I guess, check, themselves and say, "Okay, yep, do-- does-- does this quorum look right to me?" Correct?

kind of. Yeah. They, they would be essentially-- well, all, all the standard is, is, is creating a, a, a secure channel between them. That's it. Gotcha. Got Other standards could be whatever, ideally is descriptors, then, then that information, the, the setting up information cannot be fudged because it was sent safely between each other.

I'm also thinking now as well back to my conversation with Michael Flaxman where, and also his, btcguide dot github dot io where he talks about, you know, multisig. One of the points there is that you should theoretically, you know, depending on how secure you wanna be, you should be checking your receive address anyway. So one way To do that, I guess, is to check that does this, address belong to me? And in, for example, the Coldcard Address Explorer, you can do this, and on some of the other, Like those hardware wallets with the screen and so on, you can go through and actually see, does this address belong to me? So that's another way. Yeah, make sure,

make sure whichever hardware wallet you use can, can show you on screen deposit addresses, right? You need to be able to double-check on screen that the address that you're receiving Bitcoin to is controlled by you. Like that's, that's one of the most paramount things, a-and because there will be more attacks, right? There will be more vulnerabilities, and there probably are some out there, right, that we don't know about. So, The buck really stops with you, so like if you're just double, triple checking stuff, you know, y-you're safe, right? because again, like everything is hackable. I, I, you know, but I can't repeat this enough, right? Like, all this stuff is very targeted, it's very complex, like most people in most circumstances aren't exposed to a lot of these attacks, and, and the alternatives are less safe.

Yeah, I think that's a very crucial point. I think it's one of those things where, you know, in the earlier days, there was less awareness about these things, and so we were just kind of running that risk without even knowing it, right? Especially for me, when I was a total newbie in the space You know, like you're just using, you know, for example, things like Trezor Web Wallet, or you're just using, you know, Armory or something, and without knowing about all these kind of, "Oh, I need to verify the receive address," and I need to, you know, these things that you learn and you pick up over time, and you, you sometimes, hey, I'm, I'm pretty sure most-- if you talk to anyone who's been around the Bitcoin space, they've got battle scars, right? They've lost coins on something.

Yeah, it

Of an exchange from the browser, right? Yeah, exactly. With all the extensions on, I mean, it's crazy to think, right?

yeah, I, I think a lot of it is like also fault of, of the market, like I, I think there's just like too much sort of like weaponized PR between competitors and things like that. I, I, I really think that like it's a shame really that, that, like Y-y-you're gonna depend your, your, your sales on sort of like attacking like competitors on, on like vulnerabilities, right? Because you're essentially just eroding the trust in these devices in general. a-and, and I, maybe this is an ask for any security researcher listening, I know that like a lot of the stuff you do is, is incredible, and you want visibility when you find a bug or something, but you have to understand that like most people don't know the context of like what it takes to maybe achieve one of those attacks or that it's not sort of like everywhere all the time. And, and then when you do responsibly disclose, I think the market has grown now enough, there is enough- Like so many players now and so many projects depending on things that simply emailing, like one manufacturer who, who did find the, like, I don't care how much you like or dislike, right? 'Cause this isn't about the manufacturer, this is about the users. So I, I think it's a time now where like, if you wanna be a pro at security research, you open a CV, and, and you do the process, the, the, the, the traditional way Of creating responsible disclosure, right? Because, y-y-you know, you will probably affect other projects that aren't necessarily, like a hardware vendor With your disclosure, you know, it can affect a lot of things, a-and then, and then you just essentially like leave users there. Well,

there are obviously a lot of rivalries in the space, and so often it's hardware wallet manufacturers, slinging things at each other, or it's security researchers, and I guess the, the bad case is when vulnerabilities just get openly, publicly disclosed out in the wild, and then they basically force the hardware If the, if the hardware wallet manufacturer is blindsided, now they're scrambling and trying to quickly fix this and patch it and put out a fix and try to get their users upgraded so that they are now safe against this attack, while at the same time, it's like there's a funny incentive in some ways where some people who are disclosing a vulnerability might, let's say, overstate the- Feasibility of running that kind of attack because more press for them, more publicity, we kind of live in this crazy sort of attention economy, so it, it, it, you have to I guess it's a difficult thing, and I can understand for a, a new person coming into this space, that they don't know how to maybe assign the right level of risk. You know, they don't have the right level of context to sort of understand, okay, is this kind of more like a edge case kind of crazy, you know, freak scenario, or is this kind of like a really everyday thing and like a lot of people will lose money because of this, and you have to try and assess that.

Exactly, right? And ultimately, the person who, who, the, the entity That chooses the severability of a vulnerability is the target vendor. The target vendor is, is like sort of like the ultimate source of truth there, even if they minimize it, right? then they would just lose trust. But, I, I would always take with a bit of a grain of salt, the, the party who's disclosing it. I guess you take both parties with a grain of salt, but But mind you, right? Like, when a bug or a problem is big enough that is an huge problem, like you will hear in the wild people already losing money to it or something, right? That, that's very likely scenario. I, I just think that like, we're now grown up enough, there's enough people using this stuff that, like, a, a lot of the security vulnerability disclosure, like, needs to be toned down a bit instead of like- Sort of, you, you know, like, "Oh, I'm gonna put on the blog of my company," because like you're not gaining that much anything, right, with that, and everybody's losing a bit with erosion of trust on these devices. I, I think that's very different than we all sort of like shitting on each other on Twitter. I mean, like, that's like fine, you know what I mean? Like You know, you can say, "Oh, you know, I find your stuff to be crap, or I find your stuff to not be that safe, or whatever, right?" I mean, so be it, right? Like, we're all human and we have our concerns or, you know, concerns or whatever. But I think on the actual formal sort of like disclosure, and on the, on the blog posts, like, things need to be sort of like toned down a bit because they, they do scare people, and, and, and reporters are completely like ignorant about like these also like hyperbolic, with a hyperbolic sort of view just to get clicks too, right? and we see this like everywhere, all the time in this space.

Yeah, it's a very tricky thing. So I can definitely appreciate for somebody who's new coming into the space and they're told, "Don't trust Verify," but then, okay, then who do I-- what, what do I even, what do I even do? Well, who do I look at? What do I-- I need some guidance here. And so that's a typical concern that I can Into this space has, and obviously I'm try- I'm doing my best to try and guide people into kind of safer pathways, but I'm wondering, do you have any tips around how you help guide newbies in the space, you know, without kind of scaring them off?

So first is like, look who actually- Like the people who are actually sources of, of not sources of truth, but sources of some truth or at least a trusted point of view, right? If somebody claims to be a security researcher, like where's the security research for you to review, right? Like otherwise the person is a security pundit, right? So there's a big difference there. you know, people have opinions, but are, are they like producing something that like sort of creates, some rep on them, right? Like do they have- Have a trusted project, are they involved in a similar industry or a similar s-part of the stack, right, for them to have an opinion? you know, or maybe other people who are trusted in the community freaking out about the specific bug, not about the rhetoric around the bug, right? not like around sort of the drama around it, but like are they actually concerned about the vulnerability, right? Are they saying like You shouldn't use this device anymore or you should update immediately, right? What are they saying in that sense? But the best heuristic is, when it is disclosed, right, was it fixed, right? So Does, does, do both parties, sometimes they may not agree if it's a, like a vulnerability or, or a feature, right? That could have some, some-- I'll, I'll give an example there later, but, Does, does the party who gives you the device, most importantly first, right? Do they, do they say that it was fixed? If it was fixed, it was fixed. Everything else is just drama. Right? Yeah. if it wasn't fixed or it's not fixable, there are some major concerns with some devices out there, you know, it's not fixable. Now, is this something that you're okay with? Going forward, right? Like, you have to make a decision for yourself if you wanna have a device that has a, say, a major flaw on it or not, right? Well, maybe that's fine for you, and so be it, right? A-and then, and then, for example, there is things like, y-you know, Coldcard has advanced features for developers, right? so we support testnet on the same, derivation path as, as your wallet. It is under a danger menu, and it was brought in because somebody brought to our attention that there was sort of like a, a path to attack that. I'm like, great, but I don't wanna remove that feature just because some people could be possibly, maybe, not even confused, they'd have to like- Be convinced to actively go into a menu and change something. So, you know, we took the device that could be a concern, so we just made it a little bit more, like, more scary to do that. but that, that's not necessarily a bug or a vulnerability, that's just a choice in design, right? It's very different than sort of having a flaw in your chip that, like, you can't secure it, right? Well, if you can't secure it, you can't secure it. There is no way to sort of spin that, right? so It's, it's important to, y-y-you know, first listen to your vendor, and then, and then look to the community to see like if people are freaking out about updating it, replacing it, outright trashing it. if nobody's freaking out and the vendor, y-y-you know, gave a satisfactory answer, but the, the disclosure, is sort of like being a little bit hyperbolic, well, then you have your answer. That Doesn't take away from the, the great thing that happened, which is there was a flaw or there was a bug that was disclosed and something was fixed. That's a fantastic outcome.

Yeah, very much so. okay, so I guess one other thing that people might be thinking, okay, so many listeners of the show are themselves, you know, they're an uncle Jim to somebody else, they're teaching somebody else. What are some ways that you like to, I guess, teach a newbie? Let's say, a person is a total newbie, they've Coins on the exchange, and we are trying to teach them, hey, now's a good time to learn, you know, get a hardware wallet and learn how to take your coins off the exchange. Now, some of the difficulty for them might be, okay, I've got to, you know, maybe they might be worried about, oh, I've got to run a full node to do this, and how do you do that? Now, I guess for me, some of the ways that I've been going about this is I might tell them, okay, as an example, you can run a prune Derev has that website, pruned-node dot today, that's one example, and then you can do that with Specter. Or another way is maybe you can get the Raspberry Pi set up, right? You can do an Umbrel or a Minnode or a Ronin Dojo or a Raspiblitz or a Nodle. what are some of your tips when you are, you know, out there, or if you have, you know, customers and they come to you and they're like a bit more newbie, how do they learn how to use a cold

card?

Just because for all its flaws, it is sort of like plug and play, right? You can just sort of like get running. There's privacy considerations in there, I'm very clear about that. you know, if they wanna, what's cool now is that, Blue Wallet is now on desktop, right? So we send a lot of people to Blue Wallet. I asked, the Nunes and the, the Blue Wallet team, they're looking to, adding USB support. to cold cards. So then, as much as I prefer people use that erga app, the our USB protocol is fairly like tight, and I think it's a good set of trade-offs for a new person starting, right? Because I want people to get used to holding their own money, right? I want them to get used to doing Bitcoin transactions. Once they build that confidence, it's gonna be a lot easier and self-evident to them what next steps they should take, right? so, and, and I, you know, we, we, we sponsor some and, and, Kiss, Kiss B, so Keep It Simple Bitcoin. He has a lot of amazing videos explaining every single step. Some of them are a little bit complicated, some of them are great and simple. I think Metodell has like really good noob friendly, I call them like whole banana, explanations of things where he'll take you from ordering the device to checking the device to setting up the node, setting up the wallet And, and going through the whole shebang. y-you know, Citadel, Livera's, company has like some really good, information on that too. it's just like the wealth of knowledge is huge, right? and, and no single solution will be the best solution for everybody. I think people really need to- To learn to research, because if you're just taking things at face value, you, you have a lot more room to get screwed.

Yeah, it's ultimately a personal responsibility culture. So while there are many of us in the space trying to make the education accessible to people, it, sometimes, you know, you still have to, you have to go and, you have to find those people. So we do what we, we do what we can.

Yeah, it's, it's not easy. Yeah, it's not easy. There's a lot of people who sort of scream from the soapboxes, right? "I'm an expert, I'm a security expert." You, you know what I mean? Like, "Do the thing I told you to do," and they're not necessarily providing the, the, the

the, the solutions that most people would sort of suggest.

Yeah. Because ultimately it is always about trade-offs, and again, nothing, nothing's perfect, but ultimately I think those of us who are in the space trying to teach people, it's ultimately about trying to Give them a reasonable balance between usability and security, and just while at the same time getting people to take that incremental step. Okay, I'm, I'm on the exchange. Okay, at least get off the exchange, even if it's imperfect, just get off the exchange. And then, okay, take the next step, next step, and so on. And I think that's kind of-- That's, that's been the approach that I've been taking when I'm out there teaching new, new Bitcoiners, but, you know, and part of it is, I see it Their need, what is their use case, right? If they want a day-to-day spending wallet, well, then you probably wanna get them on Lightning. But if they want a cold storage, which is most people, you gotta get them on a hardware wallet. So it's just, choosing the right tool.

Yeah, and, and then like, and then people have different security considerations, right? Like, some people live in dangerous people, dangerous places, right? Some people live in, in, not so dangerous places. People have different amounts of money, right? some people

Big service like Casa and Chain, right? Because their coins already doxed. Like, people have different needs and, and we can't just like sort of bang this idea into the, into the, like, like scream this idea out to the world where it's like everybody has to do this, right? No, there is no single solution, right? We just know that most people will be well served with a hardware wallet, right? But is that for everybody? No, right? It's, it's multi-sig goes A service or, or, or a phone wallet good for, for other people? Yes. Y-y-you know, it's, it's-- I think it's important to just give people sort of like all the options they have that are safe, and explain the trade-offs of each.

Excellent. Well, look, I think we're coming up to the hour, so it's probably a good point to wrap up. So if you've got any closing thoughts for the listeners, and of course, tell them where they can find you online.

Yeah, wow. can't believe it's been an hour. Been talking, I think today for eight hours on a Clubhouse. Yeah, no, it's, been spending a lot of time there, answering questions of noobs. It's a lot of fun. so yeah, so you can find me on Twitter at n v k. you can find the stuff that we do at coinkite dot com. you can find, my security, my, my security preference for most people at Bitcoin Security Guide. Sorry, Bitcoin Security Dot Guide. And, yeah, and, you know, don't, don't get caught up in the, in the, in the drama in regards to your security.

Fantastic. Thanks very much, NVK.

Oh, it's been a pleasure.

Get the show notes at stephanlivera.com/252 for this episode. Make sure you subscribe and share it with your friends. That's it from me. I will see you in the citadels.