SLP414 Matt Hill RaspiFUD or fact? Bitcoin nodes and security

Posted in: bitcoin Posted on Posted in: , , , , , ,

Matt Hill of Start9 rejoins me on the show to chat about the recent show on Bitcoin nodes with K3tan. We chat:

  • What’s the actual risk with Raspberry Pi’s? 
  • Where do they hit their bottleneck? 
  • What about more performant nodes? 
  • What about DIY? 
  • EmbassyOS and new EmbassyPro device
  • Should you separate bitcoin and non-bitcoin applications? 

Links:

Sponsors:

Stephan Livera links:

Podcast Transcript:

Stephan Livera – 00:00:08:

Hi and welcome to Stephan Livera podcast, a show about bitcoin and Austrian economics. Now, my recent episode with K3tan did ruffle a few feathers in the bitcoin community, in the community of node runners. So I’ve got an episode now with Matt Hill to give his perspective and perhaps clear the air a little bit, but also offer his own views on what the difference is with Raspberry Pi’s versus using other platforms. 

The show is brought to you by Swan Bitcoin. And Swan is organizing a new conference. It’s called Pacific Bitcoin. It’ll be on in November, so 10th and 11th of November. And the big announcement that we recently made is that Michael Saylor is coming and he’s coming in person. So this is going to be a fantastic opportunity for those of you who want to hang out and meet all kinds of bitcoiners from all different walks of life. There will be three main stages. There’ll be content and programming on all of those. There will be education, there will be deep dive sessions, there will be panels, there’ll be fireside chats, and there will be a range of events leading up to it in that week. So make sure you are there early. But the main event is November 10th and 11th. That’s pacific bitcoin. Go and get your tickets. Get your friends together. Pacific Bitcoin.com use code livera for a discount on your tickets. 

One site I regularly use is bitbo.io. I regularly keep track of what’s going on, whether that’s the bitcoin price over the short term, where you can Zoom out and use log scale. You can also see a range of things like a Bitcoin Magazine, newsfeed of articles. You can see various statistics, whether that relates to bitcoin mining, the Halving, bitcoin network, Lightning stats, sats per dollar, as well as comparison versus normy assets. You can see things like the inflation rate also. So this site is great to bookmark or potentially use as your homepage that’s bitbo.io. 

When it comes to securing our bitcoin private keys, we need to use good quality bitcoin hardware. My favorite bitcoin hardware signing device is the ColdCard, which you can get over at Coinkite.com. The ColdCard is such a versatile device, you can use it in all kinds of configurations. But if you’re a beginner, you can just buy this device and a USB C cable and plug it to your computer. And that’s easy to use with wallets, like Sparrow Wallet. Now, of course, if you are intermediate or advanced, you can use it in other configurations, like as part of a multi signature setup and as part of a multi signature set up. We do this feature called registering the Quorum so that the ColdCard actually knows the state and holds the memory of the XPubs of the other signers in that set up. So it’s also really useful in that way. It’s got a range of security features, it’s got two secure elements it’s got improved CPU and memory. So if you’re interested to get your ColdCard either for yourself or your family and friends, go to Coinkite.com, use code livera to get a discount on your ColdCards onto the show with Matt. Matt, welcome back to the show.

Matt Hill – 00:02:53:

Thanks for having me back on.

Stephan Livera – 00:02:55:

So Matt, we’re going to talk about Rasp pi FUD or Raspberry Pi’s bitcoin nodes, and of course the latest with what’s going on with Start9. So, yeah, let’s start with that. Just to set the context of listeners, I had a recent episode with K3tan where we spoke a little bit about Raspberry Pi’s, and I think K3tan and I were, let’s say, sharing some of our experiences where we’ve had issues with our Raspberry Pi’s, but I wanted to give you a chance to chat about your views on this also, Matt. So do you want to just give us your high level thoughts on that topic?

Matt Hill – 00:03:37:

Yeah, absolutely. So after that episode, we have a pretty strong community and they all reached out being like, hey, what’s up? So I thought it was pretty cool to be able to get on here and talk about it in order to answer some of their questions as well as some others, which I’m sure other communities have as well, because the Raspberry Pi has essentially come to dominate bitcoin enlightening home node operatorship. And so with an episode coming out that’s like, hey, this thing is no good anymore, obviously there’s going to be a lot of feathers, rustled and so, yeah, it doesn’t mean it’s wrong, it just means that adding some context and providing some advice going forward is warranted. 

So now you guys did a great job of that as well. Actually, K3tan did a wonderful job. I thought his analysis was very thorough and not inaccurate in any meaningful sense, and maybe lacking a little bit of context from sort of what our community is expecting. So that’s the main reason I’m here now. So to answer your question directly, now that I’ve sort of set the stage as to how this even came about and why, the Raspberry Pi, as was mentioned in your initial broadcast with K3tan, was designed as a low priced computer board for children to learn on. And it got really good. It became a very powerful entry level computing device. Powerful compared to, say, other super cheap single board computers. It actually became quite performant over the years, enough so that you could run a Bitcoin full node on it and as a show of as a demonstration or of the limited resource needs of Bitcoin and Lightning as well, it’s like, hey, you can run these things on a little Raspberry Pi. Anyone can do this. They’re available all over the world, they’re super cheap and very few other systems sort of blockchain cryptocurrency style systems could possibly run on a little Raspberry Pi. And that was a testament to Bitcoin’s engineering feat that it had achieved. And as you discussed, it’s starting to now kind of get bigger. Not just bitcoin itself in terms of the size of the blockchain, it’s actually gotten more efficient in terms of RAM and compute needs, the storage needs are bigger. If you are running a full archival node, if you are running a pruned node, then bitcoin is actually more runnable on a Raspberry Pi today than it has ever been before. And if you have a two terabyte SSD, then it’s also just as if not more runnable on a Raspberry Pi today than it ever has before. Where things start to get tough is the dependent services, things like ElectRS, things like Lightning and then layer three services that are using the Lightning nodes, like they can really start to add up and then that can start to put stress on this little Raspberry Pi. It has nothing to do with bitcoin essentially. It has to do with the ecosystem that is exploding around bitcoin and that people want to do with their bitcoin nodes. That can start to make the Raspberry Pi a bit inadequate. So where does it start to become inadequate? Right. It’s not Ram. That is not the first place that we see the Raspberry Pi start to break down. 8GB of Ram is pretty good. Right? So I have a little old Embassy one running here that I’ve been running for two and a half years and it’s in pi four, eight gigabyte variety and I have 25 services running in parallel on it and I’m using about 70% of my Ram on any given moment. Now during times of sync, it gets a lot worse, right? So if you’re syncing bitcoin from Genesis and you’re syncing ElectRS and you’re syncing the NBX Explorer from BTCPay, and you’re doing all these at the same time, you’re going to bog it down. But actually, RAM might not even be the first place that that happens. It’s going to be the CPU, right? So the pi that is the biggest shortcoming of the pi is actually the chip. It is pretty low powered and some of the bitcoin services, bitcoin and related services can really get CPU hungry at times. And so if you are trying to do a lot of syncing at the same time, you’re going to have a bad time. Like we tell our community members when you first spin up your Embassy, don’t install 20 things and turn them all on right at the exact same moment. The operating system just isn’t sophisticated enough yet to protect you from yourself in those circumstances. On even a normal computer, if you were to crank up 40 things at the same time and try to sync them all and when I say normal computer, I mean something much more powerful, you’re still going to notice a significant performance drop off, if not total throttling. So there’s a little bit of user discretion involved into making these things work as expected.

Stephan Livera – 00:08:49:

Yeah, fair points as well. I think that’s been part of the story of the history of the Raspberry Pi bitcoin node. It became almost like a meme and it became a thing in the community and I’m not denying that and obviously I’ve promoted that in the past myself. I think the other question that probably comes up is around cost today. The cost today to buy a Raspberry Pi compared to the cost, let’s say if we went back, if we rewind the clock two or three years, the cost of our Raspberry Pi, it might have actually been more like $150. Right? Whereas if you’re looking at it now, that cost has come up a lot. So I think that maybe that’s also the point that people have to think about whatever they’re doing, whether they are doing their own, like whether they’re buying a node off the shelf, whether they’re doing their own thing. Maybe that’s where the conversation is. Maybe it’s kind of like Raspberry Pi’s if you’re buying a new device, right? So let’s say if you  have your device and it’s already working and you already don’t have any issues with it, well, hey, no issues, right? No harm, no foul, you’re all good. I think maybe the question then is more about what’s the story going forward or what should we do? If you are advising somebody today who’s like, hey, Matt, I want to run a bitcoin node, what are my options?

Matt Hill – 00:10:08:

Yeah, so when the Raspberry Pi shortage kicked in, we had to really look at ourselves. This was probably a year and a half ago. It was the first time we started to witness some shortages. We were lucky because we’re buying in bulk and because we kept inventory and because we have some good suppliers and because we’re aggressive, we have always been able to get our hands on baseline price Raspberry Pi’s. We are, have been and continued to buy the eight gigabyte varieties at $75 a piece. It’s how we’ve been able to keep our prices of the lower end device reasonable while still maintaining something of a margin for ourselves so we can do business. But we immediately were just like, this is a single point of failure. There’s no way that we’re going to build a successful company around a supply chain like this that can be disrupted. Plus it’s unnecessary because there’s lots of computers that can run all the services that we’re looking to run. So we began looking at other options and so we’re actually quite familiar with some of the other options out there.

Stephan Livera – 00:11:06:

Right.

Matt Hill – 00:11:06:

We went to the Rock Pro 64 1st, this other single board computer variety that has a little bit more power. And we found out that we were going to have to do a lot of customization, we were going to have to build our own encasements. There wasn’t a big ecosystem built out around that single board computer, not as much as the Raspberry Pi. And then, so we started looking at more traditional things, right, like even some of these mini PC options, which ultimately is where we’ve landed. So, as a company, what we have done now is we’ve created two distinct products. We have. Like. The lower level entry level product. Which I can get into specifics in terms of how we recommend usage and what we recommend using them for. Which is still based on the pi at present. But does so in a way that alleviates many of the shortcomings of the pi. Which first and foremost. I should have mentioned earlier. The biggest problem with the pi by far is power. The pi is notoriously bad at power usage, as in it doesn’t supply it very well. And so if you’re running an external SSD plugged into the USB three port and you have a, you know, normal stock Raspberry Pi power supply, you’re going to have issues. We’ve seen these issues, we learned them the hard way, actually. We had to do some replacements for people because it took us a minute to figure out what was going on. And what was happening was like voltage leaks, we just weren’t supplying enough power. And it’s like your home power mattered. Like if your house had any kind of like lower levels of power or power fluctuations, all of a sudden these ghost-like database issues were popping up. We were just like, wow. So the Raspberry Pi just isn’t very good at delivering power to these external drives. And so what we have done with the Embassy one, the new Embassy one is we’ve gone for an all in one enclosure. So it is a very small all in one enclosed device where the SSD is actually inside of the box with the pi. And the encasement has its own added power supply. So we’ve solved both the dangling cord external USB issue as well as the shortcomings of the Raspberry Pi when it comes to power. And that is our new offering and that is the only thing that we now recommend and support. So even our DIY guide, this is all happening, like, right now, right? So that’s why this was so timely. Your conversation with K3tan was just like, yes, that’s exactly why we’re making the changes that we’re making is because the old recommendation of get a pi, put it in a box, get an external SSD with an enclosure, plug it in and then use it for 25 things in parallel just is causing problems. People are having a bad time with this. So the all in one enclosure with its own power supply alleviates all of the issues with the pi, save for one, which is the low CPU. So we recommend that if you’re going to go with the one, use it for your non heavy things.

Stephan Livera – 00:14:05:

Right.

Matt Hill – 00:14:06:

And this dovetails into a conversation about bifurcation of services as well, like Bitcoin non bitcoin. Is that even a proper bifurcation, but we recommend the one now. And we say, look, if you want to use it to run Bitcoin and Lightning, you absolutely can. Just don’t sync everything at once. Once they’re all synced, it’s going to be smooth sailing. We have years of data to prove that the Pi four, eight gigabyte can provide smooth sailing on probably about 15 parallelized services running Bitcoin stuff. Or we say use your Embassy one to do all your non heavy stuff and then the pi is a breeze. You can run Vault Warden and Nextcloud and all these messaging applications and services on a Raspberry Pi and it doesn’t even touch the CPU and the Ram. It’s perfectly adequate for that stuff. ElectRS is really the beast in the room that’s making this conversation a lot more interesting around, right?

Stephan Livera – 00:15:00:

I think that’s like the elephant in the room, right?

Matt Hill – 00:15:02:

Yeah.

Stephan Livera – 00:15:02:

And so for listeners who aren’t familiar, ElectRS refers to Electrum Rust server. And so the idea is when we run our Bitcoin node, we can think of the electrs. It does this address indexing function. And so when we run our wallet, so for example, Sparrow wallet or Electrum wallet on the computer that connects to our node, and that’s normally talking to the Electrum server. So it adds hard drive usage. It has a lot more CPU. So, as you rightly point out, I think that’s probably the elephant in the room. Although I wonder, I’m curious, how much does adding Lightning add to things like, let’s say if you have a lot of channels going, does that add a lot?

Matt Hill – 00:15:41:

No, not comparatively. If you’re running a Bitcoin node and an LND node and so on Embassy, we offer LND and CLN. I run them both on the same box. They don’t touch the system resources, not in the same way that ElectRS does. And again, during initial sync, they’re going to use a lot more. And if you have a ton of channels that are going to use a lot more. But if you are a power routing node, like if you’re a serious Lightning node operator and you have tons of channels, because look, the average person running a Lightning node, if you’re using it to pay and get paid on the Lightning Network, you only need like three good channels, right? Find a couple of well connected peers, open sizable channels, make sure they have inbound and outbound liquidity, and you’re going to be able to pay anyone anywhere. You only need two to three channels. If you’re running 20, 30, hundreds of channels, you’re not using a Raspberry Pi with an SSD, otherwise you don’t get it. Like you need a serious machine, which is where our new product comes into play, right? Where it doesn’t have to just be for power Lightning node users. It’s also for the Uncle Jim model, for content creators who want to reach tens of thousands of people, you need a serious computer. And so that’s our other product that we are now offering, which is the Embassy Pro. The Embassy pro is a 32 gigabyte. Two terabyte NVMe. It’s got an Intel I seven four 9 GHz processor with Intel management engine disabled and firmware which is the only device that can claim that running anything on Intel is sort of inherently backdoored. Okay, Intel management engine which exists beneath the operating system is sort of a built in backdoor and it’s just a known vulnerability that people learn to live with. Our hardware partner that we got with to manufacture the Pro Purism, they make pure OS which has a boot process that actually disables Intel management engine underneath the operating system. So it is totally private, not back door, at least not by that. There’s something lower we don’t know about, nobody knows about and whatever, but we can claim that the one known vulnerability has been disabled in the Pro. So anyway, it’s a rocket ship of a computer. It’s built on an X 86 architecture with an Intel I7 chip. It will satisfy you to do pretty much whatever you want. You can run probably 50, 60 services in parallel. We don’t even offer that many on our marketplace yet. While also serving as Uncle Jim for your friends and family, while also serving your community followers with blog posts and stuff like that, hosting your own website. So it is a serious professional tool or just for somebody who needs the biggest, baddest latest thing. And then the lower end again is still the pi, which we think is okay so long as you understand the sort of limitation of how much you can do with it at the same time. That’s really the big caveat asterisk of the lower end Embassies or pis in general. It’s just like don’t run 20, don’t sink 20 things at the same time on it. That’s the big asterisk that’s it.

Stephan Livera – 00:18:56:

Other point probably just to clarify on that as well is some users with Raspberry Pi’s have commented publicly and we’ve seen people talk about this kind of thing around their SD card getting corrupted. So I’m curious if you have any experience on that. Now I’ve seen different comments on that. So for example, with of Mental Space, he’s spoken about this idea where again there are people using different approaches. One approach he’s mentioned is this idea that that SD card should just be like a read only thing and then that’s how you can help lower the risk of that SD card getting corrupted. And so the idea is you’re using the SSD to do most of the actual work which can deal with more read and write. I’m curious what your view is on that and how the Start9 team are thinking about this question.

Matt Hill – 00:19:35:

That’s exactly how we think about it. Our SD card is a read only thing. I shouldn’t say that stores a little bit of metadata about the SSD just so that it sort of knows who its parent is. Like if for instance, you were to take out your SSD and put in somebody else’s SSD, it’s going to boot into diagnostic mode and be like something’s weird here, you’ve swapped data on me. So it sort of knows who its SSD is. But other than that, the card itself strictly holds the operating system and then has a fully empty partition that is designed for holding another operating system, another copy of EmbassyOS, because we do these deployments, these red green deployments like that. So for instance. When you update your Embassy. What you’re doing. It’s like you’re on 31 and you’re downloading 32. It downloads 32 to the SD card and then on next boot it just switches over and then it actually keeps a full copy of 31 in case there’s some sort of emergency rollback necessary. Which we don’t expect and have never had. But it has both the old and the new copy of MS OS on the SD card. So it’s not read only, but it is OS only. Meaning if it became corrupt or you just took it out and snapped it in half, nothing bad would happen. All you would have to do is go get another SD card, flash EmbassyOS on it, plug it in, and 2 seconds later you’re back up and running right where you are. So the SD card is no threat to our system. And if anyone is relying on the SD card as an essential part of their system, it’s a mistake. We were doing that for the first year. We sort of knew we were doing it when we first launched, but we also knew that there was no like the threat model was not there. There was nobody using our device. We were iterating and growing. But as usual, we identify our own shortcomings and try to fix them, right?

Stephan Livera – 00:21:11:

Yeah. And I understand maybe part of the tradeoff was for people who it depends what data they were storing on that SD card. And maybe for some reasons where people wanted a certain level of, let’s say, portability from one to another, maybe that was why they were using that SD card, let’s say more actively. But I think it seems that the community has sort of settled on that idea of kind of relatively less rights to that SD card so that there’s less risk around the corruption there and so that the user has a better time.

Matt Hill – 00:21:38:

Let’s say the best thing is just not use it for anything important. And then if it corrupts and it breaks, it doesn’t matter, you just replace it.

Stephan Livera – 00:21:45:

Right? Yeah. And so I think the broader question as well. And so I guess going to that broader, you mentioned the Ime, the Intel Management engine and so as you said, this is like one of those well known vulnerabilities that people talk about in, let’s say, Foss and Fosh, you know, free open source hardware world. And so there are various manufacturers I know like Purism and I think System 76 and various others who try to do this idea whether like saying, okay, we’re trying to mitigate that in some way or whatever. Right. So I think it also comes to that question of which architecture is it an Arm chip? Are we using like X 86 for our infrastructure while we’re here as well? I think it might be interesting if you have any comment on, let’s say, Warren to Gami’s views as well. Whereas Warren has put out his view where he’s saying one of the problems with some of these Raspberry Pi nodes, in his view, is that they’re not bringing up the security updates that are feasible on the full blown level of Linux. And so that’s why, from his point of view, he would rather people are using the full blown Linux as opposed to like, Raspberry Pi operating systems. Do you have a comment on that?

Matt Hill – 00:22:50:

I get it. I also think that from a threat model perspective, it’s probably more on the margin. You know, you want the latest software, but at the same time, doing automatic updates, for instance, to the latest thing all the time is actually a vulnerability too. So you’re kind of like always staying up to date with the latest software can be dangerous. And having software that doesn’t get updated frequently enough can also be dangerous. I would not be able to cite off the top of my head any sort of reason to believe that the Raspberry Pi Foundation does not, you know, stay up with security vulnerabilities in Raspberry Pi OS. It’s possible that there is some precedent where a security vulnerability was discovered and they didn’t release an update for four or five months. I’m not aware of an incident like that. Could it happen? Of course it could happen. It could happen to any distro. EmbassyOS is based on Raspberry Pi OS for the one, and it’s based on pure OS for the pro. And so it is up to us to make sure that we are keeping an eye on Raspberry Pi OS and pure OS and the underlying sort of deviant and Linux kernel to make sure that those distros are staying up to date. If they’re not, we have ways of getting in there and trying to patch them up as well. But at the same time, this is just sort of an inherent kind of challenge when it comes to software development is dependencies. It’s like, what are you built on? Because nothing is built on binary, right? Like we’re like six layers up in the stack for most of these services and applications. Embassy OS is obviously much lower, but we’re still built on, like, you know, we’re built on pools, which is built on devian, which is built on the Linux kernel. And it’s just like software is a layered thing. So dependencies are always something you just have to keep an eye on and.

Stephan Livera – 00:24:39:

Be diligent about sure and so on the question of cost as well. So can you just outline for listeners what’s the cost difference here? Let’s say if we’re going for the Embassy One, what’s the cost of the Embassy Pro device? Just to give people an idea.

Matt Hill – 00:24:53:

So the Embassy One, we are literally in transition right now. So the Embassy One, we have just transitioned from the external drive, plug and play thing to the all in one encasement. We’re actually lowering the cost as a result of this. But that is more of a sort of company decision. Rather. It’s not like our costs have gone down. We’re just trying to keep the entry level device lower while we push the Pro for more advanced usage. So the cost, the new cost of the Embassy One, don’t quote me to the dollar on this, is going to be right around $500 for the 1 TB variety. And that’s before any kind of discount which there’s lots of discount codes floating around out there. We also have 7% on fold by the way too. So if you have a fold card you can compound these discounts and basically just get free bitcoin. And then the all in cost is in like the low 400 plus you’re stacking bitcoin. So all in, if you take advantage of the right discount codes, you’re looking at low 400 for a 1 Terabyte variety and low five hundreds for a two terabyte variety. And this is in an all in one enclosure. So again, power problems solved. It’s a very nice device. The Embassy Pro, we’re currently pre selling because we haven’t started shipping it yet, it is on track for a fall release. So in the next couple of months we will be shipping this device. We are taking deposits currently at $42 for a deposit, which then gets you a huge discount off of our expected retail price. We’re still sort of feeling the market out on this, but we expect to retail the Embassy Pro at between 1801 $900. If you’re putting the 42 down, we’re giving you $400 off of that price. So we’re looking right around fourteen hundred to fifteen hundred dollars as your all in cost for the Pro. I know that sounds like a lot. If you were to go online and configure an Intel nook to the exact specification of the Embassy Pro, it would be almost $2,000. So we are undercutting the market from a computer performance standpoint, significantly by about 20% to 25%. These are real badass computers. Right? I would like to talk about some of the options that you and K3tan talked about as well, which is going the like refurbished mini PC route to get more power on a budget which we think is wonderful and very viable as well.

Stephan Livera – 00:27:15:

Back to the show in a moment. Have you left your coins on the Exchange or do you have family and friends who are. Just simply too scared to do this. Get your coins off the exchange and don’t delay. Go to Unchained Capital and they can help you out with a multi signature vault. This will give you that peace of mind knowing that you have removed single points of failure. After the crypto collapse of earlier this year, so many more people have learned why it’s important to hold your own keys with bitcoin. So with unchanged, they can give you a concierge onboarding experience where they will help you overhaul, set up your vault and withdraw your coins out of the exchange or out of your single signature wallet into your multi signature vault. So if you want this go to Unchanged.com, go to the concierge page and use code Livra for a discount on your package. Are you interested in bitcoin mining Brains.com? That’s brain s.com can help. They have brains. OS plus. This is firmware that you may be able to install on your bitcoin mining ASIC machine. You can increase the hash rate on your bitcoin ASIC, or on the other hand, you might choose to improve efficiency and get better jewels per Terra hash. There are all kinds of features available as part of the Brain’s OS Plus. Now. They also offer brains. Farm Proxy. This is a single application that allows miners to configure parallel usage of multiple pools. You can reduce data loads and designate backup pools. So there’s all kinds of options there that help you manage your bitcoin mining farm. Also Brains analytics dashboard, the Insides dashboard is fantastic. If you really need a deep insight into bitcoin mining, this is the place to go. So go and find all of this over@brains.com. Are you sending bitcoin transactions or are you managing a Lightning node? Mental space is really useful for you. It’s a Bitcoin Explorer built by bitcoiners for bitcoin as a. It has real time transaction tracking and Memphis visualization. So you can see both the projected blocks and the confirmed blocks, and you can also see what kind of fee rate you should set depending on the priority of your bitcoin transaction. Also, Memphis have recently launched a Lightning explorer. So you can search based on Lightning node ID. You can look at the different channels that are there. You can see things like the average capacity, the median capacity, the most connected list, the highest liquidity lists. There’s all kinds of features and it’s really indispensable if you are using a bitcoin node or a Lightning node. Mempool Space is just another tool that you need to use. So you can go and try it out today. You can even self host it and they even are offering an enterprise edition. So for those of you interested, go and get in touch. But the website is over at Memphal.Space, right? Yeah. And I think that’s just to sort of give the listeners there kind of neutral and for them to understand, okay, if I’m buying the start nine device, what am I paying for? It’s not just the hardware, it’s also that kind of the software support you’re supporting this project. And so there’s like an aspect there. I appreciate that. So let’s talk then about the comparative, the comparisons for if people were to DIY. Let’s say DIY if you went for a ThinkPad or if you went for K3tan’s Dell OptiPlex, a couple of $100. Now, to be fair, there is some more DIY required here. So, for example, you might need to actually be comfortable with opening that box up, putting in a bigger hard drive. You might need to be comfortable with depending on how you want to set it up. You might need to be comfortable with command line, you might need to flash Ubuntu and then let’s say use a PC image. So there’s different components to that. I’m wondering if you have any thoughts that you want to add there around the comparison of DIY versus buy something out of the box.

Matt Hill – 00:30:51:

Yeah, so we are sort of DIY first in terms of our messaging. So if you go to our website there’s by an Embassy and DIY side by side, ultimately we want to see this technology spread. We want to see people self hosting. That’s the point here. We sell convenience and support, right? We sell community. When you buy from Start9, what you’re doing is we’re developing arguably the most advanced kind of self hosting for everyone technology that has ever existed and that doesn’t come cheap, right? We’re a talented team of people who need to survive and we’re growing a business here so that we can continue to propagate the technology. Now, that said, it’s also not a donation. You are getting real value. If you buy from us, as opposed to DIY, which is the plug and play experience, you just close your eyes, plug it in and it just works. And you’re also getting this sort of 24/7 white glove support from us and from our very passionate community of very technical people. You’re going to get help. So that said, people who do feel confident to sort of put this together themselves can do so and save a few bucks. Probably not as much as you think. The discount is not that steep once you tally all the added costs and time it is there, but it is on the margin. So, for instance, doing a kind of DIY refurbished mini PC, like an octopus. If you find a good deal, if you find a good refurbished one that’s in excellent condition, you’re going to end up with probably an I Five. So a great processor, something that’s going to blow the Raspberry Pi chip out of the water. It’s also not going to be the I Seven or I Nine brand new thing. It’s going to be something a little bit more moderate, but very adequate for most usage. You are going to have to get some storage. If you don’t have it already and that can get pricey. Depends if you’re doing a one or two terabyte variety. So like the EmbassyOS Pro for instance, is a two terabyte NVMe which is a far cry from a 1 TB SSD. Those are very, very different price considerations. A 1 TB SSD you can pick up for under a NVMe is usually going to run you 300 if it’s of any decent quality. And there is a very noticeable difference there. Not only is it double the amount of storage but it’s way faster. Way faster. And when you have right intensive things like ElectRS you’re going to notice a big difference with something like that. So I think all in K3tan had cited a $500 kind of price target, maybe $400, right? For getting like a good I think.

Stephan Livera – 00:33:39:

You’Re speaking in a Ud term. So yeah, something like that.

Matt Hill – 00:33:44:

I think it varies a lot depending on where you are and the deal you can find. But let’s say you can pick up a good computer between the 400 and $500 range or something like $400 range versus buying a Raspberry Pi all in one, everything put together for a similar price. Which one do you do right? It depends depends on if you’re comfortable doing the DIY approach. It depends on if you value the community and support aspect of the product. But we support it, right? If you want to go get Adele optedlex and install MBA CLS on it, can’t do it yet because we haven’t launched the X 86 distro of the operating system but it’s around the corner. We will be able to support hopefully pretty much every architecture and form factor under the sun eventually that will require a little bit of Tweaking for different things. And the problem here too is just variance in hardware. Like you’re not going to have a lot of issues with the board itself like the chip and the Ram, those are all going to be okay. But storage can be really painful. There’s a lot of drivers out there that don’t conform to exact standards or claim that they do and then don’t. Exactly. So you got to be careful. You could end up having IO issues with your drive if it’s not one of the sort of like known supported quality drives got you.

Stephan Livera – 00:35:07:

Yeah. So I guess in this example, the one because K3tan and I chatted often, he’s a fan of the I think it’s the Samsung QVO. So for example, let’s say you went for that and it was a two terabyte drive. I think I was looking it up. I mean, at least for me in the UAE I can get that for about $150 for a two terabyte Samsung QV SSD, not the NBA as you’re saying. So you’d have to again be comfortable opening your computer, plugging that in. So that will require a little bit of DIY. And you know what, there might be some people for who they would rather just pay. They’d rather just pay, have somebody do it, ship it to them, they plug it in and it’s good. Also, I’m curious, you mentioned in the DIY case, let’s say a Start9 user wants to DIY. Are you selling like a license to use the operating system or what’s the model around that?

Matt Hill – 00:35:55:

Not anymore, no. So we have recently changed our approach to EmbassyOS, actually as planned. Okay. So a lot of the things that we do. We actually have kind of a vision of how we’re going to evolve this company and it gets more and more open over time. Not less and less because when you’re tiny. You need a little bit of protection and then as you grow. You can sort of get and this is opposite the way a lot of people view company growth. They view it, as you get bigger, you sort of corrupt more and more. We have an idea of doing it a slightly different way. But anyway, so Embcos is now totally free. Today you can go download the 31, which is the latest version. You can go download from either GitHub or Images Start9.com and you just type in XXXX eight of them as your product key and it’ll just download a fresh image and then you can install that on a Raspberry Pi. So it is possible today to essentially get up and running fully with an Embassy without ever touching us or doing business with us. And we are going to continue that obviously into, which is on the eve of being launched, which is a big version for us because it removes product keys altogether. So now all images are created equal. You will be able to download it from our website, download it from GitHub, download it from your friends, self hosted instance of Nextcloud, wherever you can just download Embassy OS, flash it, plug it in, run it and you’re up and running. So it is a totally free as in beer OS for the consumer. Now we retain a non commercial clause in the license that basically says at present, Start9 is the only entity that can profit from the distribution of EmbassyOS. But again, that’s part of the strategy as well.

Stephan Livera – 00:37:44:

Got you. And so 0.3.2 is that also the addition where along with releasing that, is that where you’re going to also have X 86 support?

Matt Hill – 00:37:53:

33 is X 86. So 32 was a sort of intermediate step to get rid of product keys. It is a lot of under the hood performance tuning. So we switched from using a SQL Light database to a postgres database because SQL Light just yeah, it was just causing some obscure issues from time to time. So we’ve really kind of beefed up the OS in two as well as simplified it by removing a lot of the weird stuff we were doing with product keys. And then 33 is the X 86 distro which will follow pretty shortly. We’re actually a good way into 33 already and we’ll launch with the Pro.

Stephan Livera – 00:38:33:

Fantastic. And so then if you could walk us through that install experience, I presume it’s going to be similar. Like basically the user will use something like Belina, etcher. They’ll download. Is that going to be like an ISO file? They’ll flash that either to the SD card or to a USB stick and then install it that way. Or what’s the install look like.

Matt Hill – 00:38:50:

Exactly. You would download Embassy OS to your computer. You would put in your SD card, open up Belina, etcher. Flash it, take the SD card out, put it in your Raspberry Pi, plug in an SSD, plug it into the wall and Ethernet, and then visit Embassy local. It’s a pretty standard setup procedure for these products and for good reason. It’s very simple. It’s all done over the local area network of your home, right? The Embassy is broadcasting on Embassy local. So it’s just a web page. You just go to emcee local and about three clicks later your device is up and running. Serves itself on both the local area network in Perpetuity on a unique address. By the way, it doesn’t stay at Embassy dot local because if you had like multiple embassies, those would conflict with each other. So it serves itself up on a unique dot local address and a unique dot onion address. So when you’re home, you use it over https dot local and it has a self signed Cert that you install into your browser. So it’s encrypted traffic even on the land. So if somebody was sniffing your land, they couldn’t even see what you were doing. And then when you’re on the go, you visit your onion address, which is also obviously encrypted and onion routed for remote access. What’s really cool is that shortly following the release of the Pro, our next major feature launch is going to be clearnet support. So you will be able to optionally you will not have to do this but it will be probably the best way to use your device. You will optionally be able to add any domain that you own and use it to host your Embassy. Not only like the main dashboard of your Embassy on say, stephanlivera.com would be like your Embassy main dashboard if you wanted it to be. And then BTCPay Stefanlvera.com or store stephanlivera.com would actually serve your BTCPay store. And then you could do nextcloud Stephan Livera and that would serve your own selfhosted nextcloud instance. And so you can optionally you will optionally be able to toggle on and off what domains subdomains you use for various services and these will be highly granularly configurable. So for instance, you’ll be able to register like four different domains stefanovira.com, stefanovara. Net and then pick and choose which services you host on those domains and subdomains of those domains and then turn them on and off. So if you think that one is being abused. Like somebody is trying to dos you on Clear Net. You can just go turn it off and that will stop and your tour address will remain. So censorship resistance is like a big deal here because it’s like you can go full Clear Net, but if they ever pull the plug on you, you’re 2 seconds later telling everyone to go to the onion site.

Stephan Livera – 00:41:42:

Got you. Yeah. And that could be handy as well for the, let’s say the Uncle Jim context, right? So let’s say you have Next Cloud and you have your uncle or your auntie who you want to share a photo with, or you can put that into your next cloud, and in that example, it could be in that next Cloud instance under your website. And it’s easier to share that URL with them because now they don’t need to have Tor browsers to go there. And for them, maybe they’re a bit weirded out by, oh, what’s this Tor browser thing?

Matt Hill – 00:42:06:

That’s the big thing now.

Stephan Livera – 00:42:06:

It’s just, oh, I can just use my normal browser to go there and do something.

Matt Hill – 00:42:09:

It’s weird, it’s slow, it’s unreliable, and it’s a very necessary censorship resistant fallback. To date, we and everyone else have been using Tor as a primary. It really is not a good primary. The threat model of Clear Net is pretty low. You have to be explicitly targeted to have your Clear Net domain compromised in any meaningful way. And if it is, then Tor’s just sitting right there, right? So having that fallback, that escape hatch of Tor, we think will actually act as a deterrent of Clear Net attacks because it’s like, why attack the Clear Nets? All you’re going to do is push it into the shadows, right? And then you can’t stop that. You can’t turn off a website, you can’t even denial of service attack an onion website because Tor has DDoS protection built into the protocol now. It itself is being denial of service detect the entire Tor network. That’s a different problem, and we’re dealing with that. It’s very painful. We talk with organizations who want to. This is the whole reason of Embassy pro in Clearnet, because an individual, not even highly technical your average bitcoiner, is going to put up with more inconvenience and weirdness than anyone else in the world for the sake of independence and sovereignty. Okay? We’re dealing with a very slanted market here. It’s not representative of the rest of the world. The rest of the world is not going to do this. They are not going to set up Tor on their computer and then use V three addresses to do their self hosted anything. They’re just not going to do it. So what we’re doing is we’re pushing obviously for the ideal endgame of everyone running a personal server, but we also recognize that that’s probably not realistic ever, that in the future you’re mostly going to have yes, a lot of people running their own server. But you’re also going to see families running a familial server. You’re going to see organizations running an organizational server. Churches are going to have their own server, businesses, schools, and the people who belong to those organizations or that family are going to use that server. Right. The uncle gym model, we think is the real push into the future. And then, yes, obviously if you always, ever want to take that next step and become fully sovereign, it’s not that hard. You can do it, and you can stop using your Uncle’s server for your needs. But in order for the Uncle Jim model to work, two things had to be true. We needed better computers. Raspberry Pi’s were never going to cut it if you wanted 100 of your congregation members to use a server. And two, Tor, tor was never going to cut it either. So by beefing up the hardware and going multiplatform, and by introducing clear net support, which is what we’re calling the ability to host your Embassy and its services on a normal domain, we are unlocking the door to organizations and content creators of all sorts. And we think that is the path to grow the technology.

Stephan Livera – 00:45:24:

And it may also be more viable for, let’s say, businesses or a group of people. So let’s say, hey, let’s all chip in to get one big server going rather than each individual person trying to buy their own. So you can see there’s a business case there. So in a sense, in some sense, start nine might start being a little more b to b. In some sense.

Matt Hill – 00:45:47:

Yeah, it’s not even b to b necessarily. It’s just like organizations using it for themselves for whatever those purposes are. And if you look at some of these organizations today, they’re actually getting raked over the coals in terms of SAS costs, right? We talked about this on my last episode.

Stephan Livera – 00:46:04:

This is our recent episode where we spoke about how you’re able to let’s say this organization is currently paying a lot for Dropbox or for Flag or some kind of Microsoft Office 365. Whereas maybe if we could use Nextcloud instead and host that, then, hey, we can cut out all these dropbox costs and hey, all of a sudden now it’s a lot cheaper for us to just buy this Embassy pro device than Pay, office dropbox, slack, et cetera.

Matt Hill – 00:46:30:

Yeah, it’s a one time up front capital cost.

Stephan Livera – 00:46:33:

Yeah. So I think that’s fascinating as well to see I think one other question I was really curious, and you mentioned earlier, is the bifurcation between bitcoin and non bitcoin. What are your thoughts? So I guess setting the table again for listeners, I think K3tan’s thoughts on this were more like, hey, you might want to have some segregation, have bitcoin things on one thing and non bitcoin things on another. What’s your thinking here?

Matt Hill – 00:46:55:

I think that bitcoin and nonbitcoin is the wrong delineation. That is the sort of easy one and it’s sort of the popular one to say. But that’s not actually what they mean when somebody says you should not run your bitcoin and nonbitcoin things on the same box. That’s not what they mean. They actually mean two other things. One is your hot and cold infrastructure, right? As in things that are hosting keys that protect money should not be on the same box as things that don’t. And number two, what they imply by that statement is that things that are very resource intensive should not be running on the same box as things that maybe don’t need a lot of resources. There’s sort of this crosscontamination and I generally agree with that if you are going to be running a lot of things and running any using any meaningful amount of money, right? So for instance, if you’re running five services like Bitcoin, CLN, Core Lightning and Vault, or Next Cloud, right, say that’s what you’re running, you’ve got Bitcoin, which is totally cold in most cases, unless you’re using the core wallet as a hot wallet and then you’ve got things that are inherently hot, like Core Lightning, but the resources aren’t going to be an issue. So the second reason for bifurcating goes out the window, right? Resources are not going to be an issue because you’re not running a lot of services and say you only have $200 worth of bitcoin on your Lightning node. It’s also not a big deal if the whole thing goes kaboom, right, either. So it’s like you don’t really need to bifurcate your stuff because your threat model and the disaster scenario just aren’t that big or that bad. And so it’s like it’s okay, right? You don’t need to shoot for the ideal. But if you’re running 30. 40 services and a good chunk of those things are really resource intensive and other things are holding private keys and are super hot and other things are totally benign. You should probably come up with a strategy of threat mitigation. Right. And resource usage so that your benign things are your non resource of intensive things are not getting bogged down by your resource of intensive things. So I don’t think that there is this clear like you should run these things on one box and these things on another box. It really depends on how many of these things you’re running and what they are and what you’re using them for. There is unfortunately no just one size fits all do this, which I know is not ideal because people really want that. So here’s what we have been telling our community is if you want to so hold on, let me back up for a second here and state kind of what the real threat is. Why not put key things on a server and non key things on a different server? Why not put them all on the same device. The perceived threat here is malware. It is cross contamination, right? It’s like I install some new version of some benign chat app or something like that. Chat could be very private as well. So I’m trying to think of something that doesn’t matter and kind of all matters to an extent. Like you don’t want to get hacked on any of these services, but like say like, I install some new version and it’s like malware and we’re using docker for containerisation which affords some degree of sandboxing, right? Like it’s pretty good at keeping these things separate. That’s kind of the whole point is that they’re running in their own little virtual machines. But we’re not going to make a claim that if someone were to build sophisticated malware into one of the services that you install on your device, that it can’t access other services. It can. Ultimately they’re on the same metal, okay? Like they can get to each other. We are building out more sophisticated systems. Like, we don’t use docker compose, we’re not using off the shelf docker stuff to do what we do. We are building a whole new system that is far more sophisticated and featureful and secure than what something like docker compose can offer, which is how all these other systems orchestrate their containers. We’re building out a permissions API such that services will not be able to access any other services without this sort of user permission, right? It’ll prompt. It’ll be like you’ve installed LND and it’s requesting permission to access these parts of bitcoin. Do you approve? And so then you can say yes. Now, if you blindly say yes and that is malware, then you’re still had ultimately you can’t protect the user from themselves. But at the end of the day, the experience will be similar to like an iPhone or Android phone when it says, hey, you’ve installed this app and it wants these permissions to your device. Do you granted those permissions? So here’s the deal though. So people say run it on different computers because you could end up with some malware of this that infects this thing. And that is absolutely true. But putting them on different machines doesn’t really solve that, does it? Because if you have all your bitcoin stuff running on machine A, right? And by bitcoin stuff, let’s be clear what we mean here. We’re talking, like, bitcoin ElectRS l and d CLN RTL spark yada yada BTCPay. You’ve got like this whole stack of services and you’re like, oh, I’m super safe because they’re all bitcoin services. Why? What if one of those services in an update is total malware? Your whole system is compromised? The fact that it was bitcoin or non bitcoin didn’t really eliminate that threat. It just mitigated it by lowering the attack surface, right? So if I’m running 50 services on my device and they’re all over the board from bitcoin to Lightning to non bitcoin to storage to messaging, am I more likely to download malware and have my phone stolen? Absolutely. But if I’m running ten services on a device and they’re all bitcoin only, you haven’t eliminated that threat, you’ve just minimized it, which is a good idea. We’re not saying not to separate things. We’re just saying that this isn’t a magic pill. You can’t just have a device that’s running bitcoin things and have a device that’s running non bitcoin things and then just go, I’m safe. You’re not. You’ve slightly mitigated the threat. The right way to do this is to make sure that you’re downloading trusted software signed appropriately and not store your life’s fortune on your Lightning node.

Stephan Livera – 00:53:31:

Of course. Right, so let’s summarize that then. So since then, the argument is not as much about bitcoin versus not bitcoin, it’s essentially more about whether you are keys hot or keys cold, and as you’re saying, around resource intensiveness. So probably the most important distinction, really, is this keys hot versus not. Because really, so if somebody is just running an electrum server and all the keys are held in a hardware wallet instead, well, then at least you’re not that much at risk in terms of losing your coins. Like, okay, maybe there’s a privacy ramification, fine. But it’s not the end of the world compared to if your Lightning node has, like, all of your coins on it and it gets pawned, well, then you’re just completely out of luck.

Matt Hill – 00:54:23:

Our stance is this if you are a power Lightning user with a ton of funds, ton of hot funds on Lightning, your server that is running that Lightning node should have bitcoin and Lightning on it. Maybe some sort of node management software on top of Lightning, like RTL or Balance of Satoshi. But again, the fewer the services you have running, it doesn’t matter what kind of services there are, it’s not bitcoin versus nonbtcoin. It’s like you want to minimize the attack surface of potential malware installation. So you want as few services running as possible on the node that is managing your money. Maybe that’s the right heuristic to think about this. It’s like you’re sensitive things, things that you really don’t want someone to have access to, whether it’s for privacy reasons or it’s your data or your money. Minimize the number of things running on that device and then put all your sort of less important things, maybe on another device that might be the right kind of angle to think about this.

Stephan Livera – 00:55:24:

Yeah, I think that’s a good spot to finish up. So, yeah. Great chatting. Matt, how should I put this? I’ll just say it’s good to see the community having some rigorous back and forth about what kinds of devices we should be using, whether it’s bitcoin applications or self sovereignty, just generally and listeners. Go and find Matt and the team. You can find them over at start nine.com. Matt, I think your Twitter is underscore Matt Hill. Right?

Matt Hill – 00:55:52:

That’s, right?

Stephan Livera – 00:55:53:

Yeah. So yeah. Matt, anything else you want people to know before we finish up?

Matt Hill – 00:55:58:

I don’t think so.

Stephan Livera – 00:55:59:

Okay. Fantastic.

Matt Hill – 00:56:01:

This is not a settled discussion.

Stephan Livera – 00:56:02:

Right.

Matt Hill – 00:56:03:

This is a good I’m glad you had that interview with K3tan. I think he’s great. I think that he was spot on with everything. It’s just that we don’t know exactly yet how this is going to evolve. And it’s important to keep talking about it. And that’s why I wanted to come on. So I appreciate it. Thank you. Our community appreciates it. Yeah.

Stephan Livera – 00:56:23:

Fantastic. Well, thank you. So the show notes are available over at stephanlivera.com/414. Thanks for listening, and I’ll see you in the citadels.

Leave a Reply