SLP366 Nate (BeefOrBacon) – How To Run Your Lightning Node

Posted in: bitcoin Posted on Posted in: , , , , , , , , , ,

Nate (aka BeefOrBacon1, also working at Voltage) joins me to talk about how to run a Bitcoin Lightning node. We get into all sorts of information while keeping it accessible for people relatively new to running a lightning node. 

  • Different lightning user types
  • How to run your lightning node
  • Different types of node: RPi, box, VPS
  • How to manage channels
  • Who to open channels with
  • What tools you can use
  • Should you use swap providers?
  • What kind of fees do you earn? 
  • How do you backup your lightning node? 
  • Payment reliability

Links:

Relevant episodes:

Sponsors: 

Stephan Livera links:

Podcast Transcript:

Stephan Livera:

Nate, welcome to the show.

Nate G:

Glad to be here, Stephan. Thanks for inviting me.

Stephan Livera:

So Nate, we’ve been chatting for a while and I know you obviously have some expertise and some practice now in running Lightning nodes. And we’ve even been chatting in the background about it for a little while, and I thought it would be great to get an episode to get your thoughts on it. And of course, I know you’re working at Voltage as well, so you can probably share a little bit from that side also. So yeah, do you want to just give a little bit of a background on yourself for the SLP listeners? So they know who you are and what’s your involvement in this whole Bitcoin and Lightning world?

Nate G:

Sure, absolutely. So my name’s Nate. I go by @BeefOrBacon1 on Twitter and I am a Bitcoiner—been a Bitcoiner for a while, and in the bear market of 2018, I was thinking about running a node for the first time. And near the edge of 2019 there I came across the Stadicus RaspiBolt guide on building your own node on a Raspberry Pi. And I decided to dive into that with zero Linux experience whatsoever and managed to figure it out. And I honestly just wanted to run a Bitcoin node at the time, but I saw Lightning as sort of like a bonus, like, Cool, I get to run a Bitcoin node and a Lightning node at the same time. And then after that, I just went deep into Lightning and started learning about channels and liquidity. I spent a lot of time on the LND Slack. Obviously I was starting LND at that time. And eventually I noticed the community in 2020 and 2021 where it really starting to grow out, people were starting to talk about it more, so I got into just helping as many people as I could. At the time I was still working my normal non-Bitcoin job and raising a family and everything, but when I could spare time I was working with the Plebnet guys or running free Calendly consulting to help people—answer general questions about running Lightning. And then Voltage hired me last September to spearhead the support and education part of the service, which I’ve been doing, and it’s been great. So I guess that’s the general overview there.

Stephan Livera:

Fantastic. And so I think it’s probably important for listeners—maybe if you’re new or maybe you’re not as advanced into this—just to understand the different Lightning users. So let’s say as an example, there are some different types. And maybe you want to spell out some of your thoughts? I mean, let me just open with a few ideas and you can expand on that. So as an idea, you might be, let’s say, just a retail user of Lightning. And so this might be a person who is using different Lightning services like Phoenix, Breez, Muun, or even Wallet of Satoshi. You’re using Lightning services and you’re typically just making small payments, and receiving. Another user type might be the routing node operator. And so this is the person who’s trying to set up and actually set up channels. And these are distinct roles, right? And then another example might be a person who is a merchant. So you are receiving—typically—payments, and so you might need to set up a Lightning node from that point of view. And then maybe another bucket or category might be like a Lightning Service Provider. And some of these might be like a routing node, but they are kind of doing a slightly different service themselves. So there’s kind of this ecosystem of different user types. Because maybe if people are looking back to let’s say 2018 or 2019, it’s just kind of like, Oh, it was just all the one thing, right? Like, one person was just doing everything. But now it’s almost like we’re seeing a distinction in the different role types. So do you want to just expand a little bit on that, just for those people who are new to Lightning? Maybe they’re learning about Bitcoin and Lightning?

Nate G:

Yeah. I think that maybe some people, when they’re starting to hear about Lightning and nodes, they start thinking about, Everybody needs to run it the same way or have the same goal—but that’s not the case at all. Trying to build out a routing node takes time, energy, and effort. But if you are trying to be a Bitcoin sovereign—you know, there’s a lot of Lightning wallets out there that are very user-friendly, but you’re trusting third parties and whatnot. But if you just want to buy gift cards on Bitrefill or Pay With Moon and you want to run your own node and you just want to do that, that is probably the easiest way to go about it, because after you get your node running, you simply open up channels to routing nodes—and we can go over how to choose who to open up channels with later. And essentially you’re good to go: you have your outbound capacity and now you can make payments. Great—and you’re all set. Merchants are a little bit different because you’re more inbound-focused as a merchant, meaning: the liquidity that you want needs to be receivable. So that’s called your remote or your inbound capacity. So that’s the equivalent of somebody opening up a channel to your node, and that’s where a lot of node liquidity services have popped up such as the marketplaces from Lightning Labs, the Lightning Pool, c-lightning—now known as Core Lightning—has liquidity ads. So essentially if you’re running a routing node and you have excess liquidity, you can basically sell that liquidity to new node runners—or merchants, specifically, need that. And I think that is a really cool concept that is continuing to hopefully be made easier, and eventually I think will be obfuscated away for merchants that want run a node. They could simply say, Hey, I want liquidity—boom, bam—and they get some. So I’m hoping that’s the future. I think it’s going that way. And then of course [there] is the routing node operator, which is someone that has a lot of capital. And what I mean by a lot, I mean like—so now we’re getting into the part where this is my opinion—a lot, I would say to start a routing node, I think you probably need 5 million sat channels, are probably the sweet spot. And I think around 20 channels are probably pretty good. And not all of those channels have to be channels that you open yourself. It could be channels you receive from others. And that’s the most complex—so we could get into that—but that’s probably the most complex. A lot of people though I think start Lightning nodes out of curiosity, meaning they don’t want to do any of that—they just want to learn how it works, and then they maybe transition to that in the future. Maybe they decide, Okay, this was like my beginner node. I learned a lot doing it. I’m gonna shut it all down and I’m gonna make a new one maybe on better hardware or something and then take it seriously from there. So I work with a lot of those kind of people also. So yeah, that’s a general overview.

Stephan Livera:

Yeah, that’s great. And I think that is a great segue into that next question I had, which is around what kinds of Lightning nodes we can run? Because obviously we can run a RapiBlitz or an Umbrel or a myNode or various other node packages. And typically people might start out as like a beginner or learning node—they might start out with a Raspberry Pi or one of these single board, computer-type small computers. And that’s been a bit of a discussion in the community. Like, my friend Ketan is quite against the whole Raspberry Pi—he believes you should run a box. And then there’s another camp which is: you can run a VPS, and obviously Voltage can help you with that. So do you want to just spell out some of your thoughts comparing those three? So the Raspberry Pi or let’s say a RockPro64 versus [the] second option, which is running a box at your home, like your own box, or thirdly, running a VPS. If you could just explain a little bit about each of those and your thoughts on those for a Lightning node runner?

Nate G:

Yeah. So Raspberry Pis are obviously very popular, very affordable in a lot of ways and fairly powerful for what you get with them. And I think running just like a normal base layer Bitcoin node on a Raspberry Pi is great because you could turn it off whenever you want and it’s not as intensive. When you start using it with Lightning, though, all of a sudden you’re dealing with—so with Lightning you need to be online all the time. And I’m not saying Raspberry Pis are gonna break down or anything. I mean, they’ve been known to do that, but being online is a big issue, especially if you want to run a quality routing node, because that’s gonna affect your overall network reputation. But you’re going to want to eventually consider that the Lightning Network is a gossip-based protocol, which means packets and stuff—you’re sending information out to the network constantly and receiving information from the network constantly. As you get more and more channels and build out your node, this might prove to be a bottleneck in the future. So I really think that if you’re running a routing node—which in my opinion is more than 20 channels—I think the next step is to get a mini computer. So what I did after I decided to take it seriously was I went to Google and I was like, I want to build a budget mini gaming PC. And then it’s really cool, because if you Google something like that, it’ll show you the parts you need to buy. And building a computer is actually pretty straightforward these days. So a Mini-ITX—that’s basically the size of the motherboard—which is just this little maybe 8″ x 8″ board in a little case and you just put the pieces together and you’ve got a pretty powerful machine for $500, basically. You might spend a little bit more than that on the hard drive if you want to go to like a 2 Terabyte or something and really do that. But I would say that’s the next step for the home Lightning enthusiast. And now we get to like VPS stuff, which has been around for a while. And generally with VPS, you are essentially renting computer power from a data center somewhere and you are manually installing, configuring, doing all this stuff—very technical command line-type stuff, usually. And that is pretty good for those that are traveling a lot, don’t want to deal with power outages. Obviously you could connect into your node remotely with mobile apps like Zeus. So a lot of folks like to do that when on the go. And then Voltage comes along and is like, We’ve developed a way to provision you—your own node—that we don’t see what’s happening whatsoever, we don’t know what you’re doing, we have no access to your funds, and everything’s encrypted client-side with your node-unlock password, basically. So even if we wanted to see what you were doing, we can’t do anything. The worst that Voltage could do is basically remove access to your Voltage dashboard, per se, but that doesn’t mean you lose your money—that doesn’t mean that we take your money. It just means that if that worst case happened, you’d have to recover just as if your hardware broke down or something. Which is a really interesting solution. And I find that, so far, a lot of folks that are wanting to sell things on Shopify or even BTCPay Server—which Voltage does have natively integrated into the platform—really like that they don’t need to deal with updates, upgrades, uptime, and that hands-on requirement of running a physical node at home. So we’re building that out and working on some other things too. So to boil it all down, I think that it’s just there’s so many user types. So maybe, for example, you could run your routing node at home because you have physical access to that, you know the configurations and everything, but then maybe you can run a second node that you just buy things with. That could be your Voltage node, for example, which you can run on Neutrino—which is a little bit more affordable than a standard node for the Voltage platform—and connect to it with Zeus and boom. You’re all set. You open up a couple big channels to like ACINQ or something, or Bitrefill if you’re using gift cards a lot. And off you go, and you don’t have to worry about your routing node touching any of that.

Stephan Livera:

Yeah. Okay, so let me summarize a little bit. So if you are just looking to be a very basic user and you just want to be able to spend and receive and you’re not that concerned about the privacy self-sovereignty—I mean, you can just literally use one of those phone wallet, apps, right? Like the Phoenix, Muun, Breez, some of these others. And then let’s say when you’re at the level of learning about Lightning Network nodes, and you want to run one just to learn, that’s where the RaspiBlitz and Umbrel and things like that, you can do that. And of course, there are users who do use their RaspiBlitz as their routing node. So you can absolutely do that. And then let’s say you want to take it the next level up: that’s where you have like the box at your home—as you spelled out, you’re using a mini tower approach. And I guess the main trade-offs and balances there are that you might be able to run with more beefy hardware, and therefore, because of the gossip network, because everyone’s having to run this channel’s database and all this other stuff that’s going in and out of that, having a more beefy box can help you with that. And then obviously the VPS option is great for people who are wanting to run an even larger node, and they’re maybe slightly less concerned about the self-sovereignty part of it, but they want more of the availability, maybe they want to be able to travel, and maybe they want some of the ease aspect of it, because the provider—VPS, virtual private server—is running some of that in the background for you. And the professional cloud infrastructure is being done in a way where they ensure very high uptime, very high availability, some of these professional-level concerns. So that’s probably just a basic overview of the different options that you can go with there. Now I’m also curious from a Lightning implementation point of view, because there are different implementations, right? So obviously LND from Lightning Labs is one of the big ones, Core Lightning now—it used to be called c-lightning—by Blockstream, we’ve got Rust-Lightning, and let’s call it Sensei which is utilizing that, we have the Electrum Lightning implementation, and there’s also the ACINQ implementation. Those are probably the well known ones. There might be one or two other smaller ones. Do you have any thoughts on the different Lightning implementations? Do you have any that you prefer using?

Nate G:

For sure, yeah. So I guess we just need to define what a node is really quick: a node is just software running on your computer, or somewhere. And that node software has the ability to interact with other node software. So when we say these implementations, they all operate under at least the first 11 BOLTS, which is the basis of Lightning technology—a sort of a rule set for interoperability between all the implementations. There’s some debate about BOLT 12, but we can talk about that another time. But at least the first 11 BOLTS are pretty much agreed on. So if I’m running an LND node made by Lightning Labs and you’re running a Core Lightning node made by Blockstream, we can still open up channels with each other and do things no problem. There are different features though that are on top of that base interoperability that each implementation might have. For example, Lightning Labs has the Loop service and the Pool service, Core Lightning has liquidity ads and a few other things. So LND is definitely more consumer-grade, I would say, meaning the ease of use and the build out of it is focusing a lot on user experience. For example, they have a Lightning terminal app that lets the graphical user interface easily interact with their Pool and Loop services and stuff, where Blockstream’s Core Lightning is more focused on features and maybe more of an enterprise-grade set up. There are some third party apps that provide a user interface for Core Lightning—Ride the Lightning is one of them. And then we’re getting into—so like Eclair, funny enough, so Eclair is the implementation built by ACINQ, they’ve been around since the beginning. As a matter of fact, the very first time I used Lightning was on the Eclair Wallet, which was pretty fun for the time. And I think that they’re really focused on the mobile set up a little bit. Like, Phoenix is their next gen mobile wallet that’s been out for a while that I highly recommend people try out—I think it’s really great. And now we have the LDK, the Lightning Development Kit, which is basically put together by Spiral, which is a development team funded by Block?

Stephan Livera:

Block, which used to be called Square.

Nate G:

Right. And so the LDK is cool because now if you are a developer or a development team and you have a really cool idea, you can essentially use these libraries in this development kit to piece together the foundation of your own implementation, and then you can build whatever cool stuff you want on top of that. Sensei is a really good example of that. John Cantrell, [who] you had on recently, talked about Sensei, which I’m really excited about. I’m really excited about other LDK sort of ideas. I think the child node concept is very, very interesting, and I’m excited to see where that further goes. So those are definitely the major implementations right now. And a lot of people say, Gosh, well geez, if Lightning Labs and Blockstream have a stranglehold on this for a long time, what does that mean? And I really see that someday—I mean, this might not be true, but—someday I think that there’ll be a lot of implementations all for specific use cases. There might be like a merchant-focused implementation someday that has really cool merchant accountant features, or a gaming implementation for those that someday in the future are building video games that have something to do with sats and Lightning. So I’m excited to see that ecosystem build out in the coming years.

Stephan Livera:

Gotcha. And so, from your perspective, when you’re coaching and helping educate, what are some of the typical implementations you’ve been working with? Has it typically been helping people with their LND or with their Core Lightning?

Nate G:

Yeah, it’s definitely been LND probably near 100% of the time. I still have very little experience with Core Lightning, even though my good friend Lisa [Neigut] over at Blockstream is constantly trying to get me to run one—I just haven’t had the time yet. They’re coming out with a version update here soon that they’re getting excited about: 0.11 is coming out soon—they’re already at like the third release candidate, so the final version should be out any day now. The problem is too is: Raspberry Pis are really hard to find now, so I’m trying to figure out a way to get a Core Lightning going soon. I haven’t run any other node implementations, though. I have run Sensei on reg test for fun. I don’t think that really counts though because it’s not quite there yet, but definitely LND 99.9% of the time.

Stephan Livera:

Gotcha. And so let’s talk a little bit about setting up your own Lightning node and some of the getting started points there. So let’s say you’re setting up your Lightning node: what are some of the things you should be doing at the start? Maybe in terms of channel opening as well? Like, how do you pick who you open channels with?

Nate G:

Yeah, this is a great question. And I’m assuming that maybe we want to eventually have a routing node, but we want to learn a little bit first. The first thing that you’re gonna want to do is decide how much capital you want to allocate to your node at the beginning. You want to say, Hey, I want to open up 10 million sats worth of channels or 20 million sats worth of channels. You want to have these numbers in your head—you don’t want to just open up willy-nilly because that will help you decide what size channels to open with once you get going. So for example, if I want to put a whole Bitcoin into bootstrapping a new routing node, I would probably say, Okay, I probably want 15-20 channels of like 5-7 million sats each, and just have a nice little divisibility on that. The hardest part about bootstrapping a node though—meaning, getting started—is you need inbound liquidity for other people’s payments to flow through to you, which is, for some, the hardest to get. Some people have it easier because some people might have a footprint on social media or something and they could say, Hey, I just started a new node, open up channels with me—and the whole community will jump in and give them tons of inbound. But for the normal person that just wants to like get going—and not only that, but the sovereign person that wants to not even tell anybody about it, now it’s pretty challenging, unless you start a new Nym or something and go into the Telegram groups and say, Hey, I’m starting a routing node, someone open up a channel with me, yada, yada, yada. That’s a hard sell, because routing nodes want to open up channels with folks that will reciprocate or had value to their own node as well, because your routing node is only as good as the peers you’re connected to, at the end of the day. So there has to be some sort of incentive there. So that incentive can come from something like Lightning Pool or liquidity ads for Core Lightning where you’re opening up a channel to somebody but they’re paying you to do so, basically. Now that might not be for everybody, but that is an option. One of the quickest ways I found to bootstrap that initial inbound liquidity is using atomic swap services like Loop or Boltz. What that is, is you essentially open up a channel and then you go to the service and you say, Hey—so for example, if I open up a 5 million sat channel, I could say, Hey, I want to move 4 million stats out of that 5 million sat channel to the other side. And what I’m essentially doing is I’m paying somebody those sats. And then they’re saying, Okay, pay me those 4 million sats, and I will send you those 4 million back on-chain—and now you have 4 million of inbound. And of course they take a little bit of a fee for that service, but now you’ve bootstrapped a good amount of inbound for your node. And that’s probably the hardest thing to do. One of the easiest ways to get inbound though is to just spend sats, because if you’re spending it, you’re moving it over. So if you want to spend sats on a bunch of gift cards or something, you can do that as well. Any questions so far?

Stephan Livera:

Yeah, no, that all makes a lot of sense. So in terms of who do you pick, though, for channel opening, how does a new person do that?

Nate G:

Yeah. So if I’m looking for quality peers, you’re basically playing a game with incomplete information. It’s a lot like poker. But thankfully, a lot of information about public-facing nodes and channels are put together in very easy to read public databases such as amboss.space. 1ml.com used to be the leader in that—I don’t even know who runs 1ml, actually—but they haven’t really updated it or done anything in a long time. So Tony, who built ThunderHub.io, and Jestopher, who’s a really good friend of mine also, are building this amboss.space platform where you can go and you can basically look at a node, you can see what their channels are, see what their capacities are, see what their what their fee rates are like, and that sort of thing. So there’s multiple factors. My biggest factor, though, is: how many channels does the node I’m opening up with have? And what is their average channel size? If I’m looking to open up a 5 million sat channel, I don’t want to open up a 5 million sat channel with a node that has 500,000 sat average size, right? Even with really cool things like multi-path payments, that’s still a bottleneck because not everybody uses AMP and multi-path payments right now. So I want that average channel size to be near the size that I want to open. And generally, if they have at least 20 channels, that’s good. The age of the node also needs to be taken into account, and you can go to terminal.lightning.engineering—which is a Lightning Labs ranking database using their own arbitrary metrics—but it gives you a good idea if they’re online or offline a lot, because they’ll have this pretty green color next to their node name. So I would use terminal.lightning.engineering in conjunction with amboss.space to vet nodes that I’m looking to open up with. A lot of people think that they just want to open up channels to the biggest nodes on the network and they’re gonna be all set, but for a routing node that’s not quite the best idea—it’s okay to have a few channels like that, but the issue is if you’re just connecting to big routing nodes on the network, or just big nodes in general on the network, you’re essentially competing with all the other channels that are with them. So the ultimate goal of a routing node is to find paths from medium to smaller nodes that are making payments to go to services and where you think that traffic is flowing to. So maybe you want to open up a channel with Bitrefill, and then you want to go and maybe open up some channels to some smaller nodes also, and have a variety of connections. And yeah, we could go deep into theory on that.

Stephan Livera:

Yeah, no, I think that’s a good high-level idea.

Nate G:

Which is a heck of a rabbit hole itself, but yeah.

Stephan Livera:

Yeah, for sure. I think just to give people a flavor and just to summarize, and I think that’s right: the general idea is that you want to open channels in the direction that you believe the flow is going to go. So let’s say if you knew there’s a new hot merchant in town and you believe everyone’s gonna be paying them because they’ve got some really cool Bitcoin merchandise or some other thing they’re selling for Bitcoin, you, as a smart routing node operator are thinking, Yeah, all right. Sweet—I know a lot of people want to pay this guy. So let me try and get a channel open to him because then, as they want to pay to him, they’re gonna route it through me. And so then I’m gonna collect some routing fees. And so that’s like a high-level or at least one way of thinking about that part of it. But then, like you said, you still want to have other people who want to pay you through that, so you still have to think about that aspect of it.

Nate G:

That’s a great example, because I think that as a routing node operator being constantly aware of new merchants and services that are just starting, you’re really incentivized to give them inbound liquidity, which means open up a channel with them, because you know that the flow is—especially if they’re a high-level merchant or whatever. Like, Great American Mining, for example, just enabled Bitcoin Lightning on their store. So I’m thinking, Okay, if I’m one of the first people to open up a channel with them, I can set my fee rate higher because there’s less competition right now. But it’s more competition, meaning more people open up channels to them—that might drive the fee rate of forwarding a payment to them down. That happened with the Loop node. So the Loop node at Lightning Labs is constantly doing these submarine swaps for folks, which could sometimes be really big and really high in demand. But a lot of people started realizing that and opening up channels with the Loop node and it drove that fee rate pretty much into the ground. So I think that that is a really cool incentive for merchants, because if you get going, I think you’ll see channels just open up to you naturally as part of the ecosystem.

Stephan Livera:

Excellent. And so how should people think about fee rates and setting the fees? So I guess the quick overview for people is: generally in Lightning there’s the base fee component, which is like a flagpole, like as soon as you’re routing a payment—and that might be a certain level. And then there’s the parts per million, or the fee per how much you’re routing. And listeners, also check out my earlier episode with René Pickhardt, who’s also got some thoughts on this. But Nate, I want to get your thoughts: how should users be thinking about their fee rates when they’re setting up a Lightning routing node?

Nate G:

Yeah. I mean, these days there’s a lot of talk, but I guess I’ll rewind to: the initial idea was base fee could be whatever you want it to be. Base fee, meaning: no matter what the size of payment you’re forwarding, you’re at least collecting that base fee. And then you can set a percentage on top of that that you can also take, basically. And at the time, before selling liquidity was easy as it is today, basically you would guess—you would use these 1ml or amboss and you’d say, Okay, so and so, you could see the fee rate and say, I want to undercut some of this and I’ll still make a profit. Or you can open up a channel and just set a default fee rate. A lot of the times it’s really not that much—50 or 100 parts per million, which is 0.005%, I think. So it’s not a lot. I think that if your goal is to earn fee income running a routing node, the bulk of your income is going to come from your channels to in-demand merchant-style nodes the most. So those would be a higher fee set than just a normal size node on the network, because you’re not really gonna be earning a lot from that, generally. That sometimes can change, but generally that’s true. And then after that, so say Bitrefill, for example, say that I have a 10 million sat channel to Bitrefill—it gets drained out all to the inbound. Well now I have to figure out a way to get that liquidity back into that channel so it can happen again, and I could just keep earning from that. So the challenge to that is: what is the cost of what’s called a rebalance, which is essentially paying yourself out of one of your channels and into the channel that you’re looking to profit off of. And one of the big challenges is: calculating the rebalance cost minus the fees that you get from the drain of that rebalance is your net profit, and it’s actually really, really tight most of the time, and it’s generally not that much. So I really don’t think earning fees on a node that isn’t gigantic—like the Yalls node, for example, it’s like, Cool, I get some sats or whatever. So now you have folks like @zerofeerouting who realize, Hey listen, I’m not gonna be making that much from fees, so I’m gonna provide a service saying that I’m gonna do zero fee, zero base rate, everyone can freeflow through me, and then he’s monetizing—I say, he, I don’t know who they are, it’s a Nym—by basically selling liquidity, selling channels. Saying, Hey, I’ve got this gigantic node with 500 channels—I don’t charge any fees. If you want some liquidity from me, e-mail me and we’ll discuss terms and I’ll open up a channel with you. I think that’s really interesting. I think that is the general way that earning income on Lightning is gonna go for the nearish future. In the far future, who knows what it’ll be, but in the nearish future, selling liquidity is probably going to pick up, and the volume of that market in all of its integrations or types is just gonna explode. So I really think that fees are like cool or whatever, but compared to selling a 10 million sat channel to a merchant and they’re paying you 7% APR or something, that’s gonna be where the money is for running a Lightning node, I think.

Stephan Livera:

Yeah. Also, I want to get your thoughts on payment reliability. So for some users who maybe are struggling to get more reliable payments back—out or in, to receive—what are the typical causes for that? Do you believe that’s because they’re not having big enough channels? Or they don’t have it with the right channel participants? Or maybe their Lightning node isn’t online enough? Like, what are the typical issues that you run into when you’re helping troubleshoot for someone you’re helping with?

Nate G:

Yeah. I think the hardest thing to wrap your head around when you’re brand new and just trying to understand how channels work and stuff is the liquidity, really. I mean, I’ve got folks—through no fault of their own, right?—they say, Hey, I opened up a couple channels and I’m trying to pay X and it’s not working. Or, I opened up a couple channels and I’m trying to receive a payment and it’s not working. And that could be frustrating at the start because you don’t have that foundational understanding of that, Oh, yeah—you need inbound liquidity to do that. Oh, well how do I do that? It is a big headache and it is a little bit of a learning curve, absolutely. And I hope in the future, this sort of thing will be more automated. So it’s really a lot of folks that get frustrated because they open up channels—and when I say channels, maybe two or three—and then they’re frustrated because their payment isn’t going through. And obviously the solution to that is to choose who you’re opening up your channels with very deliberately, meaning what we went over before: when you’re initially getting started, you should open up channels to bigger nodes on the network. It doesn’t have to be the biggest nodes on the network—there’s a lot of big nodes on the network now. And go from there. So yeah, if your payment’s failing, we can troubleshoot that by figuring out who you’re connected to, who they’re connected to. It might not be a good peer to begin with. Or if it’s the other way and they’re trying to receive, we need to figure out how to help them get inbound liquidity through the methods that we were talking about earlier. So that could be pretty frustrating for people.

Stephan Livera:

Yeah. And so I guess that’s the concern or the question from a payments reliability point of view. How easy would you say it is to earn sats? Like, if you’re trying to actually run a profitable routing node, do you believe that a lot of people are running a profitable routing node? Or do you think maybe, once all costs are considered—or let’s say on-chain open and close fees, swap fees, rebalancing, once you account account for all of this—do you think there are a lot of profitable routing node operators? Or do you think right now it’s still in a, You’re only gonna make a small amount?

Nate G:

Yeah. I think from fees, for 99% of people, it’s gonna be breakeven at best when you factor in all of that. Like I said earlier, I think that the idea of selling channels is probably gonna be the way to go in the future. There are, like I said, those services, and some people even taking it into their own hands proving that they have a really, really, really good node and proving that they have enough value that they think that others are willing to pay them to open up a channel with them—which is probably absolutely true! That’s what makes running a routing node exciting: you get that reputation up and then you’re like, Yeah, I’m here to help the network—and you get value for that, and that’s just gonna keep growing. I think that we don’t have to go too deep into it, but I’m pretty sure—and this is just me speculating—that the Taro network that Lightning Labs is building is going to be plugged into their Lightning Pool liquidity marketplace in some way. So even though your node might not be dealing with tokens or whatever are built on that, you could still sell channels. And I hope the volume picks up on Pool because of Taro. I think that’d be really great.

Stephan Livera:

Yeah. That’s interesting as well. So the high-level summary of some of that insight is that maybe historically, in earlier years—let’s say 2018-2019—people were thinking about this idea more in terms of the routing fee income, and that was arguably the way people were thinking about in terms of Lightning Network Reference Rate—the research by Nik Bhatia. But actually I think what might be more where the money is at is the market for opening a channel to each other. So that idea of, Hey, I’m gonna sell you this channel. So I know, in the early days, Bitrefill had their Thor channels and nowadays there’s Lightning Lab Pool, there’s liquidity ads, there’s, there’s this zero fee routing guy. There’s this model where you’re selling a channel upfront. And I know even with Voltage, you’ve got Flow. And I believe that’s working using Lightning Labs Pool. So can you tell us a little bit about how that part of it works?

Nate G:

Yeah. Flow is really, really cool. So when Pool came out about a year and a half ago now, they also had this idea of what they called sidecar channels or sidecar tickets. And all a sidecar ticket is, is essentially an order for liquidity from the Lightning Pool marketplace that somebody else can print out saying, Hey, I want 5 million sats and I’m willing to pay 5,000 sats for this. And then a little ticket comes out—which is just string of letters and numbers—and then you could redeem that ticket as if you put in the order yourself. So one of the big headaches with Pool is you have to have a Pool account, they call it, which is essentially a registered UTXO with Lightning Labs to prove that you have funds. And that’s where you pay fees in and out of Pool. But with sidecar channels, you don’t need a Pool account. So at Voltage, we run a Pool account using the Voltage sats, and we essentially have a little user interface. And Flow’s cool, because anyone using LND is welcome to go to voltage.cloud, go to Flow—no KYC or anything—and make your own sidecar ticket if you’d like and redeem it on your own node. You don’t need to be running a Voltage node. But you could say, Hey, I want a 4 million sat channel, I want to pay 10,000 sats, I want it for 2016 blocks and whatever. And you hit Create, and in the background we generate a sidecar ticket that you can go redeem. You could give it to your friend. And essentially that puts your order into the Lightning Pool marketplace as if you were running Pool yourself. So that’s been really popular. Because Voltage nodes are TOR-only, there’s been a little bit of hiccups because TOR can sometimes slow down. And the way that Lightning Pool matches, there’s a timer for the match before the total peer connection channel open thing can happen. So if TOR just times out on that, it doesn’t fully complete. But thankfully we’ve been working really hard with the team at Lightning Labs to help alleviate a lot of that. And the newest Pool version helped out a lot with matching with our sidecar tickets. And according to our metrics, about 30%-ish of the bid side of the Pool order book—meaning those looking to buy liquidity is—is done through Flow sidecar tickets right now, which is really cool.

Stephan Livera:

Oh, wow. That’s a fair amount, yeah. So I guess it is just early days of Lightning. And I know Voltage is used both for the people who want to run their Lightning node as an individual but also companies who want to run a VPS-level routing node. They might use Voltage for that. So that’s interesting to see where things are going with all of this. Also I wanted to get your thoughts on backing up a Lightning node. So how is this done currently? What are the best practices around this?

Nate G:

Yeah, so there’s two pieces of information to do a full recovery—this is strictly for LND—that’s the only one I really feel comfortable giving advice about. So when you start up an LND Lightning node, you’re provided with what’s called a cipher seed phrase. And the seed phrase that you’re given is a unique one though—you cannot use this on a Blockstream Green Wallet or a Samourai Wallet to recover. It is strictly an LND cipher seed phrase. So you write down the seed phrase—just a bunch of words, like the kind you’re used to—and then off you go. Now, every time something happens to one of your channels behind the scenes, your signature and your peer signature—because a channel is just a 2-of-2 multisig—is signing an on-chain transaction. It’s not broadcasting—but just signing it, updating the channel state. Meaning: if I pay you 500,000 sats, we both sign the new update channel state. And this is important because if we close the channel or, worst-case scenario, we force-closed the channel, these are broadcasted and we all get our money back. To go off on a side note there: for security purposes, this is another really important reason why you want to be online all the time, because there is the possibility that if you go offline, your peer could broadcast to the network that they own more sats than they actually do, in a fair world. And they could—say, before you paid them a million or before they paid you a million, they could broadcast a channel state transaction that said that they had what they actually didn’t. And now you’re out of money because you’re not online to contest the agreement—the contract. So Lighting Labs has Watchtowers, which is essentially: you can have the node of a friend or you can run a secondary node that can watch yours and keep the channel states correct. And the cool thing about Watchtowers is: any nefarious actor doesn’t know if there’s a Watchtower watching or not. And that’s important because the punishment for the justice transaction—they don’t call it that anymore, I forget what they call it now, but I still like justice transaction—is basically: if you get caught cheating, and it’s really obvious that you’re cheating, that’s an old channel state because of timestamps and everything, the person being cheated gets the whole balance. So people that want to try to use that cheat are very, very disincentivized from doing so, especially because Watchtowers are so easy to run and there’s a bunch of public Watchtowers you could connect your node to to watch your node if you go offline nowadays. All right, so let’s talk about recovery now. So say I’m running a Raspberry Pi—it broke—crap. So I’ve got my cipher seeds here. So what you would do at that point is you would start a new LND node somewhere. It could be on Voltage—it doesn’t matter where you start a new LND node. And then you recover from seed. Great. Boom, boom, boom, recover from seed. But the node that you’re now on doesn’t have the funds that are in your offline channels. You might have access to the on-chain component of your Lightning node, but you don’t have your channels. So what do you do there? Well, there’s a secondary file which is very, very small called the channel backup file or the static channel backup, the SCB—and this is what you use. This is what is constantly being updated, also, when your node opens or closes channels. It records a history of these things and it records the force close pre-signed transaction. So if you upload that also, it triggers all of those channels to force close. It can sometimes take a few days, but eventually those funds will be in the on-chain component of your Lightning wallet minus fees, on-chain—essentially, you get all your money back at that point. And a lot of people that I’ve talked to that this has happened to, they then say, Cool, I got all my sats back from my on-chain, I’m gonna start opening up channels again. And I highly recommend that, after, if this ever happens and you actually use your static channel backup and you have everything, get those funds off of that node as soon as possible, put it somewhere safe—and you can put it into a cold storage or something if it’s a lot—and then just start a brand new Lightning node with a whole new cipher phrase and everything. Just start fresh. I can’t really articulate why that’s important, but I just feel like if you’re gonna start from fresh anyway, just start fresh and you should be all good. So the static channel backup is something that is constantly put in your LND directory. You can download it also from ThunderHub if you’re using ThunderHub. If you’re using Balance of Satoshis, there’s a Telegram bot that’ll autodownload your channel backup for you into Telegram, which is really cool. And if somebody finds your static channel backup, they can’t do anything with it without your cipher seed phrase. So don’t worry about saving that on your computer or anything, because I’m assuming that your seed words are not on your computer along with it, so that shouldn’t be an issue. And there’s a few other things: I think amboss is actually doing a way for them to pull the static channel backup for you also. So you don’t have to constantly like, Ugh, I gotta go download my thing—it should be automatic, hopefully. Voltage does something similar also.

Stephan Livera:

Gotcha. Because I mean, I’m sure people are thinking, Hey, I can set up like a Cron job to automate the saving of that or pushing that somewhere else. But I mean—for people who aren’t going to that level. Also, just any thoughts around swap providers? So I know you mentioned earlier Boltz as an example. Do you have any thoughts on the use of swap providers? Should Lightning routing node runner be using them? Should you be paying for this kind of swapping in and out service?

Nate G:

Yeah. So there’s a couple scenarios where you would want to use it. And the first scenario is: you are just starting a node and you really want to get that inbound liquidity fast. And the swap like Loop or Boltz costs some money to get going, but it gets you going faster for that inbound capacity. Now a really cool strategy that I like—because I’ve bootstrapped several routing nodes at this point—I take the initial capital that I’m looking to put into my node and I take a good chunk of that and I open up a big channel to maybe the ACINQ node or something, or just like a big node. And so say, for example, I have 50 million sats. I’m gonna open up a 40 million sat channel with ACINQ and I’m gonna Loop out 35 million of it. So now I’ve got a huge chunk of inbound on that channel. I did pay for it—I probably paid over 100,000 sats to do that Loop out, but I’ve got 35 million sats of inbound. Now I could take that chunk that I got back on-chain, now I can open up my 5 or 4 million sat channels going down. And you can only do that when you’re first starting a node though—you can’t open up 5 million channels and then say, Ah crap, I should have done this. So I think that’s a really cool first step for getting a ton of inbound. And then after you do that, you could try doing the circular rebalance where you’re just trying to pay into that big channel that you have with the inbound to get everything balanced—assuming the fee rates are good for doing that. So I think that’s a really cool hack to bootstrap.

Stephan Livera:

Yeah, that makes a lot of sense. And so then the other aspect, I’m curious: how big are the payments that you can easily route through the Lightning Network today? Like, we see varying reports on social media and even with people’s own nodes, they might be easily able to do small payments, but then once you start trying to do larger payments, it becomes less reliable. Do you have any thoughts on that today and how that could be improved? Or what that means for the state of Lightning in terms of payment reliability?

Nate G:

Yeah. So the more you’re looking to pay, the less of the network you can access—just by design, that makes sense. So if I’m looking to pay 2 million sats, I essentially can just minus out all of the channels out there that are less than 2 million, and then work from there. Again, we’re not talking about AMP or anything—which is really cool. Like, AMP will let me pay out 2 million sats using three different payment paths to piece it together, which is actually amazing, and I think we’re gonna see more of that going forward. So hopefully that’ll alleviate some of the pressure in the future. And the other thing is: obviously, as the fiat exchange rate increases thanks to Bitcoin, the amount of value that a channel can process also increases. When I first started, we were sub-$10,000, so a 10 million sat channel was like, whatever. But now a 10 million sat channel is $4,000 US dollars. So that’s pretty darn good. That’s pretty good. It’s like a TV, right? So I guess it’s just like a time thing. But also, as time goes on, the ability to open up bigger channels for most people also decreases. So the routing nodes now that were around enough to be able to afford at the time was $1,000—which was 10 million sat channels—that might be a premium in the future for routing. So it is what it is at that point. It’s a double-edged sword—Number Go Up, on this. I hope that AMP and a few other of these tools that allow you to make bigger payments through multiple payment outflows really takes off though, because I think that’ll take away a lot of that pressure.

Stephan Livera:

And I’m also curious to get your thoughts around what you think the average percent of the fee is going to be taken for payments. So as an example, if we’re comparing Lightning with Visa, MasterCard, et cetera, where the merchant in the background is actually paying 3%, 4%, maybe even 5% for that payment, whereas today in the Lightning Network—what kind of numbers are we seeing people paying? Like, is it in that 0.5% range? 0.4%? Do you think it’s roughly gonna be in that range? Or do you have any thoughts where that currently is? And if you have any speculation where you think it might be?

Nate G:

Yeah. I kind of see what you’re saying. So if I’m paying for a gift card, I’m using ThunderHub, Bitrefill gives me a invoice, I take that invoice, I paste it in a ThunderHub—ThunderHub’s telling me, Okay, what is the max amount of fee that you’re willing to pay to send this transaction? And so you have to pick, Oh, I’m willing to pay 50 sats. I’m willing to pay a 100 sats. It might not be that much, but you’re just telling [it]—as it’s looking for a path to that final destination, it’s calculating the routing fees along the way and saying, Oh, this doesn’t meet the criteria—we’ll try this path. That doesn’t meet the criteria, we’ll try this path. So I don’t really think of it as a percentage. I can imagine that the high percentage routes will be brought down in time because routing nodes will catch on to that and want to open up channels with that to try to get a piece of that action, which will drive it back down. So I can’t really give a hard number on that. I mean, obviously buying something from Bitrefill is gonna cost more of a routing fee than me just paying Joe down the street that doesn’t have a big footprint on the network. That’s just the nature of the beast though. I think that the free market on that will drive fees down—I think that’d be the natural course. Some people might disagree with me. I might not be playing it through all the way, but I would think that would be the way it goes, so hopefully it drives fees down. And I think there’s a good chance that routing fees in general will just be not the focus for earning on Lightning in the future as well.

Stephan Livera:

Yeah, that’s interesting because the routing node operators still—if they’re running it professionally—they need to make money somehow. But if you’re saying that they’re not gonna make their money on that, they need to make their money in some other way, whether that’s the selling of the channel or some other means. Then I guess it’s still a bit of an open question: where does it settle down? Because of course, generally speaking, in the free market, prices are coming down. But in this case, there is a bit of an opportunity cost, right? Because you are putting your sats into a Lightning channel—there is some risk with that. It’s not just risk-free. And routing node operators and Lightning Service Providers and so on, people have to get compensated. So yeah, I guess it’s an interesting question to see where that goes, because in the early days of Lightning it might be very like hobbyist and everyone’s kind of just personally subsidizing it anyway because they just love it and they just want to do it. I’m just curious where it goes as the industry professionalizes, as it matures, as people actually have to start getting profit for it. Or, as you’re saying, maybe the profit is gonna be there, but it’s gonna come in a different way.

Nate G:

Yeah, I’m with you on that. And that’s what makes everything so exciting though. It’s only been a few years and things change all the time and there’s always something else to learn. And there’s all these cool open source apps that help you monitor your node like LNDMon and LNbits is Ben Arc. LNbits is just awesome. There’s just all these cool things. It’s kind of overwhelming, but it’s a fun hobby. And the thing is, if anyone is thinking about running a Lightning node, I recommend just diving in, join the Telegram groups, the Plebnet groups are still super active and they’re super helpful. And just give it a shot—you’ll learn a lot. And I think that it’s totally worth it, whether it’s Umbrel or RaspiBlitz or RaspiBolt—I have a soft spot in my heart for RaspiBolt, and Stadicus taught me Linux, basically, and I’m very, very grateful for that. So I think that folks that have a technical aptitude and you want to learn Linux, learning it with Bitcoin is a lot of fun too. So I recommend just diving in, and don’t worry about being overwhelmed or anything. Just take it one step at a time and you’ll learn what you need to learn.

Stephan Livera:

Fantastic. Well, I think that’s probably a great spot to finish up. So Nate, before we let you go, where can people find you online? And where can they find Voltage online?

Nate G:

Thanks Stephan. Yeah, it was a great time. So I’m @BeefOrBacon1 on Twitter. Please follow me, please DM me. Let me know if you need anything. We do have also a Voltage community, voltage.cloud/discord to join our Discord group. That’s for anyone running Lightning—we have tons of channels, talking about things all the time. And Zebedee has a Discord bot where you can sling sats at people in the Discord group, so we do that sometimes—it’s a lot of fun. And then voltage.cloud, we have a 7-day free trial on Testnet if you want to play around with the dashboard. Check it out. Please do. We’ve got a lot of cool things coming, so stay tuned for everything. And thanks again, Stephan.

Stephan Livera:

Thanks Nate. Bye.

Leave a Reply