c-lightning Q and A session

https://twitter.com/kanzure/status/1230527903969841152

Hello everyone. I’m Christian. I’m a software developer at Blockstream and I work on Lightning, both the specification and the implementation. Today we are going to have a shortish webinar on how to get started with Lightning.

Q - Is it possible to run c-lightning and lnd on the same host in parallel?

A - It absolutely is. We actually do that on a regular basis to test compatibility between the implementations. We spin up an entire network of these nodes and then we just run them against each other to see if they are able to perform actions such as opening channels, connecting and sending payments through. It is very much possible. What you need to do is to ensure that they run on different ports though because the ports are a shared resource on all processors running on the same machine. You will need to make sure that they are indeed running on different ports. The way you do that in c-lightning is —addr=: and then a port number that is not 9735. I hope that answers Dimitry’s question. You can run all of these implementations in parallel. You just need to make sure that they don’t step on each other’s toes. It is very much a scenario that we run on a regular basis ourselves. It should work fine.

Q - We recently discovered some tracking problems with cross-implementation multi hop payments. Is there a test environment with different Lightning nodes (c-lightning, lnd, eclair) on testnet so one can test multihop payments?

A - Absolutely there is. For one we have the testnet nodes that run 24/7 and basically just work exactly the same as the Bitcoin mainnet works. You might have an issue though finding which node is running which implementation because that is information that we don’t expose. You can of course use regtest to actually set up your own testing network and then test using that. Lately most of these multihop payment issues should have been addressed and if they haven’t please let us know. It might just be that one of the nodes along the route was down. From experience mostly it is a case of there not being a path or one of the nodes being temporarily offline and therefore not being able to forward payments. If you are encountering these issues and want to help debug them I strongly suggest using regtest because that actually allows us to have reproducible tests as well.

Q - How can we close a channel that won’t close a WAITING_UNILATERAL_PEER_DISCONNECTED. Age = a year?

A - Waiting unilateral is a state in which one of the two endpoints has closed the channel but in a non-collaborative way and therefore published one of the transactions that was signed off before. In this case it doesn’t seem to be a cheat attempt, just an unilateral case so this should eventually clear. Since you mention that it has been over a year that this has been stuck it might be that the closing transaction just never got confirmed. What you can do is take the database in c-lightning, extract the transaction that is the close transaction and then reissue to the Bitcoin network and therefore try to close it again. Once that confirms it will actually work out. A bit more of a sledgehammer is that if you don’t have any funds in that channel it might be safe for you to just forget the channel. For that you need to compile with the developer mode and then use the lightning-cli tool to issue lightning-cli dev-forget-channel and then the channel ID. Be careful if you had funds on that channel, you will lose them. That is also one of the reasons that this is a dev command. We expose some really dangerous stuff in the developer commands. Never run dev stuff if you are not being told to or know exactly what you’re doing.

Q - If I run bitcoind on another machine in my local network could there be any latency problems?

A - No, that’s the short answer. bitcoind is being pulled once every 30 seconds in the mainnet mode. The only thing it does is check if there is a new block. Once every 30 seconds we go out to bitcoind and ask “Is there a new block?” If yes we fetch the block. Since we already have a timeout of 30 seconds it is not really a time critical thing because in the Bitcoin world we talk about time deltas of minutes to hours. Especially when it comes to closing a channel we talk about maybe a day to close the channel. You are absolutely safe to run this over even long distance link. The important part is that it actually returns after a while. I think we start complaining if bitcoind doesn’t respond in ten seconds or so. That’s above most of the network latency that I’ve ever seen.

Q - Is there any literature out there on installing c-lightning on Windows?

A - Sadly we don’t support Windows. It is quite a different platform from what we currently develop on which is Linux. The Mac OS is there because the two operating systems are similar enough but for Windows we would need to change quite a lot of stuff. It is a lot of work but we might eventually do it. We just don’t have any Windows experts in our team. If somebody is out there and knows about Windows development please contact us. I’d definitely be happy to have a Windows version eventually.

Q - Can I safely switch from building the source to using a PPA?

A - Absolutely. The only issue that you might encounter is that the version in the PPA is older than the version you compiled your stuff from the source code. In which case c-lightning will tell you and will refuse to start. You’d basically have to wait for the next release to be newer than the source version that you installed. Then you should be good to go.

Q - Is there a UI to explore the c-lightning database?

A - The c-lightning database is just a SQLite database so if you have any tool that is able to talk to the SQLite protocol you should be able to use that. I personally use the sqlite3 command line tool because that allows you to execute a query from the database and extract the necessary information. I know there is also SQLite browser which is a graphical user interface that allows you to execute queries or just go and browse through the database in a clicky way.

Q - What are the tradeoffs between the different Lightning Network implementations?

A - That’s a very good question. After all we are actually collaborating on the specification and we try to stay interoperable. That being said we have very different goals. lnd is trying to become the go-to desktop environment. eclair is well suited for mobile devices and they are working really hard to make the resource consumption as low as possible whereas lnd tries to have as many features as possible. c-lightning, we want to make it the Swiss army knife of Lightning. We want to have c-lightning on low powered devices because it has a very small footprint. We want c-lightning to be extensible because we are of the opinion that we cannot guess what everybody wants to implement or wants to use it for. We give you the tools to build whatever you’d like to implement and we don’t prescribe a way of doing things. This starts with the way we handle our remote procedure call interface. We don’t expose that over the network because we don’t want to guess what your authentication scheme is. If you use Kerberos you can expose the RPC through Kerberos. If you want to have http JSON RPC that is fine as well. REST, all of that fun stuff. It is basically just writing a plugin away and you can integrate the c-lightning core which is well tested, well optimized into your project and make it your own. We try to aim for hobbyists or production environments that need to have some adaptation to their own needs. I’d have liked to have shown you some plugins but the demo gods are not with me. I find these live coding demos really scary. This time it didn’t go well.

Q - Are there any other ways to minimize node traffic on Lightning port other than dev-suppress-gossip and dev-channel-update-interval? I want to minimize traffic for proxying via meshnet so every packet counts.

A - That’s a challenging environment you are working in. You might want to talk to Richard Myers. Richard is working for GoTenna and he has been looking a lot into these things, how to minimize traffic. On c-lightning you can basically power down c-lightning for when you don’t need it. What you do is pre-empt the c-lightning process and you stop communicating and start it up again once you actually need it. That way you just have a short burst of activity rather than a continuous background noise. Depending on your access pattern you might get away with a much, much lower network footprint than if you run c-lightning continuously. That being said, depending on how much you want to get your hands dirty you might be able to build a very low network traffic mode in there. The networking code is actually quite accessible so I would probably try that.

Q - Is there any channel rebalancing strategy implemented in c-lightning?

A - Not c-lightning itself. Like I said before our goal is not to come with all the batteries included, even the ones you don’t need. What we do is we give you the tools to build this kind of thing yourself and we have a repository on GitHub collecting all of these community contributed plugins. One of them is actually a rebalancing plugin. What we give the plugins is all the little primitive tools that are needed to actually build more complex functionality on top. One of these tools is the sendpay command the waitsendpay and the getroute command and using those three you can build a rebalancing plugin yourself or as I said we do have an existing rebalancing plugin that does quite well. If it doesn’t suit your needs it is a tiny Python script that is rather easy to adjust or to adapt to your own needs. If Python isn’t your thing the plugin protocol is really just talking JSON over standard input and standard output. So if you have ever written a “Hello world” you are able to write a plugin in that language you have written the “Hello world” in.

Q - In lnd if you don’t backup channel.backup file you are at risk of losing your in-channel funds. Is c-lightning similar in this respect?

A - Backups in Lightning are always a challenging issue because if you restore a version of your node that is not the latest one and you then try to perform any action on the Lightning channel, be that a fee rate update or an actual payment going through. You are basically going back to an old state and that is cheating in Lightning. The way that lnd solves this or tries to solve this is by relying on the counterparty not to be malicious. What they do is basically backup a minimal version of the channel state at the startup and then when restoring from that they go out and talk to the peer and say “Hey. Is this the latest state?” If the peer says “Yes this is the latest state” then it will continue. Otherwise it will try to close the channel or it will tell the counterparty to close because it can’t do it itself anymore. It is very much relying on the counterparty. With c-lightning we go a different route. We don’t want to rely on our counterparty to be non-malicious. We want to have full control over our channels. What we encourage instead is to avoid data loss in the first place. What we do is we have the database that is the holding the current state of our channels and we allow you to synchronously replicate it through the use of plugins. What you’d be doing is on every change of the channel we tell the backup plugin about this. The backup plugin stores this somewhere and then reports back. Only then do we apply this change. This allows you to write a backup synchronously to some offsite storage and then restore from there. A more recent development is to allow different database backends as well and that’s what I am currently working on. We do have very strong, very reliable databases that have very strict durability guarantees so we can have MySQL, we can have MariaDB, we can have PostgreSQL. If you set them up in a production environment you can replicate them and use those databases as a backend for c-lightning itself. Even if one of your machines dies, takes a replica of the database with it, you can still continue operating on the remaining replicas. Then bring in a new replica in to replace the failed one. This is more into the direction of a production environment where you have to have high reliability and fault tolerance. For the lower end of the spectrum we usually go for backup plugins that replicate any changes to our databases in realtime to offsite storage. I hope that answers the question.

Q - Is there recommended or safe block height, date, gigabytes for using a pruned bitcoind node with c-lightning or does c-lightning control the bitcoind pruning so that it doesn’t delete blocks that it needs?

A - Currently we don’t instruct bitcoind to do anything. We only use read acts to bitcoind. Being able to tell bitcoind which blocks it can delete or not is actually write access to bitcoind, we try to avoid those. Generally speaking it is quite ok for you to prune starting from block height 500,000 upwards because we didn’t have any Lightning channels below that. You will not need those blocks. Since the last release we will also cache most of this stuff in our internal database such that if you’ve seen a channel onchain once then you will be able fetch that result from your own database and not need bitcoind for it anymore. If you start it at block height 500,000 and let c-lightning run alongside bitcoind while they’re syncing it should be safe to discard all blocks even if there are channel announcements in there. Alternatively you can always rely on block explorers. You can build a small wrapper around bitcoin-cli that will go out and check on block explorers whether that could actually exist or not. That is how I personally run most of this stuff. I have bitcoind providing me with the latest couple of thousand blocks and for historical data I will go out and fetch these from a block explorer. It is not as much a security issue. If you get a wrong result from a block explorer about the existence of a channel the worst thing that can happen is that you take a channel into your map of channels that doesn’t exist and you will notice when you try to pay through it because the payment will not work. Or you can get lied to and be told “No this channel does not exist” in which case the channel will be missing from your view of the network and you will simply route around it. It is not as much a security issue as if you were to run the latest thousand blocks off of some external source.

Q - When running lnd and c-lightning on the same host in parallel, namely if you run in production, is it possible to run both nodes publicly?

A - That’s definitely the case. If you run them on different ports then they will gossip those different ports out as well so people will be able to connect to them or learn about the existence of these nodes independently of their port. They will be able to connect as long as you allow incoming connections to them. Yes they will also show up on Lightning explorers and show your nice name and color. There is no clash in IP addresses alone. They just need to have different ports and need to be reachable from outside to be useful.

Q - Is using Lightning Charge still a good option to use for payment integration into web services?

A - Absolutely. Lightning Charge is a wrapper around our UNIX socket based JSON-RPC and it basically exposes the JSON-RPC as a REST interface to the network including authentication and encryption and a couple of wrappers. It predates plugins but it is still very much a good option for you to get access remotely to your Lightning node if you need that. If you have a wallet on your phone for example and it needs to connect back to c-lightning or you have a service that needs to make use of c-lightning functionality you can use that quite easily. Nowadays we would probably make that a plugin because that just makes it easier to run the REST API alongside of c-lightning. You don’t have to manage a separate process. It gets some privileged access to the internals namely notifications and hook callbacks. I personally still use it very much and I’m a huge fan of Lightning Charge.

Q - Is power channel fee setting possible in c-lightning now?

A - I’m not sure what power channel fee setting is but you can adjust the fees in a very detailed manner on each channel. Even during operation of the node. If you want to change the base fee or the proportional fee you can just use the command, let me quickly check because I don’t remember. All of these can be found using lightning-cli help and lightningd —help. I’m never set my channel fees to anything. What you can do is use lightning-cli setchannelfee and then the peer node ID for the channel you’d like to change. That will switch over your fee to a new setting. It takes a while for the information to propagate so don’t do it too often. If a payment comes in that has an old fee attached to it and it is lower than what you currently expect, the payment will just fail. Use it sparingly.

Q - What are the plans for providing plugins in hooks with TLV data for message extensions? Also is it planned to add support for more hooks covering the rest of LN messages? It would be great to be able to modify commitment transaction messages from plugins, like doing pay-to-contract key tweaks.

A - What we currently do is you have a good use case for a notification or a plugin or a hook or a RPC method that you’d like to have please come to us. We add them on a per-need basis. That being said there are things that we are unlikely to do. We probably don’t want to give access to raw peer messages because there is simply isn’t a lot of use for them. They are basically just for coordination. What we can do however is give you access to some higher level messages for example. We have the htlc_accepted hook which is fired every time there is an incoming payment whether it is destined for you or you should be forwarding it. That includes the entire payload including the information about the HTLC, the onion that was attached to the HTLC and any additional information that was in there like CLTV delta or how much fees did you get. This kind of thing is very important because it gives you a lot of power and makes some really, really nice use cases possible. For example, the htlc_accepted hook can be used for short circuiting payments. Or you can enforce any kind of policy that you’d like for forwarding. Say you don’t want to forward anything that is above 10 dollars because maybe that’s just your personal limit of what should be doable on Lightning. Or you want to block a payment with an odd bit on a certain position in the R hash, that is all possible. We are very much interested in expanding these capabilities in the future so please head over to our GitHub and write up a short description of what you would be using it for and what you’d be needing. Then we can hammer out the details and hopefully get you the tools you need to implement this functionality that you’re looking for. Especially for TLV inside of the onion…

One thing I wanted to say at the end was basically for technical issues it is probably best to use GitHub. For anything else please feel free to reach out to me and I’ll do my best to get it going. I hope this was useful to you. It was definitely an experience for me. Hope you got something out of it.