Introduction

Welcome to the documentation of the ln-history project. In this guide we will build the infrastructure to persist gossip messages of the Bitcoin Lightning network. I tried to make this setup process as simple as possible. If you struggle at some point or find an error, feel free to create an Github Issue here.

Project structure

Depending on your preferences, ln-history can be used in three different modes:

Mode	Description	Difficulty
Direct Request	Request the ln-history infrastructure and parse results locally	very easy
Bring Your Own Data	Bulk insert your self collected gossip messages into the database once	easy
Real Time	Insert your collected gossip messages in real time into the database	harder

The following diagram shows you which steps you need to fulfill to setup the infrastructure. The platform supports running bulk-import and real-time simultaneously.

I recommend the easiest mode for starters to familiarize yourself, see direct_request. Here you can play around with gossip messages from the ln-history plaform that I have collected, like creating snapshots etc.

The Bring Your Own Data and Real Time mode should run on a server and utilize docker compose. The latter will also need a running Bitcoin Core node as well as Core Lightning node(s). I have tested the real-time mode with an static ipv4 address attached to the Core Lightning node.

After finishing the setup you will be provided with endpoints:

Snapshot generation for a given timestamp
Snapshot diff calculation for two given timestamps
Gossip by node_id for a given timestamp
Gossip by scid for a given timestamp

Note that the format of the results is raw bytes. They need to get parsed by a client. The lnhistoryclient is a python package that provides classes, functions and interfaces to easly parse the gossip (in raw bytes) results into more readable formats, featuring JSON and python networkx.

When deploying the real-time mode you can also get live gossip analysis, see for example here

Architecture

Here you can see an visualization of the architecture

I do not have any gossip messages

You can still use this tool and do it yourself. Dr. Christian Decker collected and uploaded gossip messages ranging between 01-2018 to 09-2023. See this repository, which provides links to his gossip messages. You can download this data for free.

With ln-history I provide the infrastructure to analyze the gossip messages. Maybe this is all you are looking for?

Prerequisits

The infrastructure relies on docker (and docker compose) trying to make it as platform independent as possible. Nonetheless, I highly recommend running it on a Linux machine (specifically Ubuntu), as that is the only environment I tested it on.

Docker installation

We follow the official documentation.

Setup up Docker's apt repository
Install the (latest) Docker packages

ln-history@host:~₿ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Verify that the installation is successful by running the hello-world image and check if you should see this result:

ln-history@host:~/ln-history₿ sudo docker run hello-world

Hello from Docker! This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

The Docker client contacted the Docker daemon.
The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64)
The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
The Docker daemon streamed that output to the Docker client, which sent it to your terminal.

To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash

Docker compose installation

We follow the official documentation

We download and install the Docker Compose standalone

ln-history@host:~₿ curl -SL https://github.com/docker/compose/releases/download/v2.39.1/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
We apply executable permissions to the standalone binary in the target path for the installation

ln-history@host:~₿ chmod +x /usr/local/bin/docker-compose

Folder structureoptional

On your machine navigate to your desired location where the project will live. You might want to create a specific user for that.

Create user

ln-history@host:~₿ sudo adduser lnh

You can then add this user to the sudo group if needed:

ln-history@host:~₿ sudo usermod -aG sudo lnh

Switch to the new user:

ln-history@host:~₿ su - lnh

Folder creation

We will now create the project root folder ln-history where all the services will be created in.

ln-history@host:~₿ mkdir ln-history && cd ln-history

ln-history/
├── .env                    # Environment variables
├── docker-compose.yml      # Docker Compose setup
└── service-name.conf       # Optional configuration file(s)

Ready - Set - Go

First setup the ln-history Core which consists of the database and the Backend. After that continue with Bring Your Own Data or Real Time depending on which mode you want to use.

In case you want to learn about the structure and format of Lightning Network gossip messages, please check out the gossip page.

ON THIS PAGE

#Introduction

#Project structure

#Architecture

#I do not have any gossip messages

#Prerequisits

#Docker installation

#Docker compose installation

#Folder structureoptional

#Create user

#Folder creation

#Ready - Set - Go