This section sets up the docker container insert-gossip which is able to read gossip messages in bytes format from a specified file. Such a file is e. g. the gossip_store file that gets created by a Core Lightning node automatically.

In case you want to get a deeper understanding about this file, feel free to check out the official documentation as well as the source code of the insert-gossip service.

#Setup

We will start by setting up the directory structure and continue with the docker container

#Create directory

From the project root we create the insert-gossip directory with the following command.

#Environment variablesauthentication

Update the .env file and add credentials of your RPC User to access your Bitcoin Core node. The easiest way would be to just copy your gossip_store file from your Core Lightning node into the ln-history directory.

FILE_PATH=gossip_store                                                  # File path to gossip_messages
BITCOIN_RPCUSER=YOUR-BITCOIN-RPC-USER
BITCOIN_RPCPASSWORD=YOUR-BITCOIN-RPCPASSWORD

🔐 Important: Never commit .env files containing credentials to version control.

#🐳 Docker container

Add this service to your docker-compose.yml.

services:
  insert-gossip:
    image: ghcr.io/ln-history/ln-history:latest
    container_name: insert-gossip
    restart: never ?
    environment:
      FILE_PATH: $FILE_PATH
      BITCOIN_RPCUSER: $BITCOIN_RPCUSER
      BITCOIN_RPCPASSWORD: $BITCOIN_RPCPASSWORD
      LN_HISTORY_DATABASE_CONNECTION_STRING: $POSTGRES_URI
    env_file: .env
    volumes: file_path?

#Folder structure

Ultimatly the folder structure should look like this:

ln-history/
├── .env                    # .env file with credentials
├── docker-compose.yml      # Docker Compose with insert-gossip and core services
└── gossip_store            # The file that contains the gossip messages in raw bytes

Please make sure that the permissions are correctly set, such that the insert-gossip service has the permissions to read the file at FILE_PATH. info

#Noticewarning

Please note that for every channel_announcement a lookup into the Bitcoin blockchain is necessary. This might take a while depending on your hardware.

#Run

We start the service.

This will start the insert-gossip to go through your inserted file.

#Monitoring

You can monitor the progess of the insertion by executing the following command.

You can always access the logs by pressing CTRL + C.

#Phase 1: Splitting

The insert-gossip service will first iterate through the whole file and log statistics about the parsable gossip messages found. During that run it is sorting the gossip messages into three seperate files node_announcements.bin, channel_announcements.bin, channel_updates.bin. You can see the statistics in the logs:

#Phase 2: DuckDB Setup

The insert-gossip service will setup a DuckDB - an in-memory database - that temporarly stores the gossip messages in the file temp-db.duckdb. The DuckDB has the same schema as the ln-history-database.

After that it will go through the phases 2a -> 2b -> 2c.

#Phase 2a: node_anouncements

The service interates through the nodes_announcements.bin. For every node_announcement it checks if the node_id has been seen before. If not it inserts the node_id into the nodes table of the DuckDB. In any case it adds the gossip message into the nodes_raw_gossip table.

#Phase 2b: channel_announcements

The service interates through the channel_announcements.bin. For every channel_announcement it needs to request the Bitcoin Core to get the amount_sat and timestamp (timestamp of the Bitcoin block when it was mined) of that channel. It also checks if the node_ids of the participating nodes namely node_id_1 and node_id_2 exist in the nodes table. In case they don't, the missing node_id gets inserted with the from_timestamp (and last_seen) being initially the timestamp of the block. For every channel_announcement a new row gets created in the channels table.

#Phase 2c: channel_updates

The service iterates through the channel_updates.bin. For every channel_update it checks if the channel has been announced before. If not it creates a new entry in the channels table. Every channel_update gets inserted into the channel_updates table.

#Phase 3: Export data

After the duckdb tables in the temp-db.duckdb have been filled completly, the insert-gossip services exports each table as a parquet file (?). As the last step of the insertion, the service imports the created parquet (?) files into the postgresql database.

#Phase 4: Import data

#Finish

The insert-gossip service is concepted to run once over a given file and does not persist information about which files have already been read. To help the user keep track of the files it appends a .done to the file that was imported into ln-history-database.

It has finished when you see this in your logs:

#More information

Feel free to read the detailed README.md file in the repository.