monero/src/blockchain_converter/README.md


For importing into the LMDB database, compile with `DATABASE=lmdb`

e.g.

`DATABASE=lmdb make release`

This is also the default compile setting on the blockchain branch.

By default, the exporter will use the original in-memory database (blockchain.bin) as its source.
This default is to make migrating to an LMDB database easy, without having to recompile anything.
To change the source, adjust `SOURCE_DB` in `src/blockchain_converter/bootstrap_file.h` according to the comments.

# Usage:

See also each utility's "--help" option.

## Export an existing in-memory database

`$ blockchain_export`

This loads the existing blockchain, for whichever database type it was compiled for, and exports it to `$MONERO_DATA_DIR/export/blockchain.raw`

## Import the exported file

`$ blockchain_import`

This imports blocks from `$MONERO_DATA_DIR/export/blockchain.raw` into the current database.

Defaults: `--batch on`, `--batch size 20000`, `--verify on`

Batch size refers to number of blocks and can be adjusted for performance based on available RAM.

Verification should only be turned off if importing from a trusted blockchain.

```bash
# use default settings to import blockchain.raw into database
$ blockchain_import

# fast import with large batch size, verification off
$ blockchain_import --batch-size 100000 --verify off

# LMDB flags can be set by appending them to the database type:
# flags: nosync, nometasync, writemap, mapasync
$ blockchain_import --database lmdb#nosync
$ blockchain_import --database lmdb#nosync,nometasync
```

## Blockchain converter with batching
`blockchain_converter` has also been updated and includes batching for faster writes. However, on lower RAM systems, this will be slower than using the exporter and importer utilities. The converter needs to keep the blockchain in memory for the duration of the conversion, like the original bitmonerod, thus leaving less memory available to the destination database to operate.

```bash
$ blockchain_converter --batch on --batch-size 20000
```
Add README for blockchain converter, importer, and exporter utilities 2015-03-22 19:57:14 +02:00
			For importing into the LMDB database, compile with `DATABASE=lmdb`

			`e.g.`

			`DATABASE=lmdb make release`

			`This is also the default compile setting on the blockchain branch.`

			`By default, the exporter will use the original in-memory database (blockchain.bin) as its source.`
			`This default is to make migrating to an LMDB database easy, without having to recompile anything.`
Update blockchain utilities with portable bootstrap file format Remove repeated coinbase tx in each exported block's data. Add resume from last exported height to blockchain_export, making it the default behavior when the file already exists. Start reorganizing the utilities. Various cleanup. Update output, including referring to both height and block numbers as zero-based instead of one-based. This better matches the block data, rather than just some parts of the existing codebase. Use smaller default batch sizes for importer when verifying, so progress is saved more frequently. Use small default batch size (1000) for importer on Windows, due to current issue with big transaction sizes on LMDB. file format ----------- [4-byte magic \| variable-length header \| block data] header ------ 4-byte file_info length file_info struct file format major version file format minor version header length (includes file_info struct) [rest of header, padded with 0 bytes up to header length] block data ---------- 4-byte chunk/block_package length block_package struct block txs (coinbase/miner tx included already in block) block_size cumulative_difficulty coins_generated 4-byte chunk/block_package length block_package struct [...] 2015-05-08 23:46:26 +03:00			To change the source, adjust `SOURCE_DB` in `src/blockchain_converter/bootstrap_file.h` according to the comments.
Add README for blockchain converter, importer, and exporter utilities 2015-03-22 19:57:14 +02:00
			`# Usage:`

			`See also each utility's "--help" option.`

			`## Export an existing in-memory database`

			`$ blockchain_export`

			This loads the existing blockchain, for whichever database type it was compiled for, and exports it to `$MONERO_DATA_DIR/export/blockchain.raw`

			`## Import the exported file`

			`$ blockchain_import`

			This imports blocks from `$MONERO_DATA_DIR/export/blockchain.raw` into the current database.

			Defaults: `--batch on`, `--batch size 20000`, `--verify on`

			`Batch size refers to number of blocks and can be adjusted for performance based on available RAM.`

			`Verification should only be turned off if importing from a trusted blockchain.`

			```bash
			`# use default settings to import blockchain.raw into database`
			`$ blockchain_import`

			`# fast import with large batch size, verification off`
			`$ blockchain_import --batch-size 100000 --verify off`

			`# LMDB flags can be set by appending them to the database type:`
			`# flags: nosync, nometasync, writemap, mapasync`
			`$ blockchain_import --database lmdb#nosync`
			`$ blockchain_import --database lmdb#nosync,nometasync`
			```

			`## Blockchain converter with batching`
			`blockchain_converter` has also been updated and includes batching for faster writes. However, on lower RAM systems, this will be slower than using the exporter and importer utilities. The converter needs to keep the blockchain in memory for the duration of the conversion, like the original bitmonerod, thus leaving less memory available to the destination database to operate.

			```bash
			`$ blockchain_converter --batch on --batch-size 20000`
			```