4rkal/monero
1
0
Fork 0
mirror of https://github.com/monero-project/monero.git synced 2024-12-16 13:26:31 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,390 Commits 9 Branches 69 Tags 249 MiB
4c7f8ac04f
Commit Graph

227 Commits

Author SHA1 Message Date
moneromooo-monero
eb56d0f994
blockchain_db: add functions for adding/removing/getting rct commitments 2016-08-28 21:28:11 +01:00
moneromooo-monero
f88029e72a
db_lmdb: do not try to modify the database in read only mode 2016-08-26 19:33:25 +01:00
Riccardo Spagni
0faf572db8
Merge pull request #948 
11dc091 Fake outs set is now decided by the wallet (moneromooo-monero)
1593553 new unlocked parameter to output_histogram (moneromooo-monero)
 2016-08-11 22:43:14 +02:00
moneromooo-monero
240864f529
db_lmdb: errors dropping hf starting height db on reset are not fatal 
This db is now dropped unconditionally, so may or may not be there
in the first place.
 2016-08-10 10:50:17 +01:00
moneromooo-monero
1593553e03
new unlocked parameter to output_histogram 
This constrains the number of instances of any amount
to the unlocked ones (as defined by the default unlock time
setting: outputs with non default unlock time are not
considered, so may be counted as unlocked even if they are
not actually unlocked).
 2016-08-01 22:16:00 +01:00
moneromooo-monero
121165f222
db_lmdb: add some missing api call checks 2016-07-26 22:39:51 +01:00
moneromooo-monero
d7b681cd65
remove hf_starting_height db 
It's not really needed, it used to be an optimization for when
that code was not using the db and needed to recalculate things
fast on startup.
 2016-07-13 21:38:34 +01:00
Riccardo Spagni
30dce45643 Merge branch 'performance' of https://github.com/LMDB/bitmonero 2016-04-29 21:41:43 +02:00
moneromooo-monero
513a658c87
add a --max-concurrency flag 
It sets the max number of threads to use for a parallel job.
This is different that the number of total threads, since monero
binaries typically start a lot of them.
 2016-04-28 20:33:59 +01:00
Howard Chu
2b0fa05f0d Another take on migration 
Delete old indices and recreate them, rather than updating them
Maybe not quite as slow as before.
 2016-04-24 17:46:50 +01:00
Howard Chu
c14f9efd52 Migration 
Migrate from DB version 0 to version 1 on startup
 2016-04-08 03:11:05 +01:00
Howard Chu
d7ea7d9a23 Merge branch 'performance' into master 2016-04-05 21:13:16 +01:00
Howard Chu
372acee723 Cleanup 
drop obsolete remove_output()
fix get_output_key(global), fix crash in blockchain_dump
 2016-04-05 21:05:24 +01:00
Howard Chu
591e421875 Cleanup and clarify 
Try to rationalize the variable names, document usage.
 2016-04-05 20:57:45 +01:00
Howard Chu
b2f1c58805 Use cursors in some remove functions 
Helps when they're called repeatedly in one txn
 2016-04-05 20:57:31 +01:00
Howard Chu
118dd69dd5 Use DUPFIXED for block_info and output_txs 
Saves another ~150MB or so on the full blockchain
 2016-04-05 20:55:16 +01:00
Howard Chu
6225716f3c More outputs consolidation 
Also bumped DB VERSION to 1
Another significant speedup and space savings:
Get rid of global_output_indices, remove indirection from output to keys

This is the change warptangent described on irc but never got to finish.
 2016-04-05 20:55:12 +01:00
Howard Chu
7c5abdc3a3 Use DUPFIXED for output_keys 
Saves another 90MB on 200000 block import.
Had to bring back compare_uint64 for this, but it's safe since
this table is always 64-bit aligned.
 2016-04-05 20:54:42 +01:00
Howard Chu
8e9d8e3364 Use DUPFIXED for tx_indices 
Small space savings, no measurable speedup
 2016-04-05 20:54:39 +01:00
Howard Chu
a12f9365f8 Use DUPFIXED for block_heights 
Only a small savings...
 2016-04-05 20:54:07 +01:00
Howard Chu
38c2277d6f Use DUPFIXED for spent_keys 2016-04-05 20:54:07 +01:00
warptangent
bf769c32ba Add back changes from revert. 
m_tx_outputs doesn't need to be changed, as it's no longer dup list.
 2016-04-05 20:54:07 +01:00
warptangent
46b991b362 Use MDB_APPEND mode with two tx subdbs 
This is possible on those using a tx index as a key.
 2016-04-05 20:54:07 +01:00
warptangent
9aadedb1d0 Schema update: tx_indices - consolidate the tx subdbs from 5 to 3 2016-04-05 20:54:06 +01:00
warptangent
a2f518aa01 Schema update: tx_indices - yet less indirection 2016-04-05 20:54:06 +01:00
warptangent
8d12a8df2c Schema update: tx_indices - improve further with less indirection 2016-04-05 20:54:06 +01:00
warptangent
7c013f66e9 Add batch warning for further review 2016-04-05 20:54:06 +01:00
warptangent
ae0854a431 Schema update: tx_indices 2016-04-05 20:54:06 +01:00
Howard Chu
8d252a4214 Consolidated block info 2016-04-05 20:53:59 +01:00
warptangent
132c666f67 Update schema for "tx_outputs" to use array containing amount output indices 
This speeds up wallet refresh by directly retrieving a tx's amount output indices.

It removes the indirection and walking the amount output duplicate list
for every amount in each requested tx.

"tx_outputs" is used by:
Amount output indices are needed for wallet refresh.
Global output indices are needed for removing a tx.

Both amount output indices and global output indices are now stored in
an array of 64-bit unsigned ints:

tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ]

Previously it was:
tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...>

The amount output list had to be walked for every amount in order to
find each amount's output index, by comparing the amount's global output
index with each one in the duplicate list until a match was found.

See also d045dfa7ce
 2016-04-05 20:30:50 +01:00
Howard Chu
efbdde2c66 Detect map resize failures 2016-03-30 20:37:28 +01:00
moneromooo-monero
600a3cf0c0
New RPC and daemon command to get output histogram 
This is a list of existing output amounts along with the number
of outputs of that amount in the blockchain.

The daemon command takes:
- no parameters: all outputs with at least 3 instances
- one parameter: all outputs with at least that many instances
- two parameters: all outputs within that many instances

The default starts at 3 to avoid massive spamming of all dust
outputs in the blockchain, and is the current minimum mixin
requirement.

An optional vector of amounts may be passed, to request
histogram only for those outputs.
 2016-03-26 21:10:43 +00:00
Riccardo Spagni
0d30b65785 Merge pull request #749 
bfd4a28 Update BlockchainDB documentation (Thomas Winget)
797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget)
c835215 remove defunct code from cryptonote::core (Thomas Winget)
50dba6d cryptonote::core doxygen documentation (Thomas Winget)
8ac329d doxygen documentation for difficulty functions (Thomas Winget)
540a76c Move checkpoint functions into checkpoints class (Thomas Winget)
1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget)
89c24ac Remove unnecessary or defunct code (Thomas Winget)
ab0ed14 doxygen include private and static members (Thomas Winget)
3a48449 Updated documentation for blockchain.* (Thomas Winget)
 2016-03-25 09:12:25 +02:00
Riccardo Spagni
01e0a69c17 Revert "Merge pull request #749" 
This reverts commit 7fa63a82a1, reversing
changes made to cb6be986c3.
 2016-03-25 08:42:42 +02:00
Thomas Winget
bfd4a28c41
Update BlockchainDB documentation 
BlockchainDB is now Doxygen-compliant and its documentation is
up-to-date with recent changes.
 2016-03-24 03:40:04 -04:00
moneromooo-monero
f7301c3563
Revert "Print stack trace upon exceptions" 
Ain't nobody got time for link/cmake skullduggery.

This reverts commit fff238ec94.
 2016-03-21 10:12:23 +00:00
Riccardo Spagni
37fbb7a8b7
Merge pull request #738 
79117d4 db_lmdb: include the error codes from lmdb api in error logs (moneromooo-monero)
 2016-03-20 21:29:11 +02:00
moneromooo-monero
79117d4275
db_lmdb: include the error codes from lmdb api in error logs 2016-03-20 18:06:04 +00:00
moneromooo-monero
fff238ec94
Print stack trace upon exceptions 
Useful for debugging users' logs
 2016-03-19 21:48:36 +00:00
Howard Chu
db1b2db4d5 Reduce log noise 2016-03-19 12:59:05 +00:00
Howard Chu
a74348e115 Add destructor for readtxns 
Only if we created the readtxn. Was missing cleanups from exceptions before.
 2016-03-16 11:34:13 +00:00
Howard Chu
6b0a903177 Small cleanups 
Only one return and TXN_POSTFIX_RDONLY() per function
Only log rtxn_start if the rtxn wasn't already active
 2016-03-15 13:35:31 +00:00
Howard Chu
01c1512f22 More for 92dd4ec6d6 
Make sure we stop the right txn too
 2016-03-15 13:35:23 +00:00
Riccardo Spagni
240a50f3fb
Merge pull request #723 
2abdb2c avoid some val copies (Howard Chu)
 2016-03-14 22:20:42 +02:00
Howard Chu
92dd4ec6d6 Hack for read/write txn mixup 
save the thread ID of the writer thread so we don't try to use
the writetxn from reader threads
 2016-03-14 20:19:46 +00:00
Howard Chu
2abdb2c9fd avoid some val copies 2016-03-14 09:40:49 +00:00
Riccardo Spagni
dfd0e9c97d
Merge pull request #712 
66c2fc7 Need to link boost::chrono in more places now (Howard Chu)
b937a2c Use boost::thread instead of std::thread (Howard Chu)
 2016-03-11 20:15:10 +02:00
Howard Chu
66c2fc7b70 Need to link boost::chrono in more places now 2016-03-11 16:17:08 +00:00
Howard Chu
b937a2c915 Use boost::thread instead of std::thread 
and all other associated IPC
 2016-03-11 15:09:50 +00:00
Howard Chu
8941ce0398 More for bdec7cb 
More uses of db error helper
 2016-03-09 18:24:16 +00:00
Riccardo Spagni
32dc08d19a
Merge pull request #700 
bdec7cb BlockchainLMDB: Use DB error helper consistently (warptangent)
c5932eb BlockchainLMDB: Add DB error to exception (warptangent)
a49c355 Blockchain: Omit verbose time stats messages by default (warptangent)
 2016-03-05 23:10:38 +02:00
Riccardo Spagni
c3af15702f
Merge pull request #698 
ee7a8b8 Get rid of lmdb_cur (Howard Chu)
 2016-03-05 23:06:26 +02:00
warptangent
bdec7cbfb3
BlockchainLMDB: Use DB error helper consistently 2016-03-03 21:27:13 -08:00
warptangent
c5932eb5ea BlockchainLMDB: Add DB error to exception 2016-03-03 20:15:37 -08:00
Howard Chu
ee7a8b87f9 Get rid of lmdb_cur 
We don't need it now with per-txn cursors.
 2016-03-03 19:06:27 +00:00
Howard Chu
f5affbef26 More for df239428c0 
Let ARMv7 work again
 2016-03-03 00:13:13 +00:00
warptangent
2c823c1589
Make partial revert consistent 
If user-defined comparator is used, subdb shouldn't be opened with
MDB_INTEGERKEY.

TODO: Again, this will be added back with future schema updates.
 2016-03-01 02:47:16 -08:00
warptangent
df239428c0
Revert part of 7db89ed2ee 
For now, so existing databases work.

TODO: add these back with future schema updates.
 2016-02-29 18:03:41 -08:00
warptangent
c9c4060dc6
BlockchainLMDB: Add height and db error to exception 2016-02-29 17:52:54 -08:00
Riccardo Spagni
d60bf4ee36
Merge pull request #686 
7db89ed ARMv7: fix unaligned accesses (Howard Chu)
5a07cef Wrap some more actions in a larger read txn (Howard Chu)
8cc7a36 read txn/cursor stuff (Howard Chu)
86a7f2b core: check whether an update is needed straight away (moneromooo-monero)
ea5fa5e core: print "update needed" hard fork notifications in red (moneromooo-monero)
 2016-02-29 17:29:26 -05:00
Howard Chu
7db89ed2ee ARMv7: fix unaligned accesses 
And cleanup some key comparators
 2016-02-25 13:57:00 +00:00
Howard Chu
8cc7a36f0b read txn/cursor stuff 
Could wrap more later.
 2016-02-23 20:47:15 +00:00
moneromooo-monero
0eb4d054f6
db_bdb: add missing include for std::condition_variable 2016-02-20 15:38:43 +00:00
warptangent
57e75fa33f
BlockchainBDB: Check if hard fork subdbs need reset 
See f7e337e625 for LMDB equivalent.
 2016-02-18 06:38:56 -08:00
warptangent
47f6cf8bea
BlockchainBDB: Support blockchain_import --drop-hard-fork command 
See c657e772c4 for LMDB equivalent.
 2016-02-18 06:37:45 -08:00
Howard Chu
02abe3590d Use MDB_PREV_MULTIPLE 
in get_global_output_indices
 2016-02-17 16:14:50 +00:00
Howard Chu
7a4755d3a4 Fixup after lmdb master resync 2016-02-17 16:14:38 +00:00
Howard Chu
3b13a7473e Shutup about VERSION 0 2016-02-17 05:06:19 +00:00
Howard Chu
1537477c9f Use cursor in get_output_key 2016-02-17 04:05:29 +00:00
Howard Chu
f2faf8cdd9 Use MDB_APPEND mode where possible 
When keys are contiguous and monotonically increasing, this gets
denser page utilization (doesn't leave padding in page splits).
Can't be used for keys that are inserted in random order (e.g. hashes)
In total this only saves around 1.5% of space compared to original
DB code. The previous patch accounted for 0.8% savings on its own;
the blocks tables just aren't that big.
 2016-02-17 04:05:29 +00:00
Howard Chu
090b548c3b Use cursors in write txns 
Saves a bit of seek overhead. LMDB frees them automatically
in txn_(commit|abort) so they need no cleanup.
 2016-02-17 04:05:29 +00:00
Howard Chu
ed08d2152e Keep a running blocksize count 
Used in batch size estimation, avoids rereading already processed
blocks during import
 2016-02-17 04:05:28 +00:00
Howard Chu
0fc93345ea Win32 import batchsize tweaks 
Reduce frequency of resizes: bump minimum increase from 128MB to 512MB
Use a bigger safety margin at small batch sizes
 2016-02-17 04:05:28 +00:00
Riccardo Spagni
63e2d4195b
Merge pull request #657 
1995923 BlockchainLMDB: Deal with DB exceptions at block level with particularity (warptangent)
c16cc20 BlockchainLMDB: Add sanity check for inconsistent state (warptangent)
9118d0a BlockchainLMDB: Call destructor on allocated txn if setup fails (warptangent)
f5581c3 BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr (warptangent)
 2016-02-13 19:54:47 +02:00
warptangent
1995923559
BlockchainLMDB: Deal with DB exceptions at block level with particularity 
Add another DB error exception type to distinguish failed txn setup from
general use of txn.

This keeps the error handling flow the same as before the block-level
txn setup changes that moved control up a layer to BlockchainDB.
 2016-02-13 05:12:39 -08:00
warptangent
c16cc204d3
BlockchainLMDB: Add sanity check for inconsistent state 
This hasn't been known to occur in block-level txn abort, but throw
exception if it does.
 2016-02-13 04:17:37 -08:00
warptangent
9118d0a44c
BlockchainLMDB: Call destructor on allocated txn if setup fails 2016-02-13 04:17:37 -08:00
warptangent
f5581c3536
BlockchainLMDB: Replace remaining txn pointer NULLs with nullptr 
For consistency.
 2016-02-13 04:17:36 -08:00
moneromooo-monero
8649b9f1ef
blockchain_db: pass hard fork object as a simple pointer 2016-02-08 20:56:31 +00:00
warptangent
e02577f594
Move HardFork DB update to BlockchainDB::add_block() 
Ensures the database is consistent.

Also simplifes blockchain_import in that verify mode off has less to
work around.
 2016-02-08 09:28:19 -08:00
warptangent
3800875406
Make HardFork object available to BlockchainDB and derived DB implementations 
This will later allow the HardFork object's DB update functions to be
called when the DB transaction that persists across block add/remove is
open.
 2016-02-08 09:28:17 -08:00
warptangent
fd46c96dce
BlockchainDB/LMDB: Refactor block-scope DB txn handling for add block 
Move block-scope txn start and stop from BlockchainLMDB to BlockchainDB.
 2016-02-08 09:28:15 -08:00
warptangent
f3a6000094
BlockchainDB/LMDB/BDB: Extract DB txn functions for block add/remove 2016-02-08 09:28:14 -08:00
warptangent
f47d5b0fe3
BlockchainLMDB: Allow two HardFork functions to update DB during block add 
Note that this doesn't yet cause them to be called during block add.
 2016-02-08 09:28:12 -08:00
warptangent
15ee0bef4b
BlockchainLMDB: extract txn macros used during block add/remove 2016-02-08 09:28:10 -08:00
warptangent
f7e337e625
fake_core: Check if hard fork subdbs need reset at start 2016-02-08 08:50:48 -08:00
warptangent
c657e772c4
blockchain_import: Add --drop-hard-fork command 2016-02-08 08:50:47 -08:00
moneromooo-monero
3a0f4d8a57
berkeleydb: fix delete/free mismatch 
Despite being C++, the stats object is allocated by the underlying
C layer using malloc(3).
 2016-02-03 08:49:11 +00:00
warptangent
b8cc6d46bd
BlockchainBDB: Continue when tx has no outputs 
See 24b66243c2
 2016-01-28 21:15:46 -08:00
warptangent
b8707466e2
BlockchainBDB: When removing, find amount output index fast by starting at end 
This improves blockchain reorganization time by allowing one of the more
expensive DB lookups when popping a block to not have to seek through a
long dup list in the "output_amounts" db. This is most noticeable for
HDDs.

See ffcf6bdb95
 2016-01-28 21:15:44 -08:00
warptangent
1a5c3fa729
BlockchainBDB: Remove tx outputs in reverse order 
Data should be removed in the reverse order it was added.

This matches the order of removal in
blockchain_storage::pop_transaction_from_global_index.

See f11def012f
 2016-01-28 21:15:42 -08:00
me0wmix
28f95eb001 OpenBSD support for Monero. 2016-01-21 11:18:26 -07:00
warptangent
24b66243c2
db_lmdb: Continue when tx has no outputs 2016-01-19 16:55:31 -08:00
Howard Chu
30f92f5630 Fix hf when import with verify off 
Delete the hf tables, so the next open will rescan and regenerate
 2016-01-15 17:26:19 +00:00
Riccardo Spagni
de03926850
updated copyright year 2015-12-31 08:39:56 +02:00
Riccardo Spagni
810a11267c
fixed copyrights with bad year references 2015-12-31 08:37:27 +02:00
Riccardo Spagni
fd36eea6dd
Merge pull request #572 
b39aae7 Tweak 45800a25e9 (hyc)
4a5a5ff blockchain: always stop the ioservice before returning (moneromooo-monero)
78b65cf db_lmdb: safety close db at exit (moneromooo-monero)
45800a2 db_lmdb: fix a strdup/delete[] mistmatch (moneromooo-monero)
 2015-12-30 09:38:41 +02:00
hyc
b39aae7aa5 Tweak 45800a25e9 
trivial cleanup
 2015-12-29 00:11:35 +00:00
moneromooo-monero
78b65cf7e8
db_lmdb: safety close db at exit 
Noticed by hyc

Also set m_open to false when closed
 2015-12-28 19:23:02 +00:00
moneromooo-monero
45800a25e9
db_lmdb: fix a strdup/delete[] mistmatch 2015-12-28 19:22:37 +00:00