Logs & errors reference
Although there is sufficient logging in GoQuorum (geth), the error messages are not always straightforward to decipher. This is our take on cataloguing of error messages along with possible cause and remediation actions, to serve as first point of reference before reaching out to the support team.
note
The log level “ERROR” is written out in uppercase as part of the log message and can be used for alert monitoring.
Reference
| Message & Parameters | Cause | Action |
|---|---|---|
"Unable to attach to remote geth: %v", err | This issue occurs when attempting to attach to a local geth node which is not running | Check if the geth process is running, if not then restart it. Note that if the geth process crashed or was force-killed, the ipc file can exist even though the node is not running. |
"unable to connect to private tx manager using %s due to %s", socketPath, err | This issue occurs when geth is unable to connect to the private transaction manager | Check that the transaction manager is running, if not then restart it |
"Could not search for pattern", "pattern", pattern, "contract", contracts[types[i]], "err", err | Generated by abigen due to an internal error when generating ABI for a contract | Raise an issue ticket |
"account listing failed", "error", err | Call to Clef API method account_list failed | Depends on the error detail in the log message |
"operation SelfDerive not supported on external signers" | An unsupported operation was performed by an external wallet | Contact the third party provider of the wallet |
"Failed to enumerate smart card readers", "err", err | A failure occurred when scanning for smart card wallets | The logged message should contain further error details and determine the action to take |
"Failed to enumerate USB devices", "hub", hub.scheme, "vendor", hub.vendorID, "failcount", failcount, "err", err | A failure occurred when scanning for USB wallets. The logged message should contain further error details | Depends on the error detail in the log message, if not using USB wallets then add --nousb to command line |
"Import error", "err", err | Message is generated when using geth import and indicates an issue with the import file | You need to use a valid import file |
"Import error", "file", arg, "err", err | Message is generated when using geth import and indicates an issue with the named import file | You need to use a valid import file |
"Failed to retrieve signer address", "err", err | Message is generated when using puppeth and indicates an issue with signer JSON key file | Ensure you have a valid key file |
"Bootstrap URL invalid", "enode", url, "err", err | One or more enode values specified on the command line for a bootstrap node, is not a valid url. | You need to provide a correct url |
"Invalid smartcard daemon path", "path", path, "type", fi.Mode().String() | This message is generated if the socket file specified for the smartcard daemon (pcscd) is not actually a socket file | Ensure that the daemon is running and the correct socket file is specified on the command line |
"Failed to get signer address", "err", err | This indicates that the public address could not be obtained for the signature on a message in an IBFT network. This is potentially an internal error or an issue with the crypto package. | Depends on the root cause in the log message - an issue ticket may need to be raised |
"Failed to serialize JavaScript exception", "exception", msg, "err", err | This is an internal error, occurring if the given message could not be serialized into a JavaScript message by the Otto JS parser. | Raise an issue ticket |
"Non contiguous block insert", "number", block.Number(), "hash", block.Hash() | Message is generated if an 'out of sequence' block is received for insertion into the chain. This usually occurs if node is out of sync or holds a corrupt chain. | See instructions under the section on Resolution of database corruption issues |
"Non contiguous receipt insert", "number", blockChain[i].Number(), "hash", blockChain[i].Hash(), "parent", blockChain[i].ParentHash() | Message is generated if an 'out of sequence' receipt is received for insertion. This usually occurs if node is out of sync or holds corrupt data. | See instructions under the section on Resolution of database corruption issues |
"Found bad hash, rewinding chain", "number", header.Number, "hash", header.ParentHash | Message is generated on startup if a block is found in the database with one of a set 'bad hash' values predefined in core/blocks.go. The node will rewind the chain to prior to the bad hash and resync from that point | No action should be necessary as node will rewind and recover |
"Chain rewind was successful, resuming normal operation" | This message is generated after the chain has been rewound following a "Found bad hash, rewinding chain" message | No action is necessary |
"Impossible reorg, please file an issue", "oldnum", oldBlock.Number(), "oldhash", oldBlock.Hash(), "newnum", newBlock.Number(), "newhash", newBlock.Hash() | This can occur if there was an issue during sync | See section on Impossible reorg for actions |
"########## BAD BLOCK #########" | This can occur if there was an issue inserting a new block into the chain | See section on Bad block for actions |
"Failed to commit recent state trie", "err", err | This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. | See instructions under the section on Resolution of database corruption issues |
"Dangling trie nodes after full cleanup" | This occurs if the in-memory cache was not fully flushed to the underlying database during shutdown | No action is possible |
"Failed writing private state root", "err", err | This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. | See instructions under the section on Resolution of database corruption issues |
"Non contiguous header insert", "number", chain[i].Number, "hash", chain[i].Hash() | This message is generated if blocks are received with non-contiguous block numbers or block hashes | This is most likely a coding error - raise an issue ticket |
"Pricing query for empty pool" | This message is generated to catch a coding error scenario | Raise an issue ticket |
"Demoting invalidated transaction", "hash", hash | This occurs if a nonce gap appears in the pending queue after housekeeping (which should not happen) | No action should be necessary as the transactions are automatically moved to the non-executable queue |
"Unrooted old chain seen by tx pool", "block", oldHead.Number, "hash", oldHead.Hash() | This can happen on a chain reorg and indicates that the block at the head of the 'old' chain did not have a valid parent hash. This may cause issues with the contents of the transaction pool. | In order to ensure integrity of the transaction pool, it may be necessary to follow the instructions under the section on Resolution of database corruption issues |
"Unrooted new chain seen by tx pool", "block", newHead.Number, "hash", newHead.Hash() | This can happen on a chain reorg and indicates that the block at the head of the 'new' chain did not have a valid parent hash. This may cause issues with the contents of the transaction pool. | In order to ensure integrity of the transaction pool, it may be necessary to follow the instructions under the section on Resolution of database corruption issues |
"Failed to reset txpool state", "err", err | This indicates that the transaction pool state could not be retrieved after a reorg; the err message will provide more detail | This can be down to mis-configuration, or a corrupt database. See instructions under the section on Resolution of database corruption issues |
"Impossible fork ID validation", "id", id | This indicates that an internal error occurred when validating the fork ID for chain compatibility checks as per EIP-2124 | Raise an issue ticket |
"Invalid block body RLP", "hash", hash, "err", err | This occurs if a block retrieved from the database could not be decode. This implies an internal error or an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Invalid block header RLP", "hash", hash, "err", err | This occurs if a block header retrieved from the database could not be decoded. This implies an internal error or an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Invalid receipt array RLP", "hash", hash, "err", err | This occurs if the transaction receipts retrieved from the database for a block, could not be decoded. This implies an internal error or an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Missing body but have receipt", "hash", hash, "number", number | This occurs if the block body relating to a transaction receipt could not be retrieved from the database. This implies there is an issue with the database or an internal error occurred. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Failed to derive block receipts fields", "hash", hash, "number", number, "err", err | This occurs if the transaction information relating to a receipt could not be retrieved from the database. This implies there is an issue with the database or an internal error occurred. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Invalid block total difficulty RLP", "hash", hash, "err", err | This occurs if the block difficulty was retrieved from the database but could not be decoded. This implies an internal error or an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Receipt not found", "number", blockNumber, "hash", blockHash, "txhash", hash | This occurs if a transaction receipt could not be retrieved from the database for the block which contains it. This implies there is an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Transaction referenced missing", "number", blockNumber, "hash", blockHash | This occurs if a transaction could not be retrieved from the database for the block which contains it. This implies there is an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Transaction not found", "number", blockNumber, "hash", blockHash, "txhash", hash | This occurs if transaction data could not be retrieved from the database for a known transaction. This implies there is an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Invalid transaction lookup entry RLP", "hash", hash, "blob", data, "err", err | This occurs if a transaction retrieved from the database could not be decoded. This implies an internal error or an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Invalid chain config JSON", "hash", hash, "err", err | This occurs if the chain configuration JSON read from the database could not be un-marshalled. This implies an internal error or an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Database contains unaccounted data", "size", unaccounted | This occurs if unrecognised/unaccounted data is found in the database. This implies there is an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Failed to append ancient hash", "number", f.frozen, "hash", hash, "err", err | This occurs if 'ancient' data could not be appended to the tables in the freezer database. This implies there is an issue with the database. | Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node |
"Failed to append ancient header", "number", f.frozen, "hash", hash, "err", err | This occurs if 'ancient' data could not be appended to the tables in the freezer database. This implies there is an issue with the database. | Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node |
"Failed to append ancient body", "number", f.frozen, "hash", hash, "err", err | This occurs if 'ancient' data could not be appended to the tables in the freezer database. This implies there is an issue with the database. | Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node |
"Failed to append ancient receipts", "number", f.frozen, "hash", hash, "err", err | This occurs if 'ancient' data could not be appended to the tables in the freezer database. This implies there is an issue with the database. | Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node |
"Failed to append ancient difficulty", "number", f.frozen, "hash", hash, "err", err | This occurs if 'ancient' data could not be appended to the tables in the freezer database. This implies there is an issue with the database. | Raise an issue ticket or try to resolve by removing the freezer database and performing a resync of the node |
"Current full block number unavailable", "hash", hash | This occurs if the block number for the current head block could not be read from the database when checking whether any blocks can be moved to the freezer. This implies there is an issue with the chain database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Current full block unavailable", "number", *number, "hash", hash | This occurs if the data for the current head block could not be read from the database when checking whether any blocks can be moved to the freezer. This implies there is an issue with the chain database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"Canonical hash missing, can't freeze", "number", f.frozen | This occurs if the hash for a block could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. | Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node |
"Block header missing, can't freeze", "number", f.frozen, "hash", hash | This occurs if a block header could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. | Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node |
"Block body missing, can't freeze", "number", f.frozen, "hash", hash | This occurs if a block could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. | Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node |
"Block receipts missing, can't freeze", "number", f.frozen, "hash", hash | This occurs if the receipts for a block could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. | Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node |
"Total difficulty missing, can't freeze", "number", f.frozen, "hash", hash | This occurs if the block difficuly could not be read from either the chain database or the freezer and may cause archiving to halt. This could be an internal error or an issue with either database. | Raise an issue ticket or try to resolve by removing chain database and freezer, and resync of the node |
"Failed to decode the value returned by iterator", "error", err | This is most likely due to an internal error. | Raise an issue ticket |
"Failed to decode state object", "addr", addr, "err", err | This occurs if state data retrieved from the database could not be decoded. This implies an internal error or an issue with the database. | Raise an issue ticket or follow instructions under the section on Resolution of database corruption issues |
"EIP activation failed", "eip", eip, "error", err | This is most likely due to an internal error. | Raise an issue ticket |
"Unknown metered peer event type", "type", event.Type | This occurs if an unknown event message type was received from a peer. This implies an internal error or some incompatibility between nodes. | Raise an issue ticket |
"Cannot set etherbase in Istanbul consensus" | This occurs if an attempt is made to perform the miner_setEtherbase() API call - this is not allowed if Istanbul consensus is in use | Action is not possible |
"Cannot start mining without etherbase", "err", err | This occurs if the etherbase account is not set. Note that for some consensus mechanisms, this is automatically set to the signing account. | Ensure node has an etherbase account set up |
"Etherbase account unavailable locally", "err", err | This occurs if the clique consensus mechanism is used and the configured signing account (etherbase) could not be found in the wallet. | Check the configuration - in particular that the signing account defined in the genesis extradata matches the value in the keystore and is set up as a local account |
"Propagating dangling block", "number", block.Number(), "hash", hash | This occurs when propagating a block to peers, if the block does not have a parent. | Raise an issue ticket (this scenario should never occur in normal operation) |
"Failed to encode receipt", "err", err | The list of receipts for a block could not be RLP encoded. | The error detail should give more information and will determine the action to be taken |
"Peer removal failed", "peer", id, "err", err | A request to remove a peer failed. | The error detail should give more information and will determine the action to be taken, but it is most likely caused by attempting to remove a peer that was already removed |
"Unknown downloader chain/mode combo", "light", d.lightchain != nil, "full", d.blockchain != nil, "mode", d.mode | This occurs if the node was unable to report sync progress due to an invalid combination of downloader type and sync mode. | Raise an issue ticket (this scenario should never occur in normal operation) |
"Failure in running pprof server", "err", err | This occurs if the pprof HTTP server could not be started. | The error detail should give more information and will determine the action to be taken |
"Database compaction failed", "err", err | A database error occurred during DB compaction. | The error detail should give more information and will determine the action to be taken. It may be necessary to follow instructions under the section on Resolution of database corruption issues |
"Oracle contract binding failed", "err", err | This occurs when using a checkpoint oracle contract and means that the node could not bind to the oracle contract. | The error detail should give more information and will determine the action to be taken |
"Already bound and listening to registrar" | This occurs when using a checkpoint oracle contract and means that the there is already a binding in place for the oracle contract. | No action is necessary |
"Refusing to mine without etherbase" | This indicates that the node is set up to mint blocks, but the coinbase account is not set. | Check the configuration is correct for the consensus mechanism in use |
"Failed to prepare header for mining", "err", err | This occurs when a node is attempting to create a block, but fails during the preparation stage when creating the header. | The error detail should give more information and will determine the action to be taken |
"Failed to create mining context", "err", err | This means an error has occurred when setting up the state after creation of a new block. | Raise an issue ticket |
"Failed to fetch pending transactions", "err", err | This error should never occur. | Raise an issue ticket |
"Block found but no relative pending task", "number", block.Number(), "sealhash", sealhash, "hash", hash | This indicates that an internal error has occurred. | Raise an issue ticket |
"Failed writing block to chain", "err", err | This indicates that a failure occurred when committing a block and associated state to the database. | The error detail should give more information and will determine the action to be taken |
"Failed writing private block bloom", "err", err | This indicates that a failure occurred when writing a bloom filter to the database for private transaction receipts. | The error detail should give more information and will determine the action to be taken |
"Failed to persist node key: %v", err | This occurs if the node is unable to save a newly generated node key into the file specified by the configuration. | The error detail should give more information and will determine the action to be taken |
"Can't load node list file: %v", err | This occurs if the node is unable to read static node information from the static-nodes.json (or trusted-nodes.json) file. | Check that the file exists in the expected location and that it contains valid JSON |
"Node URL %s: %v", url, err | This indicates that there is a mis-configured URL in the static-nodes.json (or trusted-nodes.json) file. | Check and correct the offending URL in the JSON file |
"Read Error for permissioned-nodes.json file. This is because 'permissioned' flag is specified but no permissioned-nodes.json file is present.", "err", err | This is self-explanatory | Either remove the --permissioned flag or add permissioned-nodes.json file |
"parsePermissionedNodes: Failed to access nodes", "err", err | This indicates that the file permissioned-nodes.json could not be read. | The error detail should give more information and will determine the action to be taken |
"parsePermissionedNodes: Failed to load nodes", "err", err | This indicates that the node information in permissioned-nodes.json could not be unmarshalled. | Check that the contents of the file are well-formed |
"parsePermissionedNodes: Node URL blank" | This indicates that an entry in permissioned-nodes.json has an empty node URL. | Either remove the node details or specify a valid URL |
"parsePermissionedNodes: Node URL", "url", url, "err", err | This indicates that an entry in permissioned-nodes.json is not valid. | Check that the entry with this URL is correctly formatted |
"Failed to expire nodedb items: %v", err | This message should never be seen | Raise an issue ticket |
"error encoding packet:", err | A message could not be RLP encoded prior to sending to a peer node. | The error detail should give more information and will determine the action to be taken |
"could not sign packet:", err | A message could not be signed prior to sending to a peer node. | The error detail should give more information and will determine the action to be taken |
"Failed to execute permission action", "action", action, "err", err | This indicates a problem when invoking the specified action on the permissions contract. | The error detail should give more information and will determine the action to be taken |
"bootupNetwork SetPolicy failed", "err", err | This indicates an issue occurred when invoking setPolicy() on the permissions contract. | The error detail should give more information and will determine the action to be taken, but it is most likely due to misconfiguration of the permissions contract(s) |
"bootupNetwork init failed", "err", err | This indicates an issue occurred when invoking init() on the permissions contract. | The error detail should give more information and will determine the action to be taken, but it is most likely due to misconfiguration of the permissions contract(s) |
"failed to updated network boot status", "error", err | This indicates an issue occurred when invoking updateNetworkBootStatus() on the permissions contract. | The error detail should give more information and will determine the action to be taken, but it is most likely due to misconfiguration of the permissions contract(s) |
"failed to get raft id", "err", err, "enodeId", enodeId | This occurs with Raft consensus, if the permission contract generates a NodeDeactivated event, but a Raft id could not be found for the enodeId that is being removed. | Check whether the node has already been removed, if not then this may be an internal error and a ticket should be raised |
"failed parse node id", "err", err, "enodeId", enodeId | This occurs with clique/istanbul consensus, if the permission contract generates a NodeDeactivated event, but the enodeId could not be correctly parsed for the node that is being removed. | Raise an issue ticket |
"Revoke role - cache is missing role", "org", evtRoleRevoked.OrgId, "role", evtRoleRevoked.RoleId | This occurs if the permission contract generates a RoleRevoked event, but the specified role could not be found in the cache. | Raise an issue ticket |
"can't open file", "file", fullPath, "error", err | This indicates that the specified file could not be opened. | Check that the file exists at the given path and has read permission |
"error reading file", "err", err, "file", fullPath | This indicates that the specified file could not be read. | The error detail should give more information and will determine the action to be taken |
"error unmarshalling the file", "err", err, "file", fullPath | This indicates that the contents of the specified file could not be unmarshalled. | Check that the file contains valid JSON |
"Read Error for disallowed-nodes.json file", "err", err | This indicates that the disallowed-nodes.json file could not be read. | This message is normally logged the first time a node is being added to the file, in which case no action is needed |
"Failed to create disallowed-nodes.json file", "err", err | This indicates that the disallowed-nodes.json file could not be created. | Check that the data directory location for this file is writeable |
"Failed to access the file", "fileName", fileName, "err", err | This indicates that the specified file could not be read. | Check that the file exists at the given path and has read permission |
"Failed to load nodes list from file", "fileName", fileName, "err", err | This indicates that the contents of the specified file could not be unmarshalled. | Check that the file contains valid JSON |
"Error writing new node info to file", "fileName", fileName, "err", err | This indicates that the specified file could not be created. | Check that the location for this file is writeable |
"unable to delegate RPC API calls to plugin", "provider", interfaceName, "error", err | This message is generated by the plugin service if it fails to delegate an RPC call to the specified provider. | The error detail should give more information and will determine the action to be taken, however it is most likely due to misconfiguration |
"error decoding pub key from enodeId", "enodeId", address.NodeId.String(), "err", err | This occurs when adding a node for Raft consensus, and the enodId could not be decoded into a public key. | Ensure the correct enodeId is specified |
"failed to extend chain: %s", err.Error() | This occurs if the node was unable to add a new block to the chain when using Raft consensus. | The error detail should give more information and will determine the action to be taken. Note that it may occur during node shutdown with error detail abort during blocks processing, in which case it can be ignored. |
"error decoding block", "err", err | This can occur on a Raft node when restoring a snapshot if a block could not be decoded. | Most likely the Raft snapshot is corrupt. Raft may recover, or a restart may be required; if not, the data directory to be deleted and the node resynced. |
"error inserting the block into the chain", "number", block.NumberU64(), "hash", block.Hash(), "err", err | This can occur on a Raft node when restoring a snapshot if a block could not be added to the chain. The node may be out of sync or hold a corrupt chain. | The error detail should give more information. Raft may recover, or a restart may be required; if not, the data directory to be deleted and the node resynced. |
"RPC method " + method + " crashed: " + err) | This indicates that an issue occurred when responding to an RPC call. | The error detail should give more information, however check that the arguments for the RPC call are correct |
"Invalid smartcard socket file type", "path", scpath, "type", fi.Mode().String() | This message is generated by clef if the socket file specified for the smartcard daemon (pcscd) is not actually a socket file. | Ensure that the daemon is running and the correct socket file is specified on the clef command line |
"Failed to commit preimage from trie database", "err", err | This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. | See instructions under the section on Resolution of database corruption issues |
"Failed to write flush list to disk", "err", err | This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. | See instructions under the section on Resolution of database corruption issues |
"Failed to commit trie from trie database", "err", err | This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. | See instructions under the section on Resolution of database corruption issues |
"Failed to write trie to disk", "err", err | This occurs if an error occurred when writing to the underlying database. Most likely cause is lack of resources, or corrupt database. | See instructions under the section on Resolution of database corruption issues |
"Attempted to dereference the trie cache meta root" | This occurs if an error occurred during garbage collection. | The node may need a restart |
"Unhandled trie error: %v", err | This indicates that an error occurred when retrieving data from a trie. | The error detail should give more information; it may be necessary to follow the instructions under the section on Resolution of database corruption issues |
Fatal: Error starting protocol stack: can't download from Plugin Central due to: HTTP GET error: code=404, status=404 Not Found, body=The requested path was not found.. Please download the plugin manually and copy it to <dir> | The provided plugin config does not match any available plugins in the Central server | Check name and version fields in plugin definition config are correct |
Fatal: Error starting protocol stack: stat <dir>/Central.pgp.pk: no such file or directory | A default public key cannot be found to verify the integrity of plugins | Create the necessary key at this path or use the --plugins.publickey flag to use an alternative path |
"plugins: unable to create reader due to %s", err | An issue occurred when reading the plugin config file specified with --plugin. | Ensure the correct config file name is specified on the command line and contents are correct. The logged message will contain further details which may help. |
"plugins: unable to resolve plugin base dir due to %s", err | Incorrect value of baseDir in the plugin config file specified with --plugin. | Ensure the baseDir value is correct in the config file. The logged message will contain further details which may help. |
"Plugin failed to start", "error", err, "took", time.Since(startTime) | GoQuorum failed to start a configured plugin. | Ensure that all vaues in the config file are correct and that any required separate plugin processes are running. The logged message will contain further details which may help. |
"Error parsing version components from the tessera version: %s. Unable to extract transaction manager features.", version | GoQuorum is configured to use Tessera as the transaction manager, but failed to parse the API version number returned by the running Tessera process. | This is most likely due to using an older version of Tessera, prior to v1.0. You should upgrade your Tessera (note that geth will continue to run, but some privacy features may not be available). |
"Error invoking the tessera /version/api API: %v.", err | GoQuorum is configured to use Tessera as the transaction manager, but failed to retrieve the API version number from the running Tessera process. | This is most likely due to using an older version of Tessera, prior to v1.0. You should upgrade your Tessera (note that geth will continue to run, but some privacy features may not be available). |
"Invalid status code returned by the tessera /version/api API: %d.", res.StatusCode | GoQuorum is configured to use Tessera as the transaction manager, but failed to retrieve the API version number from the running Tessera process. | This is most likely due to using an older version of Tessera, prior to v1.0. You should upgrade your Tessera (note that geth will continue to run, but some privacy features may not be available). |
"Unable to deserialize the tessera response for /version/api API: %v.", err | GoQuorum is configured to use Tessera as the transaction manager, but failed to parse the API version number returned by the running Tessera process. | This is most likely due to using an older version of Tessera, prior to v1.0. You should upgrade your Tessera (note that geth will continue to run, but some privacy features may not be available). |
More details
Impossible reorganization (reorg) issue
Impossible reorganisation issues can occur if there was an issue during sync.
Possible causes are:
- Node was not cleanly shutdown previously, causing database corruption
- An edge case where two sealers generate blocks at the exact same time (depends on consensus type in use)
- Insufficient resources (disk or memory)
- Some (unknown) issue during resync
The action to be taken will depend on the cause. Note that more recent versions of GoQuorum may already include fixes to prevent some of these issues.
In the event of database corruption, see instructions below for Resolution of database corruption issues.
Bad block issue
This can occur if there was an issue when inserting a new block into the chain and is logged in the form:
########## BAD BLOCK #########
Chain config: %v
Number: %v
Hash: 0x%x
'transaction receipt'
Error: %v
##############################
The error detail gives more information as to the root cause. Here are a few possibilities and suggested actions:
"invalid gas used": This occurs if the local node calculates a different gas usage for transactions than the node which created the block. This may be due to a configuration issue."insufficient balance to pay for gas": This is similar to"invalid gas used", but occurs when there is insufficient gas to complete a transaction."invalid merkle root": Cause is usually a corrupted database or failed import. See instructions under the section on Resolution of database corruption issues."unknown ancestor": This is usually due to an issue with fast sync. Restarting the node may resolve it."blacklisted hash": The cause and resolution of this is like the"Found bad hash, rewinding chain"error.
Note that more recent versions of GoQuorum may already include fixes to prevent some of these issues.
Resolution of database corruption issues
Corruption of the database (chain data) can occur due to lack of resources, or after a 'forced' shutdown. Sometimes, GoQuorum will recover automatically or require a restart of the failing node.
If the node does not recover then it may require the chaindata to be deleted and resynced.
The chain data can be removed by shutting down the node and running geth removedb --datadir /path/to/data/directory.
If Raft consensus is in use, then the Raft logs must also be removed before restarting the node, these consist of all directories containing raft under the data directory.