The Quest for Lightning Block Propagation

TurtleCoin has 30 second block time which makes transactions on the network faster than other projects; however, it poses some challenges in making sure that new blocks propagate across the entire network as quickly as possible to prevent the creation of unnecessary network forks. While increasing the number of nodes on the network helps to achieve this goal, reducing the size of the blocks that are transmitted between nodes allows us to propagate a block so fast it’s almost as if they were transmitted by lightning.

Network Propagation

Understanding what information is propagated throughout the network is a requirement for making sense of what we’re trying to accomplish when we talk about speeding up propagation.

Each node that is connected to the network performs vital functions such as, but not limited to:

  • Verifies transactions (signatures, no double spending, structures, etc.)
  • Verifies blocks (makes sure that they meet the difficulty requirements, are valid in the chain, contain valid transactions, etc.)
  • Maintains a transaction pool (transactions that have been submitted to the network but not included in a block)
  • Relays transactions & blocks to peer nodes (those the node is connected to)

Each time that a new transaction, in its entirety, is submitted to a node for processing by a user, that transaction is added to the node’s transaction pool and then relayed to all its connected peers who relay it to their peers until everyone on the network has those transactions.

Mining a Block

Once a transaction is in the transaction pool, it’s hash is then included in the block template that is passed to miners via pools or directly from the node if solo mining.

When a miner (or pool) finds a block the block, with all its glory, it is submitted to the node, it is validated, added to the chain, and then the full block containing every transaction including its inputs, outputs, signatures, etc. is relayed to all the node’s peers. Those peers relay it to their peers after verifying it and adding it to their copy of the blockchain, and so on until everyone has the new block.

Block Propagation Payloads

The full blocks that are sent between nodes have the sizes as reported on the block explorers. Some blocks are small at only a few hundred bytes while others with many transactions or transactions with lots of inputs/outputs can climb to well over 100KB. 100KB does not seem like a lot of data, but when you need 100KB to propagate between >1,000 nodes, it takes considerable time. We need the elapsed time from block creation to full network propagation to be as little as possible to make sure that the nodes that are mined against always have the latest information about the chain.

But… Why?

You may be asking yourself, “If transactions are propagated around the network for the transaction pool and a node already has them, why are we sending them again, in their entirety, with every block that is propagated?”

To put it bluntly, we don’t need to send everything all the time if a node already has most of the information needed to verify a new block. Sending all of it again is just duplicating things it already has.

Lite Blocks to the Rescue

Enter the concept of Lite Blocks. Lite Blocks are a special kind of block created just for synchronization between nodes. A Lite Block is just like a full block except it only contains the transaction hashes not the full contents of the transactions (inputs, outputs, signatures, etc.).

When a node receives a Lite Block, it looks through the list of transaction hashes the block contains, checks its local transaction pool, and plucks the needed transactions from the pool. If it encounters a transaction that is not in its transaction pool (for any reason), it reaches out to its peers and requests just that transaction. Once the node has all the transactions the block contains, the block is verified, and added to the node’s local copy of the blockchain.

The node then starts the relaying process to propagate the new block to its peers. In going through the list of connected peers, it identifies which of its peers support Lite Blocks and which do not support Lite Blocks. It relays the appropriate block type (Full or Lite) to its peers as needed.

If you’re keen for additional reading on the underlying concept of Lite Blocks check out BTC’s BIP-0152 as well as Monero’s Fluffy Blocks.

TurtleCoin’s Implementation

Adding Lite Block functionality results in significantly smaller blocks propagating across the network. In testing, we’ve observed Lite Blocks as little as 1% of the full block size. Transferring 1KB versus 100KB is a momentous reduction in the amount of data transferred between nodes. This reduction helps to improve block propagation times in a massive way. TurtleCoin nodes will now prefer to send Lite Blocks whenever possible to reduce the size of the information exchanged between nodes. The node will fall back to Full Blocks only when necessary.

@rashedmyt has completed the work required to add Lite Blocks into the core TurtleCoin code. He’s built it in such a way that it does not require a fork of any kind as it relies on the P2P version that we use in a couple of different places. Our plans are to activate Lite Block capability with an upgrade to P2P Version 4. This P2P update will make it into the next release of the core TurtleCoin software.

The addition of Lite Blocks was listed as a TurtleCoin bounty and @rashedmyt will be picking up a 2.05M TRTL bounty for the efforts he’s put forth in getting Lite Blocks implemented in the core code. When you see him in Discord give him a high-five and a round of applause. Continue Reading →

Proof-of-Work Algorithm Change

The core development team has observed the fact that the network hash rate has climbed substantially over the last few weeks. Luckily, the hashrate has remained rather decentralized and balanced over the large number of pools that the community operates. However, the significant increase in hashrate has made it more difficult for the project to meet one of its core goals. To remain mineable for everyone. After careful consideration, we believe it’s once again time to perform a soft fork to a new Proof-of-Work (PoW) algorithm.

Such a change is not taken lightly, and we’ve given considerable thought to where the project should go in this regard. We’ve seen a few networks using CryptoNight Soft Shell variants with success. Although considerable work has been done to support Soft Shell pools and the like we’re not quite ready to move over to Soft Shell just yet.

You’re probably asking yourself, “So what’s the plan?”

Supported PoW Algorithms

If you’ve read through the TurtleCoin fork guide, forked TurtleCoin yourself, or have read through the code, the next few statements are going to come to no surprise to you.

We like to collect PoW algorithms like they are pogs. The more you have, the more fun you’re having with your friends.

Keeping this in mind, TurtleCoin currently has built-in support for no less than twelve (12) different PoW algorithms. Yes, you read that right, twelve.

They are, in no particular order:

  • CryptoNight v0
  • CryptoNight v1
  • CryptoNight v2
  • CryptoNight Lite v0
  • CryptoNight Lite v1
  • CryptoNight Lite v2
  • CryptoNight Turtle v0
  • CryptoNight Turtle v1
  • CryptoNight Turtle v2
  • CryptoNight Soft Shell v0
  • CryptoNight Soft Shell v1
  • CryptoNight Soft Shell v2

Woah, Wait, What’s CryptoNight Turtle?

To put it mildly, CryptoNight Turtle is a tweak to the standard CryptoNight family that significantly increases the speed at which hashes can be computed. This is accomplished in two ways.

Iteration Change

We’re dropping the iteration count in fourth as well. From 524,288 iterations to 131,072 iterations. This will result in the base hashrate increasing for every miner.

Scratch Pad Change

We’re dropping the scratch pad size in fouth. From 1MB (CN Lite) to 256KB. Why? because we want to make everything go faster and when we’re dropping the iteration count, we need to make sure that the scratchpad is used effectively to prevent some interesting attack vectors.

Faster? Doesn’t That Mean an Even Higher Network Hash Rate?

It sure does! There’s a reason for what appears to be chaos and we’re just getting to the good part.

TurtleCoin Major Block v5

As you’re likely thinking, “This doesn’t sound better, no, no, these guys are crazy…”

Block Major version 5 is planned to activate CryptoNight Turtle v2. That’s right, variant 2. Those of you following other projects using v2 likely know that there is a performance penalty incurred by variant 2 hashing. We’ve personally seen a performance hit of anywhere from 12% to 45% on some hardware. We think that’s unacceptable; however, we do like the premise behind variant 2.

To balance the performance impact of variant 2 out, we’re going to hash faster with CryptoNight Turtle. The two effectively cancel each other out and we gain the benefit of soft forking away from potential ASIC/FPGA/etc. miners. As a matter of fact, you’ll see a hashrate increase on your miner(s) because of the move to CryptoNight Turtle.

The move to variant 2 will help strengthen TurtleCoin’s ASIC/FPGA resistance. Future PoW changes will also be designed to further strengthen the project’s ASIC/FPGA resistance to try to keep TurtleCoin mineable for everyone.

Doing Your Part

We’re confident that we have the core code ready for the soft fork to CN Turtle tentatively scheduled for block 1,200,000. The main pool software has been updated to support it as well as the underlying Node.js Hashing Library and the Node.js Cryptonote Library have been updated.

There’s still work to be done and any community assistance is, as always, appreciated.

Test Network

We’ll need help from the community to test this algorithm change to try to ensure that everything goes as smoothly as possible. The more people that are involved, the easier it is to spot issues.

Miner Support

We’re working on adding support to common miner projects to support this change including:

Miner software needs support added before we can fully test the update pool deployments. If you want to give us a hand in getting that support completed, join us on discord at http://chat.turtlecoin.lol in #dev_general.

Alternative Pools

The pool software changes need ported to the other pools that the community service operators use on a daily basis. If you are familiar with one of the following pools, your assistance is appreciated in this regard.

Upgrades

As always, be mindful of TurtleCoin core releases. Watch or star the main TurtleCoin repo to help stay abreast of changes and updates. Join Discord and read the #announcements on a regular basis. Or, sign up for the @news role by typing *news in the chat and be alerted whenever a new announcement is posted in Discord.

Make sure you’re ready for the soft fork as early as possible.

As with any soft fork, if and when the fork occurs, prior versions of the software will no longer be compatible with the rest of the network after soft fork completion.

What’s Next?

We’re already putting thought into the next PoW algorithm change after v5. Those discussions will be a different article, so as always, stay tuned.

Continue Reading →

That’s It, Old Nodes Have Got To Go!

Ever wonder when your node is syncing why it jumps back and forth between 2 days and 2 months behind? Yeah, we’ve had enough of that too.

It’s Time We Clean Up the Town!

We take network upgrades very seriously here at TurtleCoin. We want the network to be as safe, easy, and secure as possible for our Turtle brethren. We do our best to test all changes and make sure that our fellow Turtle are given plenty of time to upgrade their software before a change takes effect. Unfortunately… not everyone upgrades when necessary.

Running Legacy Software

We attempt to provide everyone ample notice before upgrades occur, what the upgrades will do, and how they benefit the network. However, not every Turtle visits the TurtleCoin discord, checks the GitHub repo, or watches the various forms of social media we have.

There are uncountable nodes running around on the network playing by antiquated network rules disrupting the network and causing utter chaos at times. When Turtles run old software it causes a variety of issues, including but not limited to the following:

  • Slower syncronization as the out of date nodes attempt to push blocks that don’t match the current network rules
    • Your node has to waste time checking their block and finding out it doesn’t fit the mold before the connection is dropped. Then your node has to find a new peer to hopefully get the correct block(s) from

    The daemon must store old alternative chains in memory in the event that they one day resolve (highly unlikely)

    • This results in the blockchain wasting space on your system

    Your log is filled with messages about unexpected hashes, failed hashes, etc.
    Many other issues we don’t know about yet

    The Solution

    TurtleCoin v0.8.3 implements a new version checking mechanism that starts the process of checking the version of peer software when trying to exchange blockchain data. When we connect to a peer, or a peer connects to us, the version of the peer is compared against a minimum version that is supported. If the peer is not at or above that minimum version, the connection is dropped. This helps us to ensure seamless network upgrades in the future that aren’t cluttered by those that do not keep their software updated.

    Note: Version 0.8.3 does not perform this purge of old nodes, it merely sets the stage for the event in future release(s).

    We’ve also added additional log messages that warn you when you may be out of date (as you’re seeing nodes with higher versions than the one you’re running). If you see such a message, as always head over to http://latest.turtlecoin.lol to make sure you have the latest version of the software.

    Future Purges

    The development team aims to utilize this feature for soft forks that are the result of a mining algorithm change, difficulty algorithm change, or any other event that requires that the project to implement a soft fork.

    Such a strategy helps us maintain the security of the network by alterting our fellow Turtles of upgrades and keeping legacy nodes off the network.

    Thanks, No, Seriously, Thanks

    We appreciate everyone that runs a node on the network, public, private, or some mix in between. It helps the stability of the network and shows everyone that we are here to stay. However, if you are one of the people running an ancient copy of TurtleCoin, please upgrade to v0.8.3 and all future releases within the posted ugprade window in the version release notes. You’ll help the community out considerably.

    Otherwise,

    Continue Reading →

Running A Public Node For Fun & Profit: Part Deux

Hopefully you’ve read Part 1 of this series of articles and already have your public node up and running. If not, we highly suggest you head over there and give it a read before continuing with this article.

Note: To make sure that you’ve read this document before proceeding, you won’t be able to copy and paste any of the commands.

What Next?

Running a public node that the community can use can be challenging at times. 100% uptime is great but that’s not always possible in some environments. Running a public node takes work. Despite our best efforts right now, we can’t just Set it and Forget it.

We can; however, use some cool tools developed by the community to help make it easier to maintain a public node. We’ll take a look at how to do this below.

TurtleCoind High Availability Daemon Wrapper

This project is designed to wrap the TurtleCoind daemon on a system and monitor it for hangups, locks, fork, or other events that cause the daemon to stop responding to requests in an accurate manner.

The wrapper does its best to keep TurtleCoind running in tip top shape and always responding to requests from your users. Follow the directions below to set it up on your system. We’ve included instructions below designed to work on a your favorite flavour of *nix variants.

Node.js Setup

You’re going to need a newer version of Node to get things started. Use of any of the older v0.x releases hasn’t been supported for a while so don’t even bother trying to use those. If you are not sure what version of Node.js is installed on your system (if at all), simple run the below command:

Hopefully you’ll see something relatively modern come back like version 8:

If not, head on over to Installing Node.js via package manager for instructions on how to install the latest version for your distro.

Once you’ve completed that step and have verified you are running an up-to-date copy, we’ll continue on below.

Setting up PM2

Setting it up from here is pretty easy going forward. We’ll start by installing PM2: Advanced, production process manager for Node.js to help keep our node script running 24/7/365 and persist across system reboots.

To do so, we’ll run the following commands:

If everything completes successfully, we’ll have the PM2 process up and running and ready to run our script.

Setting up the Wrapper

The turtlecoind-ha package is registered with NPM, so installation is pretty easy. We’ll create a new directory for our process to live in and install it there.

After a few moments, the package will download and install with all the needed dependencies. As long as the installation finishes with the line(s) like those below you are good to proceed.

Next, we’re going to create a new file in our folder named turtlenode.js and dump the code below into that file. We’ve linked to the raw code here.

Most of the code above is designed for spitting us understand what’s going on with our node at any given time. For education purposes, we’re going to leave that intact.

We want to pay attention to the section above that initializes the node parameters.

You’ll need to change those values to match the TurtleCoin address you’re using to collect your node convenience charges as well as the amount of that fee. Remember, as stated in Part 1 of this series, the fee amount is expressed in shells or “atomic units” not TRTL.

Once we’ve made the necessary changes, we’ll save that file and we’re almost ready to rock and roll.

Prepping the Node

For maximum ease of setup you’ll need to copy the latest version of TurtleCoind for your system into the directory we created above. You can compile your own, or download the latest release binaries from the TurtleCoin GitHub repository.

We also advise that you download the latest checkpoints from the Offical Repo to help your node sync up as quickly as possible. The following command will make this quick and easy for you.

Testing the Node

Now that we have all of that set up and ready to rock and roll, we’ll test our node to make sure everything is set up correctly.

Run the following command to find out if everything will start up correctly.

If all goes well, you’ll see output like this:

Depending on if you’ve ever synced a daemon on this machine – it could take a while. Don’t fret if that’s the case. We can continue with even an unsyncronized daemon.

As long as we see TurtleCoind has started followed by TurtleCoind is attempting to synchronize with the network… we know that we’ve set up our environment correctly. Press CTRL-C to stop the node from running. We’re ready to hand off control to PM2.

Launching the Node

Handing off control to PM2 is pretty simple. To do so enter the following command(s):

Checking the Node

To verify that the node is running, we can issue the following command:

PM2 will spit back a list of processes maintained by the software such as:

To view our node logs, issue the command:

Once your node is synced, you’ll start to see log entries such as:

Now sit back, relax, and let people know about your public node.

Continue Reading →