How to submarine NFTs using the IPFS

9 min readDec 27, 2021

In this article I explain the data structures for non-fungible tokens (NFTs), examine the current metadata structure for ERC721 standard for NFTS on Ethereum, look at the InterPlanetary File System, and then present a method that allows you to submarine NFTs, which means “hiding the files and then revealing them as each NFT is sold”.

Introduction

When a collection of NFTs is released, there is often a scramble to obtain the rarest and most unusual tokens in the collection. This can result in “sniping” (swooping in at the last moment and making a slightly higher offer to buy the token) or people submitting transactions with extremely high gas in order to ensure their purchase request goes through before anyone else’s.

In the world of collectible trading cards, the problem is solved by putting cards in a foil pack, so that the purchaser doesn’t know whether what they have bought until they get home and open the pack. In the world of art NFTs, those with more coding skills have the equivalent of “x-ray vision”, namely being able to look into the packs and pick the best ones off the shelf before anyone else.

If you are releasing an NFT collection where the metadata files and the image files are hosted on your own server, then the way around this problem is to put the relevant files for each token up on your server after the token is sold. This can be done as a “grand reveal”, or gradually, token by token.

But if you are releasing NFTs with the metadata and image files on the InterPlanetary File System (IPFS), it is not so easy. Once a file or folder is made public on the IPFS, if other nodes copy and store the file, you can no longer remove it. And although you can upload individual files one by one, the NFT contract requires them to be submitted in an IPFS folder, which has to be uploaded in one go, meaning you can’t reveal the images one at a time.

The solution is something known as “submarining”[1]. With a paid subscription to an IPFS hosting service such as Pinata, you can privately upload files and folders, and then make them public later.

But who wants to pay for something if you can do it yourself for free? Not me, that’s for sure.

NFT Metadata

The most common standard for NFTs is the ERC721 standard[2], which defines how functions such as “safeTransfer” and “name” should work, and so this is the standard we will examine.

An NFT is nothing more than an entry in a ledger that is created on a blockchain using a smart contract. The ledger entries show who owns each NFT (by linking the NFT identifier with an Ethereum address), and typically provides a URI (that’s a link) to a metadata file for each token.

The metadata file then contains data such as the title of the NFT, a description, and if it’s an art NFT, yet another link, this time to the actual image that the NFT is supposed to represent.

There is no formal standard for the NFT metadata, but the fact that NFT trading sites such as OpenSea expect certain data to be present, means that they are imposing a de-facto standard for us all[3].

Similarly, OpenZeppelin has provided template ERC721 contracts[4] that have become extremely popular, and their template imposes another de-facto standard. And these days, the OpenZeppelin method used is to have a “base URI”, and then tack a number on the end of that base URI in order to get the actual link for the metadata.

So if you were hosting your metadata on your own site, then the base URI might be something like https://mysite.org/nfts/and the first token metadata file would therefore be at https://mysite.org/nfts/1 and so on.

Retrieving the metadata file for the first token, we might see something like this:

{
    "description": "This is the first NFT in the contract",
    "image": "https://mysite.org/nft-images/picture-001.png",
    "name": "First NFT"
}

And then to get the image file, we need to go to the location specified by the “image” key.

So there you have it — the actual artwork is connected to the NFT through a link to a link, or in other words, a pointer to a pointer to the location of the image file.

Moving to the IPFS

The IPFS is … different to the web pages and web links we are interested in. The traditional method for locating a file or a picture uses the hyper-text transfer protocol, and specifies “where” on the network the file we want is located.

The IPFS, on the other hand, uses something called “content addressing”. You can imagine the traditional web and the IPFS as public libraries — in the former you walk in and say, “I want the book in the third bookcase, top shelf, on the very left”, whereas with the IPFS you say, “I want the book called ‘Lord of the Rings’, and it’s called that because it features a magical ring created by an evil overlord”.

Except IPFS files are called things like bafkreifjidmkvxac66mdggzni3y2rljmtaqxqmda6sqicdnefp3lbk4dq, which is a “hash” generated from the content of the file. It’s like taking the original file, and squashing and mixing it up in a particular way that produces a fixed length string that is guaranteed to be unique (well, almost guaranteed), and referring to the file with that.

And because the IPFS is a decentralized peer-to-peer file system, if the file can’t be found on your system, the node starts asking other nodes out on the network if they have it, until it is found.

Finally, the IPFS also has folders. A folder is actually a file that contains two columns — the first contains a list of human readable file names, and the second contains the IPFS file hash names.

So if you create a folder, when you go to it using your web browser, for example by entering the link in the address bar, you are shown a list of the files that the folder “contains”. I put contains in quotes, because it doesn’t actually contain them in any real sense. Instead it contains pointers to the files using the IPFS hash name.

Not one, but two

In order to make submarining work, we are going to need two IPFS nodes running on our machine. The first one is going to be a private node, which we use to construct the files and folder name, and the folder data, and the second is going to be used to publish the files as and when we choose, on the network.

The first step is to install the IPFS command line software, as explained here: https://docs.ipfs.io/install/ipfs-desktop/#ubuntu

Now we need two folders, one for the private node, and one for the public node:

mkdir ipfs-private
mkdir ipfs-public

The private node

We will begin with setting up our private node, using the guide provided here: https://medium.com/@s_van_laar/deploy-a-private-ipfs-network-on-ubuntu-in-5-steps-5aad95f7261b that you can read to understand what the commands below are doing:

$ IPFS_PATH=~/ipfs-private ipfs init
$ echo -e “/key/swarm/psk/1.0.0/\n/base16/\n`tr -dc ‘a-f0-9’ < /dev/urandom | head -c64`” > ~/ipfs-private/swarm.key
$ IPFS_PATH=~/ipfs-private ipfs bootstrap rm --all
$ IPFS_PATH=~/ipfs-private ipfs config show | grep “PeerID”
$ IPFS_PATH=~/ipfs-private ipfs bootstrap add /ip4/127.0.0.1/tcp/4001/ipfs/12D3KooWCkjxCh7gqGGV4EGBJLfuN84QtNNVgsygNSfbiSr9gA2r
$ export LIBP2P_FORCE_PNET=1
$ IPFS_PATH=~/ipfs-private ipfs daemon &

Make sure you swap out your personal PeerID for the one I have in the code above.

We now have a private IPFS node running on our machine that we can use to determine what our IPFS filenames and folder names are going to be, without actually publishing them to a public network.

The public node

Configuring this involves less comands, but a bit of editing to a config file.

$ IPFS_PATH=~/ipfs-public ipfs init

The open ~/ipfs-public/configin the editor of your choice and change the port numbers, for example like this:

"Addresses": {
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4002",
      "/ip6/::/tcp/4002",
      "/ip4/0.0.0.0/udp/4002/quic",
      "/ip6/::/udp/4002/quic"
    ],
    "Announce": [],
    "NoAnnounce": [],
    "API": "/ip4/127.0.0.1/tcp/5002",
    "Gateway": "/ip4/127.0.0.1/tcp/8082"

Finally, start the second IPFS daemon:

IPFS_PATH=~/ipfs-public ipfs daemon &

Deleting the scrap

Both repositories will be pre-populated with a number of files such as the IPFS readme file and a few others. I like to get rid of them to reduce the clutter, which means unpinning them and then garbage collecting them. Here are the commands to do that:

$ IPFS_PATH=~/ipfs-private ipfs pin ls --type recursive | cut -d' ' -f1 | xargs -n 1 -I % -0 sh -c 'IPFS_PATH=~/ipfs-private ipfs pin rm %' sh
$ IPFS_PATH=~/ipfs-private ipfs repo gc

This will leave you with one irremovable file, with the filename QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn — this is the filename for an empty directory, and that’s why you can’t remove it.

And here are two useful commands. To list the files and folders in your repo, use:

$ IPFS_PATH=~/ipfs-private ipfs refs local

And to remove any files that are not pinned, use:

$ IPFS_PATH=~/ipfs-private ipfs repo gc

Test data

Now we are going to make a test folder and some test metadata, which we are going to submarine.

$ mkdir test
$ echo '{}' > ./test/0.json
$ echo '{"rarity": "rare"}' > ./test/1.json
$ echo '{"rarity": "common"}' > ./test/2.json
$ echo '{"rarity": "mythical"}' > ./test/3.json
$ echo '{"rarity": "common"}' > ./test/4.json

So you can see here that we have some metadata for a bunch of NFTs that are even more pointless than usual, because there’s no link to an image or any kind of data other than their rarity.

I always add a0.json as a dummy test file, so that once I have completed the steps below, I can release it and check that the folder and file method is working. After all, if you’re going to deploy a frozen NFT contract, you want to be sure your base URI is functional before you start advertising it to everyone.

Private uploading

We can now add and pin the test folder and all the files in it. You can’t just add the folder and then add the files later, because a folder is actually a file that contains pointers to the individual files in IPFS named-format, with a human-readable lookup for those files.

$ IPFS_PATH=~/ipfs-private ipfs add — pin=false — recursive — cid-version=1 test

As a response you should get the IPFS names of all the files, and the IPFS folder name:

And because your folder and files should be exactly the same as mine, if you’re following this as a tutorial, you’ll have the same bafkre… and bafybeie… results that I do.

You can now dump the binary content of the folder object to a file:

$ ipfs block get bafybeie3u25sm4eercubxe73sr6tgz4lzi4wg3ow27fp7ombtu7owcioeq > folder.bin

Note that files start with bafkre… in the version 2 naming system, and folders start with bafybeie…

Then we clear out all the files that we just pinned:

$ IPFS_PATH=~/ipfs-private ipfs repo gc

Public uploading

The folder.bin file now contains just the required data for the folder and nothing more. So we will switch to the public IPFS server, and upload it:

$ cat folder.bin | IPFS_PATH=~/ipfs-public ipfs dag put — store-codec dag-pb — input-codec dag-pb

Then we need to pin the folder to the repository, without adding all the files it points to:

$ IPFS_PATH=~/ipfs-public ipfs pin add — recursive=false bafybeie3u25sm4eercubxe73sr6tgz4lzi4wg3ow27fp7ombtu7owcioeq

And let’s check that it’s there:

See — the folder object is there, but the files that it points to are not. And because those files were only added to (and then removed from) the private IPFS repository, the public can’t get a sneak preview.

It’s never simple…

But there is a problem. For some reason, just adding the folder to the repo works, but the gateway won’t serve it up. You can see this if you go to your local IPFS file server, by entering the following in the address bar of your browser:

http://127.0.0.1:8082/ipfs/bafybeie3u25sm4eercubxe73sr6tgz4lzi4wg3ow27fp7ombtu7owcioeq

In fact, the folder won’t resolve until all the individual files are added, so keep this open in a tab.

But, you can add the files one by one, and then see them. Open the following in your browser:

http://127.0.0.1:8082/ipfs/bafybeie3u25sm4eercubxe73sr6tgz4lzi4wg3ow27fp7ombtu7owcioeq/0.json

Your browser should hang. Then run the following:

$ IPFS_PATH=~/ipfs-public ipfs add — cid-version=1 test/0.json

Within a few seconds, the browser displays the metadata file.

Now add the rest of the files, one by one, and see how they too can be resolved and retrieved:

$ IPFS_PATH=~/ipfs-public ipfs add — cid-version=1 test/1.json
$ IPFS_PATH=~/ipfs-public ipfs add — cid-version=1 test/2.json
$ IPFS_PATH=~/ipfs-public ipfs add — cid-version=1 test/3.json

When you execute the final command in the above list, the folder will resolve too, and your browser will show all the file links: