Before we understand what the PrefixedStorage struct is, we should first understand the problem they attempt to solve. We may be tempted to think that the key-value model as described in Storage solves all our storage needs. And we'd technically be correct in thinking that. It is possible to efficiently store vast amounts of user data by just using the methods described in that page. However, we will see that doing this would be cumbersome for the developer in many cases. That's why there are many additional storage structures built on top of the methods described in Storage. Prefixed storage is an additional structure built on top of these methods for the convenience of the developer.
We will go over a storage problem and discuss how we might tackle it with the methods we learnt in Storage to get to the root of what prefixed storage actually is. Suppose that we have contract that stores a different password (String) for each wallet address that interacts with it (so that the wallets can later give that password when querying the contract). How would we want to store all these passwords so that they can be checked when it is time for queries?
Our first approach might be to use a hashmap that stores all the passwords, and save it as binary using the methods we already know. So we would add the following lines to the init
And then we can proceed to load, and add each users' password whenever a user sends the create_password
HandleMsg. Note that I'm assuming we would generate the &[u8]
key for the hashmap from the user's wallet address. This approach has a huge problem! That is whenever we want to add another wallet's password to the hashmap, we must load the entire hashmap with all the passwords stored inside it. This will increase gas costs as we gain thousands of users. Moreover, loading the hashmap for queries will strain the node.
We now realize that we can actually generate&[u8]
storage keys for each user wallet and save the password directly to deps.storage
as described in Storage. This way, we can use may_load
to check if a user has a password, and if he does, learn what it is without loading all the other passwords. Then we would use the following lines of code to save a password
This method does work! However, what if we want to save additional things that is associated to each user, such as token balances. Then using save on the same key would overwrite our previous data. The solution is to add a prefix to the storage key so that we know it belongs to the password property. One way to implement this is the following:
This works but can be cumbersome and ugly, especially if you need to build more functionalities around this idea. This is exactly what prefixed storage does in the background! Prefixed Storage is built to solve this problem while hiding away all the ugliness.
PrefixedStorage is a struct that keeps track of all the storage keys that are used to store various data under a shared namespace similar to described above. Prefixed storage is often used to keep track of storage keys of deps.storage
, but it can in fact be used to store the keys of other storage structures.
The compiler only allows one mutable reference to the underlying storage to be valid at one point. Thus, you cannot have more than one mutable Prefixed Storage objects at the same time.
Let's solve the password problem that was mentioned above with prefixed storage. The solution to this problem resembles viewing keys for permissioned viewing. You will learn more about this in the coming sections. We first define a namespace (prefix) in state.rs
.
The idea then is that all the wrapper functions that we've learnt in Storage also work on prefixed storage. We will only demonstrate save
and may_load
by example.
We would instantiate a mutable PrefixedStorage object using the following lines of code to save a new password
Notice that we are using the same storage wrapper functions from Storage on prefixed storage.
The reason why we can use the same wrapper functions is because Prefixed Storage implements the Storage trait, even though most of what it's doing is to add a prefix on top of the keys we provide to it.
We may use both load
and may_load
We may load data from a mutable Prefixed Storage instance with the following lines of code
In queries, we don't want to use PrefixedStorage struct. Instead we use ReadonlyPrefixedStorage struct. It works in the exact same way, except that you cannot save to it.
You can have as many ReadonlyPrefixedStorage objects as you want at the same time, unlike PrefixedStorage.
We can use the remove
wrapper function for this.
This hashmap-like storage structure uses generic typed keys to store objects.
A Keymap is Secret toolkit hashmap-like storage structure that uses generic typed keys to store objects. It allows iteration with paging over keys and/or items without guaranteed ordering, although the order of insertion is preserved until you remove objects.
An example use-case for a keymap is if you want to contain a large amount of votes and iterate over them in the future. Since iterating over large amounts of data at once may be prohibitive, a keymap allows you to specify the amount of data that will be returned in each page. We will implement this voting example below to show how keymaps can be utilized in your own projects.
To import this package (and also the packages that we will be using for unit tests), add the following dependencies to your Cargo.toml
file
To import and initialize a keymap, use the following packages in your test environment:
Now let's write our first test!
Let's start by creating a function that inserts key-value pairs into a keymap. This code defines a test function called test_keymap_perf_vote_insert()
that creates a new Keymap
that maps Vec<u8>
keys to String
values, and then inserts 1000 key-value pairs into the Keymap
. Each key is a Vec<u8>
containing an integer from 0 to 999, and each value is a String
containing the text "I vote yes".
This test is passing! Which means that it asserts that the Keymap
contains 1000 key-value pairs, which map a number from 0 - 999 with the string "I vote yes."
Iterating with Keymaps
There are two methods that create an iterator in Keymap. These are .iter
and .iter_keys
. iter_keys
only iterates over the keys whereas iter
iterates over (key, item) pairs. Let's use iter
to test the iterator functionality of aKeymap
that maps Vec<u8>
keys to a struct Vote.
We are going to create a new Keymap
that maps Vec<u8>
keys to a struct Vote
that has two fields, vote
and person
. It then inserts two key-value pairs into the Keymap
, where the keys are the byte vectors b"key1".to_vec()
and b"key2".to_vec()
, and the values are Vote
structs containing information about the vote and the person who cast it.
We then use iter()
to check that the size of the iterator is 2, which means there are two key-value pairs in the Keymap:
Our test is passing! This means that the first element returned by the iterator matches the expected key-value pair for key1
, and that the second element returned by the iterator matches the expected key-value pair for key2
.
For further examples demonstrating the usage of keymaps, refer to the Secret Toolkit repo here.
An explainer on the varying storage frameworks for Secret contracts
CosmWasm uses a key-value storage design. Smart contracts can store data in binary, access it through a storage key, edit it, and save it. Similar to a HashMap, each storage key is associated with a specific piece of data stored in binary. The storage keys are formatted as references to byte arrays (&[u8]
).
One advantage of the key-value design is that a particular data value is only loaded when the user explicitly loads it using its storage key. This prevents any unnecessary data from being processed, saving resources.
Any type of data may be stored this way as long as the user can serialize/deserialize (serde) the data to/from binary. Doing this manually every single time is cumbersome and repetitive, this is why we have wrapper functions that does this serde process for us.
All the data is actually stored in deps.storage
, and the examples below show how to save/load data to/from there with a storage key.
Creating a storage key is simple, any way of generating a constant &[u8]
suffices. People often prefer generating these keys from strings as shown in the example below.
For example, the above key is likely used to store some data related to core configuration values of the contract. The convention is that storage keys are often all created in state.rs
, and then imported to contract.rs
. However, since storage keys are just constants, they could be declared anywhere in the contract.
The example above also highlights that storage keys are not meant to be secret nor hard to guess. Anyone who has the open source code can see what the storage keys are (and of course this is not enough for a user to load any data from the smart contract).
As mentioned above, serializing/deserializing data while loading/saving it with a key is cumbersome. This is why we often use wrapper functions written by community members. There are three common wrapper functions that are included in the state.rs
of most secret contract templates.
A commonly used wrapper function to save data is the following. This function overwrites previously saved data associated to that storage key.
Note that value
can be of any Struct type. The only condition is that this Struct must derive the Serialize and Deserialize traits from serde with the following line above its Struct declaration.
There are some structs that cannot be serialized/deserialized by the bincode2 struct these wrapper functions use! See JSON Storage Wrapper Functions section below to see what happens in this case.
The wrapper may be used to save data in the following manner:
A commonly used wrapper function to load data from storage is the following:
Note that this function throws an error if there is no data previously saved with that storage key.
When loading data, rust must be told what Struct to expect after deserializing.
In some instances you may be unsure whether there is any data stored with a particular key. In this case you want to use may_load()
which wraps any data inside within an option. Returning None
if there is no value saved with that key, and returning Some(value)
if there is some value saved. An example function for this is:
One of the most common use cases of this function is when retrieving viewing keys for users. However, viewing keys use PrefixedStorage, and we will see this in the next section. So instead, I will show a line of code that retrieves a list of minters for an NFT.
This line of code returns the list of minters if it is saved to MINTERS_KEY, otherwise, it returns an empty list.
A commonly used wrapper function to remove saved data from storage is the following. This might be the only wrapper function that does not make anything more convenient, because there is no serialize/deserialize implemented.
The following code removes minters. This code does not let you know if there was any previously saved data to that storage key.
The wrapper functions we learned above use bincode2 struct (from secret-toolkit
) to serde the data being saved/read on the smart contract. However, bincode2 uses floats when deserializing rust enum variants, thus, bincode2 cannot serde enum variants in cosmwasm. This is why cosmwasm uses Json serde by default, not bincode2.
The following is an example, from the reference SNIP-721 implementation, of a struct that cannot be saved/loaded by the wrapper functions we saw above because it uses an enum.
In these cases, we can use the Json struct from secret-toolkit
to serde structs that use enums. This also creates the need for for new wrapper functions
The usage of this function is extremely similar to the save
wrapper function we discussed above.
bincode2 serde is more efficient than json serde
The usage of this function is extremely similar to the load
wrapper function we discussed above.
The usage of this function is extremely similar to the may_load
wrapper function we discussed above.
The remove
wrapper function above works the same because it doesn't serde
Optimizing your code is especially important on blockchain due to the fact that inefficient code costs your users money and can jam up the entire network. With any function you make, try to complete it in as few steps as possible and avoid unnecessary computation. One of the most costly actions you can make is loading and saving data, thus you should ensure you are only saving what is needed. Aside from this, there are a few other tips we can provide.
When dealing with large sets of data, vectors and arrays can be especially problematic because in order to save load a single item within them, you must effectively load the entire vector and all of its contents, which naturally can amount to an enormous amount of inefficiency when you only need one item! There are several ways around this: Keymap, AppendStore, or even deconstructing vectors into a set of data keyed to a counter variable. Research each options pros and cons, and choose which one is best for your particular contract.
AppendStore is meant to replicate the functionality of an append list in a cosmwasm efficient manner. The length of the list is stored and used to pop/push items to the list. It also has a method to create a read only iterator.
This storage object also has the method remove
to remove a stored object from an arbitrary position in the list, but this can be extremely inefficient.
❗ Removing a storage object further from the tail gets increasingly inefficient. We recommend you use
pop
andpush
whenever possible.
The same conventions from Item
also apply here, that is:
AppendStore has to be told the type of the stored objects. And the serde optionally.
Every methods needs it's own reference to deps.storage
.
To import and initialize this storage object as a static constant in state.rs
, do the following:
❗ Initializing the object as const instead of static will also work but be less efficient since the variable won't be able to cache length data.
Often times we need these storage objects to be associated to a user address or some other key that is variable. In this case, you need not initialize a completely new AppendStore inside contract.rs
. Instead, you can create a new AppendStore by adding a suffix to an already existing AppendStore. This has the benefit of preventing you from having to rewrite the signature of the AppendStore. For example
Sometimes when iterating these objects, we may want to load the next n
objects at once. This may be prefered if the objects we are iterating over are cheap to store or if we know that multiple objects will need to be accessed back to back. In such cases we may want to change the internal indexing size (default of 1). We do this in state.rs
:
The main user facing methods to read/write to AppendStore are pop
, push
, get_len
, set_at
(which replaces data at a position within the length bound), clear
(which deletes all data in the storage), remove
(which removes an item in an arbitrary position, this is very inefficient). An extensive list of examples of these being used can be found inside the unit tests of AppendStore found in append_store.rs
.
AppendStore also implements a readonly iterator feature. This feature is also used to create a paging wrapper method called paging
. The way you create the iterator is:
More examples can be found in the unit tests. And the paging wrapper is used in the following manner: