Working with a Singleton in CosmWasm

In CosmWasm smart contracts, a Singleton is a specialized structure that combines the concepts of PrefixedStorage and TypedStorage, allowing you to store and manage a single data entity efficiently under a specific storage key. This is particularly useful when you need to store just one instance of a state (such as contract configuration or global data) without the need for complex key management.

How Singleton Works:

pub struct Singleton<'a, T>
where
    T: Serialize + DeserializeOwned,
{
    storage: &'a mut dyn Storage,
    key: Vec<u8>,
    data: PhantomData<T>,
}

• storage: &'a mut dyn Storage: The Singleton holds a mutable reference to the storage layer, which is where the contract data is stored on-chain. This reference allows it to interact with and modify the stored data.

• key: Vec<u8>: The Singleton operates on a single storage key, provided during its instantiation. This key is typically transformed using to_length_prefixed to avoid collisions with other stored data, ensuring that the contract can safely manage this unique piece of state.

• data: PhantomData<T>: Rust’s PhantomData is a marker for the data type T without actually holding an instance of it. This allows the Singleton to enforce that it works with a specific type (T) that must implement Serialize and DeserializeOwned traits, which are essential for encoding/decoding the data stored in the singleton.

Key Features of Singleton:

1. No Need for Multiple Keys: A Singleton simplifies storage management by removing the need to manually manage multiple keys. The storage key is predefined in the constructor, and all operations (read, write, update) are tied to this single key. This ensures you are only ever working with one data object in storage.

2. Type-Safe Storage: The Singleton uses TypedStorage functionality, which ensures that the data stored and retrieved from the storage is type-safe. This means you always work with the exact type you’ve defined (in this case, T), reducing the risk of errors during serialization and deserialization.

3. Avoids Key Collisions: By applying the to_length_prefixed transformation to the given storage key, the Singleton ensures there are no name collisions in storage. This is particularly useful in larger contracts where multiple data points might otherwise share similar key names.

There are common wrapper functions that are included in the state.rs of most secret contract templates, which are save, load, may_load, remove, and update.

save

The save function overwrites previously saved data associated to that storage key. save will serialize the model and store, returns an error on serialization issues:

 pub fn save(&mut self, data: &T) -> StdResult<()> {
        self.storage.set(&self.key, &to_vec(data)?);
        Ok(())
    }

• Purpose: It takes a reference to the storage and the data to be saved (usually the contract's state).

• Usage: You call save when you want to persist new data or update existing data in storage.

load

This function reads or retrieves data from the contract's state. It is commonly used when you need to access a piece of data stored in the contract, such as the current state or configuration.

 /// load will return an error if no data is set at the given key, or on parse error
    pub fn load(&self) -> StdResult<T> {
        let value = self.storage.get(&self.key);
        must_deserialize(&value)
    }

• Purpose: It accesses the storage and loads the state associated with the provided key.

• Usage: load is used whenever the contract needs to read existing data, such as fetching the configuration or state before making any modifications.

may_load

 /// may_load will parse the data stored at the key if present, returns Ok(None) if no data there.
    /// returns an error on issues parsing
    pub fn may_load(&self) -> StdResult<Option<T>> {
        let value = self.storage.get(&self.key);
        may_deserialize(&value)
    }

• Purpose: This function attempts to retrieve and deserialize data from storage, returning Ok(None) if no data exists at the given key. It returns an error only if deserialization fails.

• Usage: may_load is used when the existence of the data is optional. It allows you to safely check if data exists and handle cases where the data might not be present.

• Behavior: It fetches the data associated with the key and attempts to deserialize it using may_deserialize. If the key does not have any data, it returns Ok(None) instead of an error, making it more forgiving than load.

remove

    pub fn remove(&mut self) {
        self.storage.remove(&self.key)
    }

• Purpose: This function deletes or removes data associated with a specific key from the contract’s storage.

• Usage: remove is used when you no longer need to store certain data in the contract and want to free up space by removing the key-value pair from storage.

• Behavior: It accesses the storage object and deletes the data associated with the provided key.

update

This function loads, modifies, and then saves data back to the contract's state. It’s commonly used when the contract state needs to be modified, such as incrementing a counter or updating a setting.

/// update will load the data, perform the specified action, and store the result
    /// in the database. This is shorthand for some common sequences, which may be useful
    ///
    /// This is the least stable of the APIs, and definitely needs some usage
    pub fn update<A, E>(&mut self, action: A) -> Result<T, E>
    where
        A: FnOnce(T) -> Result<T, E>,
        E: From<StdError>,
    {
        let input = self.load()?;
        let output = action(input)?;
        self.save(&output)?;
        Ok(output)
    }

• Purpose: It allows for atomic updates to the state by accepting a closure (a function passed as an argument) that modifies the data and then saves it back to storage.

• Usage: update is ideal for changing existing data in the state, like incrementing a value, while ensuring that the storage operation is performed safely in one transaction.

Summary

• save: Persists or updates state in the contract’s storage.

• load: Retrieves and reads existing data from the contract’s storage.

• may_load: Fetches and deserializes data if it exists, returning Ok(None) if no data is found at the key, and an error if deserialization fails.

• remove: Deletes the data associated with a specific key from storage.

• update: Modifies existing data in storage, allowing for atomic and safe updates.

These wrapper functions provide clean abstractions to manage the contract's state, making the contract logic easier to write and maintain. They are especially useful in ensuring that storage operations (which are critical for on-chain data) are handled efficiently and consistently across the contract.

Optimizing your code is especially important on blockchain due to the fact that inefficient code costs your users money and can jam up the entire network. With any function you make, try to complete it in as few steps as possible and avoid unnecessary computation. One of the most costly actions you can make is loading and saving data, thus you should ensure you are only saving what is needed. Aside from this, there are a few other tips we can provide.

Avoid Vector and Arrays

When dealing with large sets of data, vectors and arrays can be especially problematic because in order to save load a single item within them, you must effectively load the entire vector and all of its contents, which naturally can amount to an enormous amount of inefficiency when you only need one item! There are several ways around this: Keymap, AppendStore, or even deconstructing vectors into a set of data keyed to a counter variable. Research each options pros and cons, and choose which one is best for your particular contract.

Why Is Prefixed Storage Needed?

Before we understand what the PrefixedStorage struct is, we should first understand the problem they attempt to solve. We may be tempted to think that the key-value model as described in Storage solves all our storage needs. And we'd technically be correct in thinking that. It is possible to efficiently store vast amounts of user data by just using the methods described in that page. However, we will see that doing this would be cumbersome for the developer in many cases. That's why there are many additional storage structures built on top of the methods described in Storage. Prefixed storage is an additional structure built on top of these methods for the convenience of the developer.

The Problem

We will go over a storage problem and discuss how we might tackle it with the methods we learnt in Storage to get to the root of what prefixed storage actually is. Suppose that we have contract that stores a different password (String) for each wallet address that interacts with it (so that the wallets can later give that password when querying the contract). How would we want to store all these passwords so that they can be checked when it is time for queries?

The Naive Approach

Our first approach might be to use a hashmap that stores all the passwords, and save it as binary using the methods we already know. So we would add the following lines to the init

let passwords: HashMap<&[u8], String> = HashMap::new();

save(&mut deps.storage, PASSWORDS_KEY, &passwords)?;

And then we can proceed to load, and add each users' password whenever a user sends the create_password HandleMsg. Note that I'm assuming we would generate the &[u8] key for the hashmap from the user's wallet address. This approach has a huge problem! That is whenever we want to add another wallet's password to the hashmap, we must load the entire hashmap with all the passwords stored inside it. This will increase gas costs as we gain thousands of users. Moreover, loading the hashmap for queries will strain the node.

The Reasonable Approach

We now realize that we can actually generate&[u8] storage keys for each user wallet and save the password directly to deps.storage as described in Storage. This way, we can use may_load to check if a user has a password, and if he does, learn what it is without loading all the other passwords. Then we would use the following lines of code to save a password

let key: &[u8] = env.message.sender.to_string().as_bytes();
save(&mut deps.storage, key, &msg.password)?;

This method does work! However, what if we want to save additional things that is associated to each user, such as token balances. Then using save on the same key would overwrite our previous data. The solution is to add a prefix to the storage key so that we know it belongs to the password property. One way to implement this is the following:

let prefix: String = "passwords".to_string();
let key: &[u8] = ( prefix + env.message.sender.to_string()).as_bytes();
save(&mut deps.storage, key, &msg.password)?;

This works but can be cumbersome and ugly, especially if you need to build more functionalities around this idea. This is exactly what prefixed storage does in the background! Prefixed Storage is built to solve this problem while hiding away all the ugliness.

What Is Prefixed Storage?

PrefixedStorage is a struct that keeps track of all the storage keys that are used to store various data under a shared namespace similar to described above. Prefixed storage is often used to keep track of storage keys of deps.storage, but it can in fact be used to store the keys of other storage structures.

Example

Let's solve the password problem that was mentioned above with prefixed storage. The solution to this problem resembles viewing keys for permissioned viewing. You will learn more about this in the coming sections. We first define a namespace (prefix) in state.rs.

pub const PREFIX_PASSWORDS: &[u8] = b"passwords";

The idea then is that all the wrapper functions that we've learnt in Storage also work on prefixed storage. We will only demonstrate save and may_load by example.

Saving To Prefixed Storage

We would instantiate a mutable PrefixedStorage object using the following lines of code to save a new password

let mut password_store = PrefixedStorage::new(PREFIX_PASSWORDS, &mut deps.storage);
let key: &[u8] = env.message.sender.to_string().as_bytes();
save(&mut password_store, key, &msg.password)?;

The reason why we can use the same wrapper functions is because Prefixed Storage implements the Storage trait, even though most of what it's doing is to add a prefix on top of the keys we provide to it.

Loading From Prefixed Storage

We may use both load and may_load We may load data from a mutable Prefixed Storage instance with the following lines of code

let mut password_store = PrefixedStorage::new(PREFIX_PASSWORDS, &mut deps.storage);
let key: &[u8] = env.message.sender.to_string().as_bytes();
// Throws error if there is no password saved before
let password: String = load(&password_store, key)?;

In queries, we don't want to use PrefixedStorage struct. Instead we use ReadonlyPrefixedStorage struct. It works in the exact same way, except that you cannot save to it.

let password_store = ReadonlyPrefixedStorage::new(PREFIX_PASSWORDS, &deps.storage);
let key = address.to_string().as_bytes();
let may_password: Option<String> = may_load(&password_store, key)?;

Removing From Prefixed Storage

We can use the remove wrapper function for this.

let mut password_store = PrefixedStorage::new(PREFIX_PASSWORDS, &mut deps.storage);
let key: &[u8] = env.message.sender.to_string().as_bytes();
remove(&mut password_store, key);

What are Keymaps?

A Keymap is Secret toolkit hashmap-like storage structure that uses generic typed keys to store objects. It allows iteration with paging over keys and/or items without guaranteed ordering, although the order of insertion is preserved until you remove objects.

An example use-case for a keymap is if you want to contain a large amount of votes and iterate over them in the future. Since iterating over large amounts of data at once may be prohibitive, a keymap allows you to specify the amount of data that will be returned in each page. We will implement this voting example below to show how keymaps can be utilized in your own projects.

Importing the subpackage

To import this package (and also the packages that we will be using for unit tests), add the following dependencies to your Cargo.toml file

cosmwasm-std = { package = "secret-cosmwasm-std", version = "1.1.10" }
cosmwasm-storage = { package = "secret-cosmwasm-storage", version = "1.1.10" }
secret-toolkit-storage = "0.9.0"

Configuring the testing environment

To import and initialize a keymap, use the following packages in your test environment:

#[cfg(test)]
mod tests {
    use secret_toolkit_storage::{Item, Keymap};
    use serde::{Deserialize, Serialize};
    use cosmwasm_std::testing::MockStorage;
    use cosmwasm_std::StdResult;
    };

Now let's write our first test!

Inserting key-value pairs into a Keymap

Let's start by creating a function that inserts key-value pairs into a keymap. This code defines a test function called test_keymap_perf_vote_insert() that creates a new Keymap that maps Vec<u8> keys to String values, and then inserts 1000 key-value pairs into the Keymap. Each key is a Vec<u8> containing an integer from 0 to 999, and each value is a String containing the text "I vote yes".

#[test]
    fn test_keymap_perf_vote_insert() -> StdResult<()> {
        let mut storage = MockStorage::new();

        let total_items = 1000;

        let keymap: Keymap<Vec<u8>, String> = Keymap::new(b"votes");

        for i in 0..total_items {
            let key: Vec<u8> = (i as i32).to_be_bytes().to_vec();
            let value = String::from("I vote yes");
            keymap.insert(&mut storage, &key, &value)?;
        }

        assert_eq!(keymap.get_len(&storage)?, 1000);

        Ok(())
    }

This test is passing! Which means that it asserts that the Keymap contains 1000 key-value pairs, which map a number from 0 - 999 with the string "I vote yes."

Iterating with Keymaps

There are two methods that create an iterator in Keymap. These are .iter and .iter_keys. iter_keys only iterates over the keys whereas iter iterates over (key, item) pairs. Let's use iter to test the iterator functionality of aKeymap that maps Vec<u8> keys to a struct Vote.

We are going to create a new Keymap that maps Vec<u8> keys to a struct Vote that has two fields, vote and person. It then inserts two key-value pairs into the Keymap, where the keys are the byte vectors b"key1".to_vec() and b"key2".to_vec(), and the values are Vote structs containing information about the vote and the person who cast it.

We then use iter() to check that the size of the iterator is 2, which means there are two key-value pairs in the Keymap:

 #[test]
    fn test_keymap_votes_iter() -> StdResult<()> {
        let mut storage = MockStorage::new();

        let alice = "alice";
        let bob = "bob";

        #[derive(Serialize, Deserialize, Eq, PartialEq, Debug, Clone)]
        struct Vote {
            vote: String,
            person: String,
        }

        let keymap: Keymap<Vec<u8>, Vote> = Keymap::new(b"votes");
        let vote1 = Vote {
            vote: "I vote yes".to_string(),
            person: alice.to_string(),
        };
        let vote2 = Vote {
            vote: "I vote no".to_string(),
            person: bob.to_string(),
        };

        keymap.insert(&mut storage, &b"key1".to_vec(), &vote1)?;
        keymap.insert(&mut storage, &b"key2".to_vec(), &vote2)?;

        let mut x = keymap.iter(&storage)?;
        let (len, _) = x.size_hint();
        assert_eq!(len, 2);

        assert_eq!(x.next().unwrap()?, (b"key1".to_vec(), vote1));

        assert_eq!(x.next().unwrap()?, (b"key2".to_vec(), vote2));

        Ok(())
    }

Our test is passing! This means that the first element returned by the iterator matches the expected key-value pair for key1, and that the second element returned by the iterator matches the expected key-value pair for key2.

Additional Resources

For further examples demonstrating the usage of keymaps, refer to the Secret Toolkit repo here.

AppendStore is meant to replicate the functionality of an append list in a cosmwasm efficient manner. The length of the list is stored and used to pop/push items to the list. It also has a method to create a read only iterator.

This storage object also has the method remove to remove a stored object from an arbitrary position in the list, but this can be extremely inefficient.

❗ Removing a storage object further from the tail gets increasingly inefficient. We recommend you use pop and push whenever possible.

The same conventions from Item also apply here, that is:

1. AppendStore has to be told the type of the stored objects. And the serde optionally.

2. Every methods needs it's own reference to deps.storage.

Initialize

To import and initialize this storage object as a static constant in state.rs, do the following:

❗ Initializing the object as const instead of static will also work but be less efficient since the variable won't be able to cache length data.

Often times we need these storage objects to be associated to a user address or some other key that is variable. In this case, you need not initialize a completely new AppendStore inside contract.rs. Instead, you can create a new AppendStore by adding a suffix to an already existing AppendStore. This has the benefit of preventing you from having to rewrite the signature of the AppendStore. For example

Sometimes when iterating these objects, we may want to load the next n objects at once. This may be prefered if the objects we are iterating over are cheap to store or if we know that multiple objects will need to be accessed back to back. In such cases we may want to change the internal indexing size (default of 1). We do this in state.rs:

Read/Write

The main user facing methods to read/write to AppendStore are pop, push, get_len, set_at (which replaces data at a position within the length bound), clear (which deletes all data in the storage), remove (which removes an item in an arbitrary position, this is very inefficient). An extensive list of examples of these being used can be found inside the unit tests of AppendStore found in append_store.rs.

Iterator

AppendStore also implements a readonly iterator feature. This feature is also used to create a paging wrapper method called paging. The way you create the iterator is:

More examples can be found in the unit tests. And the paging wrapper is used in the following manner:

How Storage Works

uses a key-value storage design. Smart contracts can store data in binary, access it through a storage key, edit it, and save it. Similar to a HashMap, each storage key is associated with a specific piece of data stored in binary. The storage keys are formatted as references to byte arrays (&[u8]).

One advantage of the key-value design is that a particular data value is only loaded when the user explicitly loads it using its storage key. This prevents any unnecessary data from being processed, saving resources.

Any type of data may be stored this way as long as the user can serialize/deserialize (serde) the data to/from binary. Doing this manually every single time is cumbersome and repetitive, this is why we have wrapper functions that does this serde process for us.

All the data is actually stored in deps.storage , and the examples below show how to save/load data to/from there with a storage key.

Storage Keys

Creating a storage key is simple, any way of generating a constant &[u8] suffices. Developers often prefer generating these keys from strings as shown in the example below.

For example, the above key is likely used to store some data related to core configuration values of the contract. The convention is that storage keys are often all created in state.rs, and then imported to contract.rs. However, since storage keys are just constants, they could be declared anywhere in the contract.

The example above also highlights that storage keys are not meant to be secret nor hard to guess. Anyone who has the open source code can see what the storage keys are (and of course this is not enough for a user to load any data from the smart contract).

Prefixed Storage

One common technique in smart contracts, especially when multiple types of data are being stored, is to create separate sub-stores with unique prefixes. Thus instead of directly dealing with storage, we wrap it and put all Foo in a Storage with key "foo" + id, and all Bar in a Storage with key "bar" + id. This lets us add multiple types of objects without too much cognitive overhead. Similar separation like Mongo collections or SQL tables.

Since we have different types for Storage and ReadonlyStorage, we use two different constructors:

Please note that only one mutable reference to the underlying store may be valid at one point. The compiler sees we do not ever use foos after constructing bars, so this example is valid. However, if we did use foos again at the bottom, it would properly complain about violating unique mutable reference.

The takeaway is to create the PrefixedStorage objects when needed and not to hang around to them too long.

Typed Storage

As we divide our storage space into different subspaces or "buckets", we will quickly notice that each "bucket" works on a unique type. This leads to a lot of repeated serialization and deserialization boilerplate that can be removed. We do this by wrapping a Storage with a type-aware TypedStorage struct that provides us a higher-level access to the data.

Note that TypedStorage itself does not implement the Storage interface, so when combining with PrefixStorage, make sure to wrap the prefix first.

Beyond the basic save, load, and may_load, there is a higher-level API exposed, update. Update will load the data, apply an operation and save it again (if the operation was successful). It will also return any error that occurred, or the final state that was written if successful.

Bucket

Since the above idiom (a subspace for a class of items) is so common and useful, and there is no easy way to return this from a function (bucket holds a reference to space, and cannot live longer than the local variable), the two are often combined into a Bucket. A Bucket works just like the example above, except the creation can be in another function:

Singleton

Singleton works just like Bucket, except the save, load, update methods don't take a key, and update requires the object to already exist, so the closure takes type T, rather than Option<T>. (Use save to create the object the first time). For Buckets, we often don't know which keys exist, but Singletons should be initialized when the contract is instantiated.

Since the heart of much of the smart contract code is simply transformations upon some stored state, we may be able to just code the state transitions and let the TypedStorage APIs take care of all the boilerplate.

Summary

CosmWasm storage is built on a key-value storage design, similar to a HashMap, where data is stored in binary format and accessed using storage keys represented as byte arrays (&[u8]). Developers can serialize and deserialize data to/from binary, simplifying this process through wrapper functions. Storage keys are typically constants declared in state files and are not meant to be secret.

Prefixed storage allows for creating separate sub-stores with unique prefixes for organizing different data types, while Typed Storage provides a type-aware interface to reduce serialization boilerplate. CosmWasm also supports advanced abstractions like Buckets and Singletons. Buckets create subspaces for storing collections of items, while Singletons manage single elements, typically used for global contract configuration data. These APIs streamline storage handling, enabling efficient data management in smart contracts.

Now, we will explore some of these storage methods, such as PrefixedStorage, Singleton, and Keymap, in more depth to see how they interact with on-chain data.