Skip to main content
Matrix42 Self-Service Help Center

How to implement custom storage providers

Overview 

Following the topic of How to create custom storage type, it is time to see how the custom provider can be built using the real sample provided below. 

The template of the project located at the following repository: Matrix42 File Storage Template. It contains all major components described below: 

  • API controller (see Matrix42.LargeFilesStorage.Services.LargeFilesStorageAccountController
  • Custom provider (see Matrix42.LargeFilesStorage.BizLogic.LargeFilesStorageProvider) 
  • Client-side custom provider (see Matrix42.LargeFilesStorage.Workspaces\src\providers\large-files-storage-provider.ts

 For the web developers, this should become a handbook to start writing their own implementation.  

The template is intended to provide a very basic overview of how to create and register client-side and server-side providers. All the controller and provider names are given just for learning purposes and do not necessarily mean to be used in production. 

Getting started 

To get started with the template, please review README to ensure all prerequisites are met and obligatory steps are executed to compile and run solutions. 

To demonstrate the attachment control design and the relations of its components, please, review the following diagram:  

 Attachment Common Control.drawio.png

The attachment control is designed in a way to handle the following operations:  

  • Upload: uploads the file to the storage
  • Download: downloads the file from the storage
  • Get Files: gets the list of the files from all storages
  • Count: counts the amount of the files on all storages
  • Comment: comments the file
  • Delete: deletes the file from the storage

The attachment control is built based on several core components required to handle attachments: 

  • Attachment Service and client-side providers 
  • Storage Container, server-side providers, common storage API 

Attachment Service and client-side providers 

Attachment Service is a service to manage client-side providers. It allows to register new providers and store them in the local underlying collection. The conceptual role that stands behind the Attachment Service is to: 

  • Register any number of custom providers that implement IAttachmentProvider interface 
  • Identify the right provider to execute core operations: Upload and Download the file 
  • Support other common operations: Get Files, Count, Comment, Delete 

While uploading or downloading the files, in fact, we must directly interact with the 3rd party storages (i.e., Azure Blob, Google Drive, Dropbox, etc.). This imposes an additional obligation on the service to provide a custom mechanism to build custom HTTP requests (the requests may vary depending on the external storage). Therefore, client-side providers are introduced to flexibly handle core Upload and Download operations by adding custom headers, etc. 

Core operations: Upload and Download the file 

Both core operations require signed URLs to be used. Due to security reasons, the authorization to the external storages must not happen in the foreground. Thus, the corresponding API controllers must be implemented to serve GetUploadUrl and GetDownloadUrl endpoints to return signed URLs. 

Other operations: Get Files, Count, Comment, Delete 

However, the rest of the operations do not require standing communication with the external storages. This can be achieved using the meta-information the system pre-saves internally for every file ever uploaded to the external storage. The meta-information for the single file contains: 

  • File name 
  • File size 
  • Thumbnail 
  • Comment 
  • Object Id 
  • Storage Id 

Using the approach above, the system could easily perform Count, Get Files, and Comment operations. All common requests are performed against the local MS SQL database and meta information stored DWPFileInfo table. The common API controller is intended to process listed operations.  

API controller

The role of the common API controller is wider and includes a few more objectives: 

  • Get the best-matching provider: this is required when multiple providers are registered

  • Delete the file: determines the provider and deletes the file from external storage and cleans up the meta-information for the file

The best-matching provider is determined using Priority and Apply To criteria for the enabled providers. 

For the clean system, the Default Storage Provider is initiated and included in the Attachment Service as a single provider to be used when processing attachments. By default, MS SQL database is used to store the files (see Files table).  

Storage Container 

Storage Container is a container to manage server-side providers. By analogy with the Attachment Service, the server must include the corresponding set of providers which implements IStorageProvider interface (see Matrix42.StorageService.Contracts.IStorageProvider for more details).  

The provider is aimed to serve the following operations on the server-side: 

  • Delete 
  • Download 
  • Upload 

Optionally, the provider can also implement ISignedUrlStorageProvider interface to support GetUploadUrl and GetDownloadUrl. If that is the case, the custom controller could use the provider injected to deliver GetUploadUrl and GetDownloadUrl API endpoints accordingly.  

The custom provider must be registered to the Storage Container as shown below:  

public void Register(IDependencyContainer container, IDependencyResolver resolver) 
{ 
    // Register IStorageProvider and ISignedUrlStorageProvider types 
    container.RegisterType<IStorageProvider, CustomStorageProvider>("CustomProvider"); 
    container.RegisterType<ISignedUrlStorageProvider, CustomStorageProvider>("SignedUrlProvider"); 
   
    // Register provider in storage container 
    var storageProviderContainer = resolver.Get<IStorageProviderContainer>(); 
    storageProviderContainer.Register(() => resolver.Get<IStorageProvider>("CustomProvider")); 
} 
  • Was this article helpful?