FileStorage usage

The eridu-tech/file-storage component provides a way for managing files independent of underlying platform or storage.

Initial configuration

To begin using the FileStorage class, you'll need to create and configure an instance:

./samples/file-storage-initial-config.ts
import { MemoryFileStorageAdapter } from "eridu-tech/file-storage/memory-file-storage-adapter";
import { SignedFileStorageAdapter } from "eridu-tech/file-storage/signed-file-storage-adapter";
import { FileStorage } from "eridu-tech/file-storage";

export const fileStorage = new FileStorage({
    // You can choose the adapter to use
    adapter: new SignedFileStorageAdapter({
        adapter: new MemoryFileStorageAdapter(),
        urlAdapter: {},
    }),
});

await fileStorage.init();

info

Here is a complete list of settings for the FileStorage class.

FileStorage basics

Creating a file object

./samples/file-storage-create-file.ts
import { fileStorage } from "./file-storage-initial-config.js";

const file = fileStorage.create("file.txt");

info

Note the file object represents a reference to a file and doesnt create the real underlying file.

Writing buffered files

You can put a file. It upserts, so an existing file is overwritten:

./samples/file-storage-put.ts
import { fileStorage } from "./file-storage-initial-config.js";

await fileStorage.create("file.txt").put({ data: "TEXT 1" });
await fileStorage.create("file.txt").put({ data: "TEXT 2" });

info

Note you can pass the following types to the put method:

Buffer
ArrayBuffer
SharedArrayBuffer
string
Uint8Array
Int8Array
Uint16Array
Int16Array
Uint32Array
Int32Array
BigUint64Array
BigInt64Array
Float32Array
Float64Array
DataView

But usually you would use Uint8Array because it represents data as bytes.

You can pass additional optional metadata information to the put method:

./samples/file-storage-put-with-metadata.ts
import { fileStorage } from "./file-storage-initial-config.js";

await fileStorage.create("file.txt").put({
    data: "CONTENT",

    /**
     * You can explicitly set a custom Content-Type. If one is not provided, it will be inferred from the key. For example, a key ending in .txt (such as key-a.txt) will be assigned text/plain.
     * If the key contains a non-standard extension it will default to application/octet-stream.
     */
    contentType: "text/plain",

    /**
     * Note a default value is always provided. To explicitly unset a field and prevent it from being passed to the underlying adapter, pass in `null`.
     */
    contentLanguage: "en-US",

    /**
     * Note a default value is always provided. To explicitly unset a field and prevent it from being passed to the underlying adapter, pass in `null`.
     */
    contentEncoding: "gzip",

    /**
     * Note a default value is always provided. To explicitly unset a field and prevent it from being passed to the underlying adapter, pass in `null`.
     */
    contentDisposition: "inline",

    /**
     * Note a default value is always provided. To explicitly unset a field and prevent it from being passed to the underlying adapter, pass in `null`.
     */
    cacheControl: "no-cache",
});

Writing streamed files

You can put a file stream. It upserts, so an existing file is overwritten:

./samples/file-storage-put-stream.ts
import { fileStorage } from "./file-storage-initial-config.js";
import { createReadStream } from "node:fs";

await fileStorage
    .create("file.txt")
    .putStream({ data: createReadStream("./file.txt") });
await fileStorage
    .create("file.txt")
    .putStream({ data: createReadStream("./file.txt") });

info

Note you can pass the following types to the putStream method:

AsyncIteralbe<Buffer>
AsyncIteralbe<ArrayBuffer>
AsyncIteralbe<SharedArrayBuffer>
AsyncIteralbe<string>
AsyncIteralbe<Uint8Array>
AsyncIteralbe<Int8Array>
AsyncIteralbe<Uint16Array>
AsyncIteralbe<Int16Array>
AsyncIteralbe<Uint32Array>
AsyncIteralbe<Int32Array>
AsyncIteralbe<BigUint64Array>
AsyncIteralbe<BigInt64Array>
AsyncIteralbe<Float32Array>
AsyncIteralbe<Float64Array>
AsyncIteralbe<DataView>

But usually you would use AsyncIterable<Uint8Array> because it represents stream as bytes.

You can pass additional optional metadata information to putStream:

./samples/file-storage-put-stream-with-metadata.ts
import { fileStorage } from "./file-storage-initial-config.js";
import { createReadStream } from "node:fs";

await fileStorage.create("file.txt").putStream({
    data: createReadStream("./file.txt"),

    /**
     * You can explicitly set a custom Content-Type. If one is not provided, it will be inferred from the key. For example, a key ending in .txt (such as key-a.txt) will be assigned text/plain.
     * If the key contains a non-standard extension it will default to application/octet-stream.
     */
    contentType: "text/plain",

    /**
     * Note a default value is always provided. To explicitly unset a field and prevent it from being passed to the underlying adapter, pass in `null`.
     */
    contentLanguage: "en-US",

    /**
     * Note a default value is always provided. To explicitly unset a field and prevent it from being passed to the underlying adapter, pass in `null`.
     */
    contentEncoding: "gzip",

    /**
     * Note a default value is always provided. To explicitly unset a field and prevent it from being passed to the underlying adapter, pass in `null`.
     */
    contentDisposition: "inline",

    /**
     * Note a default value is always provided. To explicitly unset a field and prevent it from being passed to the underlying adapter, pass in `null`.
     */
    cacheControl: "no-cache",
});

You can also pass the file-size of the stream which used for optimizations by some adapters:

./samples/file-storage-put-stream-with-file-size.ts
import { fileStorage } from "./file-storage-initial-config.js";
import { createReadStream } from "node:fs";
import { stat } from "node:fs/promises";
import { FileSize } from "eridu-tech/file-size";

const { size } = await stat("./file.txt");

await fileStorage.create("file.txt").putStream({
    data: createReadStream("./file.txt"),
    fileSize: FileSize.fromBytes(size),
});

info

It is best practice to pass file-size whenever possible because of the optimizations.

Retrieving files

The file can be read as utf8 text:

./samples/file-storage-get-text.ts
import { fileStorage } from "./file-storage-initial-config.js";

const content = await fileStorage.create("file.txt").getText();

console.log(content);

The file can be read as Uint8Array:

./samples/file-storage-get-bytes.ts
import { fileStorage } from "./file-storage-initial-config.js";

const content = await fileStorage.create("file.txt").getBytes();

console.log(content);

The file can be read as web ArrayBuffer:

./samples/file-storage-get-array-buffer.ts
import { fileStorage } from "./file-storage-initial-config.js";

const content = await fileStorage.create("file.txt").getArrayBuffer();

console.log(content);

The file can be read as web stream:

./samples/file-storage-get-readable-stream.ts
import { fileStorage } from "./file-storage-initial-config.js";

const content = await fileStorage.create("file.txt").getReadableStream();

console.log(content);

info

Note all this methods return null if the file doesnt exists.

Checking file existence

You can check if the file exists:

./samples/file-storage-exists.ts
import { fileStorage } from "./file-storage-initial-config.js";

const exists = await fileStorage.create("file.txt").exists();

You can check if the file doesnt exists:

./samples/file-storage-missing.ts
import { fileStorage } from "./file-storage-initial-config.js";

const missing = await fileStorage.create("file.txt").missing();

Removing files

You can remove a file:

./samples/file-storage-remove.ts
import { fileStorage } from "./file-storage-initial-config.js";

await fileStorage.create("file.txt").remove();

You can remove multiple files:

./samples/file-storage-remove-many.ts
import { fileStorage } from "./file-storage-initial-config.js";

await fileStorage.removeMany([
    fileStorage.create("file-1.txt"),
    fileStorage.create("file-2.txt"),
    fileStorage.create("file-3.txt"),
]);

Retrieving file metadata

You can retrieve the file metadata. Null is returned if the file doesnt exists:

./samples/file-storage-get-metadata.ts
import { fileStorage } from "./file-storage-initial-config.js";

const metadata = await fileStorage.create("file.txt").getMetadata();
console.log(metadata);

The getMetadata returns FileMetadata type.

Patterns

Additional methods

These variants are equivalent to the standard methods but throw an error if the file does not exist.

getTextOrFail
getBytesOrFail
getArrayBufferOrFail
getReadableStreamOrFail
getMetadataOrFail

Copying files

You can copy a file. True is returned if the source exists, and the destination is overwritten:

./samples/file-storage-copy.ts
import { fileStorage } from "./file-storage-initial-config.js";

// Returns true when the source exists, the destination is overwritten.
const hasCopied = await fileStorage
    .create("source.txt")
    .copy("destination.txt");
console.log(hasCopied);

Use copyOrFail method to perform the same operations as the copy method but it throws an error if the source file is missing.

Moving files

You can move a file. True is returned if the source exists, and the destination is overwritten:

./samples/file-storage-move.ts
import { fileStorage } from "./file-storage-initial-config.js";

// Returns true when the source exists, the destination is overwritten.
const hasMoved = await fileStorage.create("source.txt").move("destination.txt");
console.log(hasMoved);

Use moveOrFail method to perform the same operations as the move method but it throws an error if the source file is missing.

Signed urls and public urls.

Create signed urls to allow clients to upload files directly to FileStorage.

Upload url methods:

getSignedUploadUrl: Returns the signed upload url string.

./samples/file-storage-get-signed-upload-url.ts
import { TimeSpan } from "eridu-tech/time-span";
import { fileStorage } from "./file-storage-initial-config.js";

const uploadUrl = await fileStorage.create("source.txt").getSignedUploadUrl({
    // All settings are optional
    ttl: TimeSpan.fromMinutes(10),
    // The content type will be infered from the filename by default
    contentType: "text/plain",
});
console.log(uploadUrl);

Create signed urls to allow clients to download files directly from FileStorage.

Download url methods:

getSignedDownloadUrl: Returns the signed download url string, and throws an error if the file does not exist.

./samples/file-storage-get-signed-download-url.ts
import { TimeSpan } from "eridu-tech/time-span";
import { fileStorage } from "./file-storage-initial-config.js";

const file = fileStorage.create("source.txt");
await file.put({
    data: "CONTENT",
});

const donwloadUrl = await file.getSignedDownloadUrl({
    // All settings are optional
    ttl: TimeSpan.fromMinutes(10),
    // The content type will be infered from the filename by default
    contentType: "text/plain",
    contentDisposition: "inline",
});
console.log(donwloadUrl);

Use these methods to retrieve a permanent link to a file that is publicly accessible within your storage provider.

getPublicUrl: Returns the public url as a string, and throws an error if the file does not exist or public access is not supported.

./samples/file-storage-get-public-url.ts
import { fileStorage } from "./file-storage-initial-config.js";

const file = fileStorage.create("source.txt");
await file.put({ data: "CONTENT" });

const publicUrl = await file.getPublicUrl();

console.log(publicUrl);

File instance variables

The File class exposes the key instance variable which is the filename:

./samples/file-storage-instance-variables.ts
import { fileStorage } from "./file-storage-initial-config.js";

const file = fileStorage.create("file.txt");

// Will return the file name
console.log(file.key);

Serialization and deserialization of file

File obejcts can be serialized, allowing them to be transmitted over the network to another server and later deserialized for reuse.

info

Note only file name will be saved when serialized and not it' content. Which makes it efficient to send file over the network.

In order to serialize or deserialize a file object you need pass an object that implements ISerderRegister contract like the Serde class to FileStorage.

Manually serializing and deserializing the file object:

./samples/file-storage-manual-serialization.ts
import { MemoryFileStorageAdapter } from "eridu-tech/file-storage/memory-file-storage-adapter";
import { SignedFileStorageAdapter } from "eridu-tech/file-storage/signed-file-storage-adapter";
import { FileStorage } from "eridu-tech/file-storage";
import { SuperJsonSerde } from "eridu-tech/serde/super-json-serde";

const serde = new SuperJsonSerde();

const fileStorage = new FileStorage({
    // You can laso pass in an array of Serde class instances
    serde,
    adapter: new SignedFileStorageAdapter({
        adapter: new MemoryFileStorageAdapter(),
        urlAdapter: {},
    }),
});

const file = fileStorage.create("file.txt");
const serializedFIle = serde.serialize(file);
const deserializedFIle = serde.deserialize(serializedFIle);

danger

When serializing or deserializing a file, you must use the same Serde instances that were provided to the FilStorage. This is required because the FilStorage injects custom serialization logic for IFile instance into Serde instances.

info

Note you only need manuall serialization and deserialization when integrating with external libraries.

As long you pass the same Serde instances with all other components you dont need to serialize and deserialize the file object manually.

./samples/file-storage-event-bus-serialization.ts
import { Redis } from "ioredis";
import { MemoryFileStorageAdapter } from "eridu-tech/file-storage/memory-file-storage-adapter";
import { SignedFileStorageAdapter } from "eridu-tech/file-storage/signed-file-storage-adapter";
import type { IFile } from "eridu-tech/file-storage/contracts";
import { FileStorage } from "eridu-tech/file-storage";
import { RedisPubSubEventBusAdapter } from "eridu-tech/event-bus/redis-pub-sub-event-bus-adapter";
import { EventBus } from "eridu-tech/event-bus";
import { SuperJsonSerde } from "eridu-tech/serde/super-json-serde";

const serde = new SuperJsonSerde();
const redis = new Redis("YOUR_REDIS_CONNECTION");

type EventMap = {
    "sending-file-over-network": {
        file: IFile;
    };
};
const eventBus = new EventBus<EventMap>({
    adapter: new RedisPubSubEventBusAdapter({
        client: redis,
        serde,
    }),
});

const fileStorage = new FileStorage({
    serde,
    adapter: new SignedFileStorageAdapter({
        adapter: new MemoryFileStorageAdapter(),
        urlAdapter: {},
    }),
});
const file = fileStorage.create("file.txt");

// We are sending the file over the network to other servers.
await eventBus.dispatch("sending-file-over-network", {
    file,
});

// The other servers will recieve the serialized file and automattically deserialize it.
await eventBus.addListener("sending-file-over-network", ({ file }) => {
    // The file is deserialized and can be used
    console.log("file:", file);
});

Separating file creation from manipulation

The library includes 3 additional contracts:

IReadableFile - Allows only for reading a file.
IFile - Allows for both reading and manipulating the file.
IFileFactory - Allows only for creation of file.
IFileStorage - Allows for creation and removal of files.

Further information

For further information refer to eridu-tech/file-storage API docs.

Initial configuration​

FileStorage basics​

Creating a file object​

Writing buffered files​

Writing streamed files​

Retrieving files​

Checking file existence​

Removing files​

Retrieving file metadata​

Patterns​

Additional methods​

Copying files​

Moving files​

Signed urls and public urls.​

File instance variables​

Serialization and deserialization of file​

Separating file creation from manipulation​

Further information​

Initial configuration

FileStorage basics

Creating a file object

Writing buffered files

Writing streamed files

Retrieving files

Checking file existence

Removing files

Retrieving file metadata

Patterns

Additional methods

Copying files

Moving files

Signed urls and public urls.

File instance variables

Serialization and deserialization of file

Separating file creation from manipulation

Further information