Advanced

Design

This reference implementation is based on the VOSpace 2.1 specification.

The design separates the metadata and data services into into 2 main HTTP(S) based processes:

  • The VOSpace metadata service; and
  • The VOSpace storage service.

The reference implementation provides a posix based VOSpace storage service out-of-the-box. The design allows for custom VOSpace storage services to be implemented by extending a basic set of interfaces. Please refer to the advanced section for details.

VOSpace Services

The VOSpace metadata service is designed to allow it to be a standalone process that implements the core metadata features of the VOSpace specification. These include the following web operations:

  • Service Metadata (Section 6.1)
  • Creating and manipulating data nodes (Section 6.2)
  • Accessing metadata (Section 6.3)

The transfer of data (Section 6.4) to and from a VOSpace node is the responsibility of the VOSpace storage process(es). The following transfer actions are supported:

  • pushToVoSpace; and
  • pullFromVoSpace.

Custom VOService

The base package implements a basic posix backend. If the developer wished to implement a custom storage backend follow the recipe below.

Note: that the storage classes in this packages only supports the HTTP and HTTP(S) protocols. Other protocols can be implemented by following the recipe found in pyvospace.server.storage.HTTPSpaceStorageServer

  1. Implement interface pyvospace.server.space.AbstractSpace
  2. To control permissions on the space extend pyvospace.server.space.SpaceServer and override pyvospace.server.space.SpacePermission.permits.

Note: The authentication and authorisation mechanisms are outside the scope of the implementation but it does implement the hooks to the aiohttp_security library which is can be leveraged. Refer to posix example for more details.

  1. Implement the storage interface pyvospace.server.storage.HTTPSpaceStorageServer

Note that when implementing the upload funtionality it’s important to first upload the data to a staging area then copy or move data to its final location in a single transaction. In this way the system can ensure consistency in a concurrent environment:

async def upload(self, job: StorageUWSJob, request: aiohttp.web.Request):

    # upload data to staging area first
    ...

    # copy data to final location and update node details in db in single transaction.
    # good practise to asyncio.shield save() and move() to ensure they are not aborted.

    async with job.transaction() as tr:
        node = tr.target # get the target node that is associated with the data
        node.size = size # set the size
        node.storage = self.storage # set the storage back end so it can be found
        await asyncio.shield(node.save()) # save details to db
        await asyncio.shield(move(stage_file_name, real_file_name)) # move file from staging to final location in the space in single transaction
  1. Create configuration file for the Space.

[Space]

Ref by pyvospace.server.space.SpaceServer

  • host: host of space metadata service.
  • port: port of space metadata service.
  • name: name of the space (unique).
  • uri: uri base for the space.
  • dsn: connection string to the database.
  • parameters: any customs parameters defined by the implementor of the space (json string)
  • use_ssl: use https (1: yes, 0: no)
  • cert_file: SSL certificate file.
  • key_file = SSL key file.

[Storage]

Ref by pyvospace.server.storage.HTTPSpaceStorageServer

  • host: host of space storage.
  • port: port of space storage.
  • name: name of the space (unique).
  • parameters: any customs parameters defined by the implementor of the space (json string)
  • use_ssl: use https (1: yes, 0: no)
  • cert_file: SSL certificate file.
  • key_file = SSL key file.

Configuration Example:

[Space]
host = localhost
port = 8080
name = posix
uri = icrar.org
dsn = postgres://vos_user:vos_user@localhost:5435/vospace
parameters = {}
use_ssl = 0

[Storage]
name = posix
host = localhost
port = 8081
parameters = {"root_dir": "/tmp/posix/storage/", "staging_dir": "/tmp/posix/staging/"}
use_ssl = 0
  1. Start each service.

VOSpace metadata service:

abstract_space = MySpecificImpl()
app = SpaceServer(<path to config>)
await app.setup(abstract_space)

VOSpace storage service:

app = MyHTTPSpaceStorageServer(<path to config>)
await app.setup()