BaseSpace™ CLI v1.0.0 is here!

By Swathi A. Ramani, Staff Product Manager – BaseSpace Sequence Hub

 If you’ve been using BaseSpace Sequence Hub for some time now, then you probably know that there is a lot more to the platform than the browser console. The Command Line Interface (CLI) is an easy-to-use command line tool that enables users to do more with BaseSpace via managing common (and not so common) tasks associated with their genomic data and analysis.

The CLI has been in development for over 4 years, and was created by our talented UK team. They needed automation tools to help sequence more than 20 petabases of data for the 100k Genomes Project. Over the past few months, we’ve been hard at work on the next generation of the CLI. We are thrilled to announce that our CLI is no longer in Beta! In our latest release, we have launched the officially supported BaseSpace (BS) Sequence Hub (BSSH) CLI v1.0.0, and all the exciting features that come with it. In the years since the initial release, we saw incredible product uptake and a lot of positive feedback from the BaseSpace community. With this launch announcement, we are delivering on some of your biggest requests for a robust feature set that simplifies data and analysis wrangling and process automation. This is a great foundation on which we can continue expanding our toolset.

Rich Built-in Features

BS CLI v1.0.0 is a completely different beast from its previous version. With just one file to download and configure, you can control multiple BaseSpace services and automate them through scripts, including uploading samples, downloading runs, launching or stopping apps and workflows, setting custom quality filters for your runs, launching analysis workflows, generate pre-signed URLs, and much more ! 

These include: 

  • Flexible install process: The CLI is installed by downloading a single binary with no additional dependencies, which enables you to install the CLI in an environment where you do not have administrator privileges.
  • Support for Linux, Mac and Windows (32 and 64 bit) operating systems
  • Rich options for listing details and filtering with customized output for seamless multi-command pipelines and scripts
  • Powerful data management features including creation, renaming and deletion of BSSH entities
  • Efficient upload of FASTQ datasets or any other file types, coupled with fast download of runs, projects, biosamples and datasets
  • Parameterize, launch, monitor and kill analyses running remotely in BSSH

Importantly, we’ve made sure the above features work nicely together so you don’t have to do the plumbing yourself. For a full list of worked examples visit our help site.

Try It Out Today! 

Our new BS CLI v1.0.0 is ready to serve as your standard toolchain to programmatically read, create and manipulate data in your BSSH account, automate routine tasks, as well as to efficiently manage your applications. You can try it out right now by following the instructions on our help site

If you are using existing tools like BaseMount or BaseSpace Copy, these will continue to work. However, as we continue to improve the developer experience, we hope to consolidate our existing tools and add new features to the BS CLI v1.0.0 toolchain. 

The more you use BS CLI v1.0.0, the more you will see how powerful it is. We can’t wait to see what you build with it! As always, let us know how we are doing. We want to incorporate best practices in the toolchain as much as possible, so it becomes customary, so please submit any requests in via this blog, twitter or techsupport@illumina.com. Happy hacking! 

  • The BaseSpace Sequence Hub Team

For Research Use only.

QB#8581

 

 

 

Introducing the ability to Launch Apps via the API!

We are happy to announce the introduction of API-driven App launch capabilities on BaseSpace. We are now extending the BaseSpace API to include app launch, so that with a call to a RESTful API endpoint, you can launch a BaseSpace app on your data. With the new feature in the API, you can now:

  • Configure and launch multiple Apps in succession or in parallel using a script, this ensures consistency of configuration and reduces the possibility of human error through manual launch
  • Automatically launch Apps in the background, without direct intervention by a human user. This allows the possibility of unattended sets of app launches, such as apps that launch in the middle of the night as soon as sequencing data is ready, rather than waiting for a manual launch
  • Create scripts to automate testing of your app for developers

Continue reading Introducing the ability to Launch Apps via the API!

Introducing the New BaseSpace HelpCenter

The team is proud to announce the release of our new help site at help.basespace.illumina.com.  The new HelpCenter makes it easier to navigate and find the relevant information about BaseSpace. It also enables greater user participation (see below for details).

image

image

Site Contents

  • Overview section for new users
  • Articles explaining the usage of different features
  • Release notes
  • Videos
  • Latest news
  • Search
  • Links to other BaseSpace related content like our Developer’s Site

Feedback

This is just the beginning of our content here and it will continue to be expanded and refined.  If you would like to suggest a topic to be covered or want us to clarify an existing article, all of the content is open source and hosted on github. You can create issues on current or new content and submit pull requests as well.

image

We still have direct access to support via our “Contact Us” widget within BaseSpace along with email and phone support through the normal Illumina support channels.

Thanks!

Deleting Samples and Projects

Our initial release of the delete feature provided users with the ability to delete Runs and Analyses, but we didn’t want to stop there.  Our users now have the ability to delete Samples and Projects from their accounts! Here’s how it works:

  • You can delete a Project from the list of all Projects in your account

Screen Shot 2014-12-19 at 3.14.55 AM

  • You can also remove a Project from its specific page

Screen Shot 2014-12-19 at 3.21.18 AM

  • Samples can be deleted from within a Project and multiple Samples can be selected at once

Screen Shot 2014-12-19 at 3.24.02 AM

  • Deleted items are first moved to the Trash, from there you may Restore items or Empty Trash

Screen Shot 2014-12-21 at 9.43.32 PM

  • If you delete a Project or Sample as a collaborator instead of an owner, the data will be un-shared with your account

For any questions, concerns, or feedback, please do not hesitate to contact us.  We are happy to help in any way that we can. Thanks!

Introducing BSFS, the BaseSpace File System

Today, together with the current release of BaseSpace, we would like to announce the release of a product that has gotten I and other developers on the BaseSpace team really excited and really busy over the past months: BaseSpace File System (abbreviated as BSFS or BaseSpace FS) – a feature that many of our developers on the BaseSpace platform have been asking for – is a way for you to directly mount your Samples and Appresults’ data residing in BaseSpace into your docker containers and access it on a strictly as-needed basis.

A range of improvements

This addition to the BaseSpace platform will bring in a great number of benefits, which I will go over now:

  • No pre-download of your Samples and AppResults

When running your apps on the BaseSpace Native App Engine with BSFS turned on, you will notice your applications executing right away upon launch. The usual pre-download step, which could take a good few hours on those very large NextSeq or HiSeq samples is now eliminated.

  • Less network data transfers

When an app executes on an input sample or app result, there is no guarantee that it will use the entire input dataset, up until today the entire input dataset had to be downloaded before any processing could happen – this is no longer the case. BSFS presents a virtual view of your data in the file system, and downloads only the data that is actually read from the files.

  • Overlap computation and network transfers

A typical data processing workflow is for an app to read data then process it in an iterative fashion. In order to make this process more efficient BSFS features a data pre-fetch mechanism: while the app is processing data at a certain location in a file, the data directly adjacent and following this location is downloaded automatically. This has the effect of mitigating issues in download speed due to network latency.

  • An Improved workflow for developers

One of the major areas of focus of the BaseSpace platform team has been to provide developers with an awesome experience, and adding BSFS to the platform will make things even more awesome!
Soon after this release we will be providing a public Amazon Machine Image (AMI) which is the same one we are using in production today. This image contains all that’s required to get started coding in BaseSpace together with BSFS. This is a huge improvement of the developer workflow as it will provide an environment that is readily usable and in which you can simply drop apps in a docker container and see them interact with your BaseSpace data within minutes of getting started!
Finally, with the download step eliminated, there is nothing left to get in the way of a highly iterative development process, where developers can work directly with their BaseSpace data.

  • New and existing apps

All new apps created in the BaseSpace developer portal will now have BSFS turned on by default. Also, we have made sure that existing apps can benefit fully from this new addition, hence if you have been following developer guidelines and conventions (ie. the /data/input drive should not be written to), enabling BSFS in your existing app should be as easy as flicking the switch.

Upon creation of a new application in the developer portal, you will notice a slightly modified launch spec callback, with a new Options array that is used to turn on bsfs:

function launchSpec(dataProvider)
{
    var ret = {
        commandLine: [ "cat", "/illumina.txt" ],
        containerImageId: "basespace/demo",
        Options: [ "bsfs.enabled=true" ]
    };
    return ret;
}

You will want to use a callback function with this new Options array, in order to enable BSFS in your existing app.

Also, as of today the 16S Metagenomics v1.0 app is running with BaseSpace FS switched on to reap all the performance benefits. In the coming weeks, we will turn on BSFS for the rest of the BaseSpace core apps.

  • Real world performance improvements

The kinds of speed-ups we are seeing on these apps are only scratching the surface for the potential speed-ups we can get. On large samples processed on a single node the performance benefits are less pronounced since the download time is dwarfed by the compute time, however multi-node applications that access part of an input sample or app result will benefit greatly as the download portion is always a major contributor to the overall execution time.

With that, I hope you will share my excitement with this announcement, and that BSFS will make your development process even more awesome in BaseSpace.

Links to more resources

BaseSpace Developer portal
BaseSpace FileSystem Developer Documentation
Using BSFS

Taking Out the Trash

If you’ve been using BaseSpace for a while you may have noticed that there wasn’t a way to permanently remove data from your account.  I say that in the past tense because it is no longer true.    The wait is over! “Move to Trash” is now available on Runs and Analyses.

MoveToTrash02

Trash Overview

This has been one of the most important features for us to get right because it has to do with removing your data and we take that very seriously.  That is why we are introducing a two-step delete process that will help prevent accidental deletes and give you the confidence you need to safely manage your data.

First, you will notice a new action available on run and analysis list and detail pages, called “Move to Trash”. On the list pages, you must first highlight the row that you want before it’s available.

MoveToTrash

This action is very similar to moving files on your desktop to the trash or recycle bin. Just like your desktop, the data can be recovered, but it can no longer be viewed or acted upon.

Trashed Items Side-Effects:

  • If the items were shared, all share recipients will lose access to that data
  • All API access is immediately removed and will return the HTTP status code of 410 (“Gone”)
  • Any attempt to view this data on the website will take the user to an 410 error page stating the content is “Gone”
  • Data, while in the trash, can only be  “Restored” or “Emptied” by the owner.
  • Purging data will cause it to be permanently removed and cannot be undone.

Moving Runs to the Trash

  • Runs can be put in the trash from the list or the detail pages.
  • Runs cannot be removed if they are in a non-terminal state.  The most common non-terminal states would be: running, uploading, analyzing.
  • The dialog may also present you with the option to remove all associated analyses that used the run as input.
    • All sequencing runs will have at least 1 associated analysis unless they were failed or used just for remote monitoring.
  • If you are not the owner of the run, moving this item to the trash will simply remove your access and cannot be undone.
    •   To restore access, just contact the owner or click on the previously sent share link if it’s still active.

movetotrash-animation

Moving Analyses to the Trash

  • Analyses can be put in the trash from the list and detail pages.
  • Analyses cannot be removed if in a non-terminal state. The most common non-terminal states would be: pending execution and running.
  • If a project is being transferred, some of the analyses may not be removed until after the transfer has been completed.
  • Apps that are leveraging data as input may fail if items are moved to the trash.
  • If you have items in the trash, we prevent project transfers until all items in that project are restored or emptied.

Emptying and Restoring Items in the Trash

The trash page can be accessed from most of the project and run list pages.  The icon is always in the right side of the grid and labeled, “View Trash”.

TrashIcon

There are only two actions currently on the Trash page: Empty and Restore.

Empty will permanently delete all items, and Restore allows you to return the items back to being active.

Restored items will keep all of their original attributes except for the share recipients.

restore

User Agreement Updates

Because of all of these changes, we have also updated our User agreements to reflect the behavior of these new features. In particular, item 7 states that even though data can be removed it may have been previously shared with other users or apps and subsequently downloaded or copied.   You will be prompted to accept these new terms upon your next login.  If you have any questions, don’t hesitate to ask!

Thank you,

-Greg