Archive | Release Notes RSS for this section

New Download Experience

With our newest release today, we are happy to introduce a revamped download experience from our run details and analysis details page.  This new experience takes advantage of the existing BaseSpace Downloader, with a couple new filtering options.

Read More…

Introducing the ability to Launch Apps via the API!

We are happy to announce the introduction of API-driven App launch capabilities on BaseSpace. We are now extending the BaseSpace API to include app launch, so that with a call to a RESTful API endpoint, you can launch a BaseSpace app on your data. With the new feature in the API, you can now:

  • Configure and launch multiple Apps in succession or in parallel using a script, this ensures consistency of configuration and reduces the possibility of human error through manual launch
  • Automatically launch Apps in the background, without direct intervention by a human user. This allows the possibility of unattended sets of app launches, such as apps that launch in the middle of the night as soon as sequencing data is ready, rather than waiting for a manual launch
  • Create scripts to automate testing of your app for developers

Read More…

Introducing the New BaseSpace HelpCenter

The team is proud to announce the release of our new help site at help.basespace.illumina.com.  The new HelpCenter makes it easier to navigate and find the relevant information about BaseSpace. It also enables greater user participation (see below for details).

image

image

Site Contents

  • Overview section for new users
  • Articles explaining the usage of different features
  • Release notes
  • Videos
  • Latest news
  • Search
  • Links to other BaseSpace related content like our Developer’s Site

Feedback

This is just the beginning of our content here and it will continue to be expanded and refined.  If you would like to suggest a topic to be covered or want us to clarify an existing article, all of the content is open source and hosted on github. You can create issues on current or new content and submit pull requests as well.

image

We still have direct access to support via our “Contact Us” widget within BaseSpace along with email and phone support through the normal Illumina support channels.

Thanks!

Deleting Samples and Projects

Our initial release of the delete feature provided users with the ability to delete Runs and Analyses, but we didn’t want to stop there.  Our users now have the ability to delete Samples and Projects from their accounts! Here’s how it works:

  • You can delete a Project from the list of all Projects in your account

Screen Shot 2014-12-19 at 3.14.55 AM

  • You can also remove a Project from its specific page

Screen Shot 2014-12-19 at 3.21.18 AM

  • Samples can be deleted from within a Project and multiple Samples can be selected at once

Screen Shot 2014-12-19 at 3.24.02 AM

  • Deleted items are first moved to the Trash, from there you may Restore items or Empty Trash

Screen Shot 2014-12-21 at 9.43.32 PM

  • If you delete a Project or Sample as a collaborator instead of an owner, the data will be un-shared with your account

For any questions, concerns, or feedback, please do not hesitate to contact us.  We are happy to help in any way that we can. Thanks!

Introducing BSFS, the BaseSpace File System

Today, together with the current release of BaseSpace, we would like to announce the release of a product that has gotten I and other developers on the BaseSpace team really excited and really busy over the past months: BaseSpace File System (abbreviated as BSFS or BaseSpace FS) – a feature that many of our developers on the BaseSpace platform have been asking for – is a way for you to directly mount your Samples and Appresults’ data residing in BaseSpace into your docker containers and access it on a strictly as-needed basis.

A range of improvements

This addition to the BaseSpace platform will bring in a great number of benefits, which I will go over now:

  • No pre-download of your Samples and AppResults

When running your apps on the BaseSpace Native App Engine with BSFS turned on, you will notice your applications executing right away upon launch. The usual pre-download step, which could take a good few hours on those very large NextSeq or HiSeq samples is now eliminated.

  • Less network data transfers

When an app executes on an input sample or app result, there is no guarantee that it will use the entire input dataset, up until today the entire input dataset had to be downloaded before any processing could happen – this is no longer the case. BSFS presents a virtual view of your data in the file system, and downloads only the data that is actually read from the files.

  • Overlap computation and network transfers

A typical data processing workflow is for an app to read data then process it in an iterative fashion. In order to make this process more efficient BSFS features a data pre-fetch mechanism: while the app is processing data at a certain location in a file, the data directly adjacent and following this location is downloaded automatically. This has the effect of mitigating issues in download speed due to network latency.

  • An Improved workflow for developers

One of the major areas of focus of the BaseSpace platform team has been to provide developers with an awesome experience, and adding BSFS to the platform will make things even more awesome!
Soon after this release we will be providing a public Amazon Machine Image (AMI) which is the same one we are using in production today. This image contains all that’s required to get started coding in BaseSpace together with BSFS. This is a huge improvement of the developer workflow as it will provide an environment that is readily usable and in which you can simply drop apps in a docker container and see them interact with your BaseSpace data within minutes of getting started!
Finally, with the download step eliminated, there is nothing left to get in the way of a highly iterative development process, where developers can work directly with their BaseSpace data.

  • New and existing apps

All new apps created in the BaseSpace developer portal will now have BSFS turned on by default. Also, we have made sure that existing apps can benefit fully from this new addition, hence if you have been following developer guidelines and conventions (ie. the /data/input drive should not be written to), enabling BSFS in your existing app should be as easy as flicking the switch.

Upon creation of a new application in the developer portal, you will notice a slightly modified launch spec callback, with a new Options array that is used to turn on bsfs:

function launchSpec(dataProvider)
{
    var ret = {
        commandLine: [ "cat", "/illumina.txt" ],
        containerImageId: "basespace/demo",
        Options: [ "bsfs.enabled=true" ]
    };
    return ret;
}

You will want to use a callback function with this new Options array, in order to enable BSFS in your existing app.

Also, as of today the 16S Metagenomics v1.0 app is running with BaseSpace FS switched on to reap all the performance benefits. In the coming weeks, we will turn on BSFS for the rest of the BaseSpace core apps.

  • Real world performance improvements

The kinds of speed-ups we are seeing on these apps are only scratching the surface for the potential speed-ups we can get. On large samples processed on a single node the performance benefits are less pronounced since the download time is dwarfed by the compute time, however multi-node applications that access part of an input sample or app result will benefit greatly as the download portion is always a major contributor to the overall execution time.

With that, I hope you will share my excitement with this announcement, and that BSFS will make your development process even more awesome in BaseSpace.

Links to more resources

BaseSpace Developer portal
BaseSpace FileSystem Developer Documentation
Using BSFS

Taking Out the Trash

If you’ve been using BaseSpace for a while you may have noticed that there wasn’t a way to permanently remove data from your account.  I say that in the past tense because it is no longer true.    The wait is over! “Move to Trash” is now available on Runs and Analyses.

MoveToTrash02

Trash Overview

This has been one of the most important features for us to get right because it has to do with removing your data and we take that very seriously.  That is why we are introducing a two-step delete process that will help prevent accidental deletes and give you the confidence you need to safely manage your data.

First, you will notice a new action available on run and analysis list and detail pages, called “Move to Trash”. On the list pages, you must first highlight the row that you want before it’s available.

MoveToTrash

This action is very similar to moving files on your desktop to the trash or recycle bin. Just like your desktop, the data can be recovered, but it can no longer be viewed or acted upon.

Trashed Items Side-Effects:

  • If the items were shared, all share recipients will lose access to that data
  • All API access is immediately removed and will return the HTTP status code of 410 (“Gone”)
  • Any attempt to view this data on the website will take the user to an 410 error page stating the content is “Gone”
  • Data, while in the trash, can only be  “Restored” or “Emptied” by the owner.
  • Purging data will cause it to be permanently removed and cannot be undone.

Moving Runs to the Trash

  • Runs can be put in the trash from the list or the detail pages.
  • Runs cannot be removed if they are in a non-terminal state.  The most common non-terminal states would be: running, uploading, analyzing.
  • The dialog may also present you with the option to remove all associated analyses that used the run as input.
    • All sequencing runs will have at least 1 associated analysis unless they were failed or used just for remote monitoring.
  • If you are not the owner of the run, moving this item to the trash will simply remove your access and cannot be undone.
    •   To restore access, just contact the owner or click on the previously sent share link if it’s still active.

movetotrash-animation

Moving Analyses to the Trash

  • Analyses can be put in the trash from the list and detail pages.
  • Analyses cannot be removed if in a non-terminal state. The most common non-terminal states would be: pending execution and running.
  • If a project is being transferred, some of the analyses may not be removed until after the transfer has been completed.
  • Apps that are leveraging data as input may fail if items are moved to the trash.
  • If you have items in the trash, we prevent project transfers until all items in that project are restored or emptied.

Emptying and Restoring Items in the Trash

The trash page can be accessed from most of the project and run list pages.  The icon is always in the right side of the grid and labeled, “View Trash”.

TrashIcon

There are only two actions currently on the Trash page: Empty and Restore.

Empty will permanently delete all items, and Restore allows you to return the items back to being active.

Restored items will keep all of their original attributes except for the share recipients.

restore

User Agreement Updates

Because of all of these changes, we have also updated our User agreements to reflect the behavior of these new features. In particular, item 7 states that even though data can be removed it may have been previously shared with other users or apps and subsequently downloaded or copied.   You will be prompted to accept these new terms upon your next login.  If you have any questions, don’t hesitate to ask!

Thank you,

-Greg

FASTQ upload is now available in BaseSpace

We are excited to announce the availability of a data upload feature for FASTQ files that were previously generated on Illumina sequencing instruments. This simple-to-use feature is accessible from any project to which the user has write access by first clicking on the project and then selecting the Import tab shown below.

ProjectTab

The user will then be prompted to select their import type. The user can upload a single sample by clicking on “Sample” as shown below.

Samples

The user can then either “Drag and drop” one or more files into the webpage or click on “select files” and select which files they would like to upload from a file browser. Note that the FASTQ files need to adhere to Illumina standards, as specified below.  Data for a single sample can constitute multiple files. The total number of files per sample and their combined size are limited to 16 and 25 GB respectively. It will take 1-2 hours to upload a 25GB sample on a network with a relatively fast internet connection.

dranganddrop

The user will then see a progress bar as the file/s are uploaded. Once the progress bar completes, the user can add additional files. The user can also set the sample name and associate a genome with the sample in the upper left hand corner of the screen.

upload_screen

Once the user has imported all of the files and the files complete uploading, the user will need to click on the  “Complete Import” button (shown above) to complete the session.

FASTQ file standards

  • The uploader will only support gzipped FASTQ files generated on Illumina instruments
  • The name of the FASTQ files must conform the following convention:
    • SampleName_SampleNumber_Lane_Read_FlowCellIndex.fastq.gz (i.e. SampleName_S1_L001_R1_001.fastq.gz / SampleName_S1_L001_R2_001.fastq.gz)
  • The read descriptor in the FASTQ files must conform to the following convention:
    • @Instrument:RunID:FlowCellID:Lane:Tile:X:Y ReadNum:FilterFlag:0:SampleNumber:
      • Read 1 descriptor would look like this:
        @M00900:62:000000000-A2CYG:1:1101:18016:2491 1:N:0:13
      • Read 2 would have a 2 in the ReadNum field, like this:
        @M00900:62:000000000-A2CYG:1:1101:18016:2491 2:N:0:13

Quality considerations

  • The number of base calls for each read must equal the number of quality scores
  • The number of entries for Read 1 must equal the number of entries for Read 2
  • The uploader will determine if files are paired-end based on the matching file names in which the only difference is the ReadNum
  • For paired-end reads, the descriptor must match for every entry for both reads 1 and 2
  • Each read has passed filter

Upload parameters

  • Only one sample can be uploaded at a time
  • A maximum of 16 files can be uploaded in a session
  • The size of the uploaded files cannot exceed 25 GB
  • A detailed description of how to use the uploader can be found in the BaseSpace user guide