Data Download Operations

Azure Data Expert provides advanced download operations allowing one to download data blobs from one or more Microsoft Azure cloud storage accounts. In order to perform a simple download operation, select one or more containers, directories or data blobs in the storage navigator view, press the 'Download' button located on the main toolbar, select a destination directory to download files to and press the 'Start' button.

Azure Data Expert Download Data

In addition, the user is provided with the ability to control the number of parallel download streams, the performance mode of the download operation, the network connect timeout and the network data transfer timeout. Azure Data Expert allows one to specify one or more subdirectories to be excluded from the download process. In order to configure one or more exclude directories, open the advanced download options dialog and select the 'Exclude' tab.

Azure Data Expert Download Options

Finally, advanced users are provided with the ability to download specific types of files or groups of files selected using one or more file matching rules. For example, the user can download all types of images than where modified during the last month and exclude all other files from the download process. The product provides a large number of different types of positive and negative file matching rules allowing one to precisely select which files to download and which files to exclude from the download process. In order to configure one or more file matching rules, open the advanced download options dialog and select the 'Rules' tab.

Downloading Specific Types of Files

Azure DEX Pro provides the ability to download specific types of files or groups of files by the file type, category, size, name, extension, last modification date, etc. The user is provided with the ability to configure a number of file matching rules for each file download operation allowing one to download the required files and precisely focus on the required data. For example, the user can download all types of images with the file size more that 10 MB that were modified during the last month.

Azure Data Expert File Download Rules

In order to configure one or more file matching rules for a file download operation, open the file download command dialog, press the 'Options' button, select the 'Rules' tab, press the 'Add' button and select the required rule type. During the file download process, Azure Data Expert will evaluate the existing files using the specified file matching rules. Files matching the specified rules will be downloaded and files not matching the specified rules will be excluded from the file download process.

Azure Data Expert Negative File Matching Rules

In addition to positive file matching rules allowing one to include specific files in the file download process, Azure Data Expert provides negative file matching rules allowing one to exclude specific files from the file download process. For example, in order to exclude all types of images from the file download process, add a file category rule, select the 'Images, Pictures and Graphic Files' file category and select the 'Not Categorized' rule operator. During the file download process, Azure Data Expert will evaluate the processed files and skip all types of image files.

Excluding Directories From the File Download Process

Sometimes, it may be required to exclude one or more subdirectories from the file download process. For example, if you need to download files from a storage container excluding one or two special directories, you may specify the entire storage container to be downloaded and add the directories that should be skipped to the exclude list.

Azure Data Expert File Download Exclude Directories

In order to add one or more directories to the exclude list, open the file download command dialog, press the 'Options' button, select the 'Exclude' tab and press the 'Add' button. All files and subdirectories located in the specified exclude directory will be excluded from the file download process. In addition, advanced users are provided with a number of exclude directories macro commands allowing one to exclude multiple directories using a single macro command.

Azure Data Expert provides the following exclude directories macro commands:

  • $BEGINS <Text String> - this macro command excludes all directories beginning with the specified text string.
  • $CONTAINS <Text String> - this macro command excludes all directories containing the specified text string.
  • $ENDS <Text String> - this macro command excludes directories ending with the specified text string.
  • $REGEX <Regular Expression> - this macro command excludes all directories matching the specified regular expression.

For example, the exclude macro command '$CONTAINS Temporary Files' will exclude all directories with 'Temporary Files' at any place in the full directory path and the exclude macro command '$REGEX \.(TMP|TEMP)$' will exclude all directories ending with '.TMP' or '.TEMP'.

Save File Download Logs

Azure Data Expert allows one to save various types of file download logs. In order to enable file download logs, open the file download command dialog, press the 'Advanced Options' button and select the 'Misc' tab. The user is provided with the ability to save file download errors log files, file download tasks log files and complete file download actions log files, which may be very large when downloading many thousands of files.

Azure Data Expert Download Files Save Logs

For all types of log files, the user needs to specify an existing directory where the selected log files will be saved with automatically generated file names. By default, Azure Data Expert will save a new log file for each download operation, add the file download operation date and time to the log file name and keep a history of the last 10 log files in the specified directory. In order to customize a log file, press an appropriate 'Advanced Log Options' button.

Azure Data Expert Download Files Logs Options

The 'Advanced Log Options' dialog allows one to control the log files generation mode, the log files format and the number of history log files to keep in the specified log directory. In addition to separated file download log files, which are saved for each file download operation, the user can save consolidated hourly, daily and monthly log files. Also, the user is provided with the ability to save log files in the Excel CSV format, which can be used to analyze log files using the Microsoft Excel application.

Send File Download Error Notifications

For mission-critical systems, it may be required to send E-Mail notifications about failed file download operation and/or status notifications about successfully completed file download operations. Azure DEX Ultimate provides the ability to enable error and/or status E-Mail notifications for any file download operation. In order to enable E-Mail notifications for a file download command, open the file download command dialog, press the 'Advanced Options' button, select the 'Misc' tab, select an appropriate notification type and specify an E-Mail address to send notifications to.

Azure Data Expert Download Files Send Notifications

In addition, open the main 'Options' dialog, select the 'E-Mail' tab and specify an E-Mail server, user name and password to use to send E-Mail notifications. Error E-Mail notifications are sent when a file download command is failed or completed with one or more file download errors. A status E-Mail notification always sent after each file download operation is completed, disregarding of the status of the file download operation. Each E-mail notification includes a summary table showing the total number of downloaded files, the number of file download errors, additional statistical information about the file download operation and a list of file download errors.

Azure Data Expert File Download Notifications Options

In order to customize an E-Mail notification, press an appropriate 'Notification Options' button. The 'Notification Options' dialog provides the ability to change the format of E-Mail notifications (HTML or text) and set a user-custom subject for E-Mail notifications. If a custom E-Mail subject is not set, Azure Data Expert will generate a default E-Mail subject, which will include the name and the status of the file download command. If a custom E-Mail subject is set, the custom subject will be used 'as is' without any changes.