Data Upload Operations

In addition to various types of analysis operations, Azure Data Expert provides flexible data upload operations allowing one to upload files to one or more Microsoft Azure cloud storage accounts. In order to perform a simple upload operation, press the 'Upload' button located on the main toolbar, select a storage account and a storage container to upload files to, select one or more files or directories to upload and press the 'Start' button.

Azure Data Expert Upload Data

Advanced users are provided with the ability to control the number of parallel upload streams, the performance mode of the upload operation, the network connect timeout and the network data transfer timeout. In addition, Azure Data Expert allows one to specify one or more subdirectories to be excluded from the upload process. In order to configure one or more exclude directories, open the advanced upload options dialog and select the 'Exclude' tab.

Azure Data Expert Upload Options

Finally, the user is provided with the ability to upload specific types of files or groups of files selected using one or more file matching rules. For example, the user can upload all types of images that where modified during the last month and exclude all other files from the upload process. The product provides a large number of different types of positive and negative file matching rules allowing one to precisely select which files to upload and which files to exclude from the upload process. In order to configure one or more file matching rules, open the advanced upload options dialog and select the 'Rules' tab.

Uploading Specific Types of Files

Azure DEX Pro provides the ability to upload specific types of files or groups of files by the file type, category, size, name, extension, last modification date, etc. The user is provided with the ability to configure a number of file matching rules for each file upload operation allowing one to upload the required files and precisely focus on the required data. For example, the user can upload all types of images with the file size more that 10 MB that were modified during the last month.

Azure Data Expert File Upload Rules

In order to configure one or more file matching rules for a file upload operation, open the file upload command dialog, press the 'Options' button, select the 'Rules' tab, press the 'Add' button and select the required rule type. During the file upload process, Azure Data Expert will evaluate the existing files using the specified file matching rules. Files matching the specified rules will be uploaded and files not matching the specified rules will be excluded from the file upload process.

Azure Data Expert Negative File Matching Rules

In addition to positive file matching rules allowing one to include specific files in the file upload process, Azure Data Expert provides negative file matching rules allowing one to exclude specific files from the file upload process. For example, in order to exclude all types of images from the file upload process, add a file category rule, select the 'Images, Pictures and Graphic Files' file category and select the 'Not Categorized' rule operator. During the file upload process, Azure Data Expert will evaluate the processed files and skip all types of image files.

Excluding Directories From the File Upload Process

Sometimes, it may be required to exclude one or more subdirectories from the file upload process. For example, if you need to upload files from a disk to a storage container excluding one or two special directories, you may specify the entire disk as an input directory and add the directories that should be skipped to the exclude list.

Azure Data Expert File Upload Exclude Directories

In order to add one or more directories to the exclude list, open the file upload command dialog, press the 'Options' button, select the 'Exclude' tab and press the 'Add' button. All files and subdirectories located in the specified exclude directory will be excluded from the file upload process. In addition, advanced users are provided with a number of exclude directories macro commands allowing one to exclude multiple directories using a single macro command.

Azure Data Expert provides the following exclude directories macro commands:

  • $BEGINS <Text String> - this macro command excludes all directories beginning with the specified text string.
  • $CONTAINS <Text String> - this macro command excludes all directories containing the specified text string.
  • $ENDS <Text String> - this macro command excludes directories ending with the specified text string.
  • $REGEX <Regular Expression> - this macro command excludes all directories matching the specified regular expression.

For example, the exclude macro command '$CONTAINS Temporary Files' will exclude all directories with 'Temporary Files' at any place in the full directory path and the exclude macro command '$REGEX \.(TMP|TEMP)$' will exclude all directories ending with '.TMP' or '.TEMP'.

Save File Upload Logs

Azure Data Expert allows one to save various types of file upload logs. In order to enable file upload logs, open the file upload command dialog, press the 'Advanced Options' button and select the 'Misc' tab. The user is provided with the ability to save file upload errors log files, file upload tasks log files and complete file upload actions log files, which may be very large when uploading many thousands of files.

Azure Data Expert Upload Files Save Logs

For all types of log files, the user needs to specify an existing directory where the selected log files will be saved with automatically generated file names. By default, Azure Data Expert will save a new log file for each upload operation, add the file upload operation date and time to the log file name and keep a history of the last 10 log files in the specified directory. In order to customize a log file, press an appropriate 'Advanced Log Options' button.

Azure Data Expert Upload Files Logs Options

The 'Advanced Log Options' dialog allows one to control the log files generation mode, the log files format and the number of history log files to keep in the specified log directory. In addition to separated file upload log files, which are saved for each file upload operation, the user can save consolidated hourly, daily and monthly log files. Also, the user is provided with the ability to save log files in the Excel CSV format, which can be used to analyze log files using the Microsoft Excel application.

Send File Upload Error Notifications

For mission-critical systems, it may be required to send E-Mail notifications about failed file upload operation and/or status notifications about successfully completed file upload operations. Azure DEX Ultimate provides the ability to enable error and/or status E-Mail notifications for any file upload operation. In order to enable E-Mail notifications for a file upload command, open the file upload command dialog, press the 'Advanced Options' button, select the 'Misc' tab, select an appropriate notification type and specify an E-Mail address to send notifications to.

Azure Data Expert Upload Files Send Notifications

In addition, open the main 'Options' dialog, select the 'E-Mail' tab and specify an E-Mail server, user name and password to use to send E-Mail notifications. Error E-Mail notifications are sent when a file upload command is failed or completed with one or more file upload errors. A status E-Mail notification always sent after each file upload operation is completed, disregarding of the status of the file upload operation. Each E-mail notification includes a summary table showing the total number of uploaded files, the number of file upload errors, additional statistical information about the file upload operation and a list of file upload errors.

Azure Data Expert File Upload Notifications Options

In order to customize an E-Mail notification, press an appropriate 'Notification Options' button. The 'Notification Options' dialog provides the ability to change the format of E-Mail notifications (HTML or text) and set a user-custom subject for E-Mail notifications. If a custom E-Mail subject is not set, Azure Data Expert will generate a default E-Mail subject, which will include the name and the status of the file upload command. If a custom E-Mail subject is set, the custom subject will be used 'as is' without any changes.