Creating a New Pipeline Source

Back to Main Help

A Pipeline Source, or Source, maps data from an external source to the Input Columns of a Template. It stores one Column Mapping for each Input Column.

When a Dataset is loaded (Load Data), a Source is selected, and the associated Template defines the Dataset. More than one Source may be defined for a single Template to allow multiple external data sources to load data into a single dataset.

Six Source Types are available to load your data. When Load Data is executed, the user is prompted for the appropriate data as specified below:

      Spreadsheet Source

o   Loads dataset from spreadsheet file.

o   Each Column Mapping maps one spreadsheet column (A, B, etc.) to one Input Column.

o   When Load is executed: User selects .xslx or .csv file.

      Text Source

o   Loads dataset from any number of .docx, .pdf, or .txt files.

o   Each Column Mapping maps a file property (name, size, last-modified, etc.) or file contents to an Input Column.

o   When Load is executed: User selects .zip file.

      XML Source

o   Loads dataset from an .xml file

o   Each Column Mapping maps an xml tag to an Input Column.

o   When Load is executed: User selects .xml file.

      SQL Source

o   Loads dataset from an SQL database

o   Each Column Mapping maps an SQL field to an Input Column.

o   SQL credentials are stored in Pipeline Source

o   When Load is executed: User enters SQL query.

      Twitter Source

o   Loads dataset from a Twitter search

o   Each Column Mapping maps one of these Tweet fields to an Input Column:

  Text, Source, ID, Created At, Retweet Count, Status Count, Favorite Count, User Name, Screen Name, Followers Count

o   Twitter account credentials stored in Pipeline Source

o   When Load is executed: User is prompted for Twitter search terms.

      E-mail Source

o   Loads dataset from an e-mail account

o   Each Column Mapping maps one of these E-mail fields to an Input Column:

  Subject, To, From, Date, ID, Body

o   E-mail account credentials are stored in Source

o   When Load is executed: User is prompted for E-mail Search.

Click Pipeline > Pipeline Sources. From this page you can view/edit Pipeline Source properties and add Mappings.

To define a new Pipeline Source

Click Create Source and fill in the following:

      Name for the Pipeline Source

      Description

      Template - Contains the Input Field specs

      Source Type One of the Types described above

After clicking GO, you will see your new Pipeline Source. Click Add Column Mapping to define a Column Mapping for each Input Column.

Next: Adding Data to an Existing Dataset