Data Importer

Important: It is the responsibility of the association – as the data controller – to ensure that your organization has permission to utilize the data you are importing. Before doing any import, the association staff member running the process will need to confirm that ‘By proceeding with this import, you are agreeing that you have acquired this data from a reputable source and that all individuals contained within this data have given consent for their data to be shared with and used by your organization, or that there are other lawful grounds for your processing of their data. MatrixMaxx and Matrix Group hold no responsibility for any privacy or security issues which may come about as a result of this data import.‘

The original importer allowed staff to import basic Individual information:

• Firstname
• Lastname
• Title
• Email*

And starting with MatrixMaxx 20.4 (December 2020), the importer also matches IndividualId and matches and/or imports basic Organization/Company along with the Individual information:

• CompanyId
• CompanyImportId
• CompanyName

* NOTE: Email is still the only field that is required for the importer to run; if a row is missing email, the row will error in most cases. The only exception is if a row matches an existing individual based on IndividualId, which is our very first check. 

You can see the core importer (firstname, lastname, title, email) in action on the recording of the Overview Session for the 19.4 release, December 18, 2020, starting at the 4 minute timestamp

How to access the Importer

Access the Importer under Contacts -> Importer

Basic Rules

• Anyone with the highest ‘SiteAdmin’ access in MatrixMaxx can access the importer, (Your association may request broader or custom access rights)
• Maxx creates an individual list using the name of the import you run
• Maxx adds all individual persons to this import list
  • If we can identify a duplicate that exists in Maxx, we do not import the record that matches the import; instead, Maxx just adds the existing record from Maxx to the newly-created list
  • If the record seems new and unique, or there is more than one possible duplicate in Maxx, we create a new individual record in Maxx, and add this record to the newly-created list

Step 1: Read Instructions and Download the Most Recent Import Template

Start by reading this support article in its entirety, then the current import template may be downloaded here: ImportTemplate_ChangeThisNameAndResaveRevised2.xlsx

Step 2: Prepare Your Import File (.xlsx)

It is imperative that your import spreadsheet file has the following critical attributes (the template has these; please be sure that you do not remove or change them):

• .xlsx extension: only files with an .xlsx extension will work with the importer
• IndividualData workbook tab: the .xlsx file that you upload MUST have a data tab named with the label “IndividualData” (a blank tab name will error)
• Header Row: the file that you upload must have a header row (i.e., the first row must not contain data but rather column names)
• Header Row Fields: the header row currently must have ONLY these fields, and they must be spelled as:
  • IndividualId
  • FirstName
  • LastName
  • Title
  • Email
  • CompanyId
  • CompanyImportId
  • CompanyName
• No special characters: please use care in including data to be imported; please remove any special or scientific characters from the data file, as these will cause the import to fail
  • TIP:  first running your import in your MatrixMaxx DEV environment will help you identify any potential data issues before you run the import on against your live database. You may request the MatrixMaxx team to refresh your data on the development server to facilitate/support your testing.

Step 3: Test The Import on the Development (DEV) Server

Not sure of your DEV url? You can find it on your MatrixMaxx Intranet at Help -> Association-specific Guides and Links

Navigate to the Importer 

• You will access it under Contacts —>  Importer
• Note: generically, only the SiteAdmin permission level has access to the importer (If your organization has custom access levels, you may need to contact Matrix to grant these access to Importer)

Name the Import (and the associated list!)

The import name will be used to identify the import and also to create the name of the list that all imported users will be added to. So name it well!

• Give the import a unique name
• We suggest potentially adding a date to the import name, too, for your own reference
•  An import name could be “Potential Student Members From Partner Company 12June2018”
• NOTE: if you used the name above, the list that is created will be named “IMPORT_Potential_Student_Members_From_Partner_Company_12June2018”
• The name given to the import will be added to the “Import ID” field for any new individuals added to MatrixMaxx. Note: names that are very long may wrap on the individual view screen.

Select the File to Upload and Import

• Be sure your .xlsx file meets all of the key setup criteria
• Browse to locate the file you intend to import to the list you just named

Run the Import 

• Click ‘save/continue’ to start the import
• You will be asked to confirm that you have verified the data source as legitimate, and then you can proceed.
• As the import runs, you will be directed to the Requested Reports page
• Please note that the percentage done will ALWAYS say 0% – this is because the import is being run via a script behind-the-scenes, and the UI is not privy to the percent completed
• The import will eventually complete, and the 0% will change to “DONE”; the link will be accessible
• Click the link to see the results of your import!

Running the import on DEV will help identify bad/invalid data or file setup

Running the import on DEV will allow you to identify any potential problems you might have with bad characters or columns in your import file.  This will allow you to fix them before attempting the actual import on the live server.

• It is imperative that you first test your import on the dev site
• Imports will potentially be adding tens, if not hundreds or thousands, of records to your database
• You want to ensure that your file is well-formed and ready for import

Running the import on DEV will NOT inform you of how many potential duplicates you may have

The duplicate-check results will be different/erroneous on your DEV site as opposed to your live production site because Matrix strips the real email addresses when we copy down live data to DEV. (We do this to ensure that no real users are accidentally emailed when various tests are performed on the DEV site.) As the importer identifies duplicates based on email matching — and all DEV emails are test@maxx.matrixgroup.net — duplicate results will be wrong.

TIP! If you’d like to see exactly what the results will be, you can contact Matrix to do a special refresh of your DEV server that keeps emails in tact. This is a very small cost – less than an hour – and could give you valuable insight!

Step 4: Run The Import on the LIVE Production Server

Once you have validated your data on DEV, it is time to go to the live production server. You will follow the same steps as on DEV, but …

• emails are REAL on the live server
• you only have 24 hours to realize a mistake and re-run the import with corrections

CRITICAL LOGIC

Please note these important importer data rules and behaviours:

• either IndividualId or Email MUST be filled in
• if the importer record contains ONLY a blank firstname, but there IS a lastname, the importer leaves the firstname field blank
• if the importer record contains ONLY a blank lastname, but there IS a firstname, the importer leaves the lastname field blank
• if BOTH firstname AND lastname are blank/null, then the importer allows for this, but it will set the firstname field as ‘ImportedEmailAddress’ and lastname field as ‘ImportLastName’

TIP! To get the best possible results with your import, if you allow dup emails in your database, first check the’ Individuals with the same Email’ Report and eliminate as many dups as possible!

Duplicate Handling Rules for Files with Multiple Data Values

Running live on the production server, with real data, will bring the dup-check logic into play.

The first check is on IndividualId.

• If the IndividualID of a row in the Import file matches the IndividualID of an existing record in MatrixMaxx, then no new individual record is created, and the existing/matched individual is added to the new list
• If the IndividualID of a row in the Import file does NOT match the IndividualID of an existing record in MatrixMaxx, then no new individual record is created, and the row is flagged as an error
• TIP: This functionality could be useful in a case of simple SSO with a 3rd party system, where the association wants to ‘write back’ info to MatrixMaxx. e.g., The association does SSO with a 3rd party LMS, but decides not to spend the budget to use the API to write back completion data. Once a year, the association could ‘import’ the IDs of individuals who completed certain course(s) into a list and then potentially also stamp an activity to that list.

Then, the Importer considers email address and name.  A record is considered a duplicate if email, firstname, AND lastname match EXACTLY on a record in MatrixMaxx

• if email, firstname, AND lastname exactly match only one record in Maxx, then no new individual record is created, and the existing/matched individual is added to the new list
• if email, firstname, AND lastname exactly match more than one record in Maxx, then Maxx creates a new individual and adds that new individual to the new list (in this situation we recommend you check the ‘Individuals with the same Email Address’ report to review these dups and potentially merge some/all of them) Note! If you have MatrixMaxx set to enforce unique email addresses, and you try to import more than one individual with the same email (i.e., multiple record rows have the same email), it will cause the importer will give an ‘error’ rather than create a new record
• if email, firstname, AND lastname do not match any existing record, then Maxx creates a new individual and adds that new individual to the new list

Duplicate Handling Rules for Files with Email ONLY

• the following de-duplication logic is in place for imports that contain values ONLY in the email address field
• other imports that have values in any other field will be subject to the above set of de-duplication rules
• if the import contains blank firstname and blank lastname, but email address DOES exists, the importer with dedup the data based just on email address:
• EMAIL ONLY Duplicate Type 1: if email matches only one record in Maxx, then no new individual record is created, and the existing/matched individual is added to the new list
• EMAIL ONLY Duplicate Type 2: if email exactly matches more than one record in Maxx, then Maxx creates a new individual and adds that new individual to the new list (in this situation we recommend you check the ‘Individuals with the same Email Address’ report to review these dups and potentially merge some/all of them)”
• If the Email matches no existing emails, a new individual record will be created.
• Note! If you have MatrixMaxx set to enforce unique email addresses, and you try to import more than one individual with the same email (i.e., multiple record rows have the same email), it will cause the importer will give an ‘unknown error’ rather than create a new record

Company Affiliations

Starting with the 20.4 MatrixMaxx release in Dec. 2020, the Importer can also process several company-related fields, for the purpose of affiliating the newly-created individuals to a company/organization (either by matching to an existing company or creating a new company)

Individual Dup check is done FIRST
– if individual already exists, we add them to the import list, and stop there
– if individual does NOT already exist (or we can’t determine if they do), and the Importer is preparing to add them as a new individual, then we start to consider the company …

CompanyId

• CompanyId is unique per company; it is the ID found in the URL of the intranet Organization View Page and it is also available in most of our standard exports
• If we can make exact match to CompanyId we will add the newly created individual to that company

CompanyImportId

• This is the ImportId from the original data source that came into MatrixMaxx (this ID is typically visible on the Intranet Organization View page, under the Membership Status accordion section)
• If we can make exact match to CompanyImportId we will add the newly created individual to that company

COMING SOON: Match Email to ‘Approved Domain’ values in the database under Organization/Company Demographics

• Some MatrixMaxx implementations include an ‘Approved Domains’ Company Demographic field which is used in conjunction with validating new WWW profiles based on email domain
• We are planning to expand this field to all MatrixMaxx implementations and tie it in with the Importer. 
• Example: You are importing an individual email jsmith@starships.com and have no other data on this row, but in MatrixMaxx you have one company – StarShips,  Inc. – with starships.com in their ‘approved domain’ field, so we will match/add jsmith to this company

CompanyName

Our last attempt to match an existing company will be to look for an EXACT NAME MATCH in the database. i.e., If ‘ACME’ is in the CompanyName field and there is one ‘ACME’ in the MatrixMaxx database, we’ll consider it a match.  So we (hopefully) don’t make mistakes, we’ll be looking for nearly an exact string match, meaning …

• we’ll ignore case (e.g., ACME will match Acme will match acMe)
• we will NOT ignore spaces (e.g., Widget Maker Inc. will NOT match WidgetMaker Inc.)
• we will NOT ignore special characters (e.g., Smith&Fields will not match Smith Fields or Smith and Fields or Smith & Fields)
• we will NOT translate abbreviations or symbols (i.e., ACME Inc. will not match ACME Incorporated)

If after all of this, if we can’t match the individual we are creating to an existing company, we will apply the final logic for company creation and handling:

• IF the CompanyName field is filled in, the Importer will create a new company in the database of BusinessStatus=Active and attach the individual to this company
  • Advanced Use Case: What if the same CompanyName exists multiple times in the import data? i.e., If ZanyZillions is a new company and gets created for the individual in line2, what happens if ZanyZillions is also part of the individual in line6 … will Maxx go in the correct order to not create ZanyZillions again?
  • Answer: MatrixMaxx will do the right thing! The Importer will NOT create a new company twice if the company matches for more than one individual. The individual in row6 will get connected to the company created in row2.
• IF the CompanyName field is NOT filled in, we create the individual in MatrixMaxx not attached to a company (which is exactly how the original 2019 Importer worked)

Rerunning the Import

On DEV, you can retry the import as much as you wish by going into the import link, clicking to reimport, uploading a new file, and running it again.

On LIVE, you can only re-run a production import for the next 24 hours. After that, the re-run function is removed/locked. So, it is imperative that association staff be ready to review/test this import quickly. Why are we doing this? Because the more time passes, the more likely that the data has been manipulated or had a transaction/activity associated with it. At that point, a dump/rerun of the import would cause significant database issues with your MatrixMaxx instance: an orphaned order, a missing meeting attendee, lost data, etc.

Troubleshooting

Time Outs and Slow Imports

In our testing, we haven’t encountered an import that takes more than an hour. If you are trying to do a large import, and it is taking more than an hour, please contact our help/support center for assistance.

Numeric ID issues conflicting with Excel ‘general’ cell formatting

Most IDs in MatrixMaxx are alphanumeric, containing both letters and numbers. Sometimes, however, IDs are numbers only. When this happens, this can cause an issue with the importer. The key is that all cells in the original excel file must be set to TEXT (not GENERAL) number format BEFORE data goes into them. If numeric IDs go into ‘general’ format cells, Excel/sheet will *sometimes* insert a hidden ‘.0’ at the end of the value, thus breaking the import of that row.

[Internal references: P-8148, MAXX-3038]