Load New 23andMe Data from DNAGedcom

Here we will go through the steps necessary to load the data from the New 23andMe that are downloaded via DNAGedcom.  I highly recommend using DNAGedcom to download 23andMe data because they contain a profile id that makes each relative unique. This avoids loading data into two different people with the same name and duplication of data when someone changes their name on 23andMe.

Set up the Profile Record in GMP

First, click on the profiles tab and add or edit the profile for the person corresponding to the 23andMe data to be loaded.

Capture

On profile line numbered 3, TYPE the name from column 1 or 2 of the 23andMe FIA file downloaded from DNAGedcom.  On profile line numbered 4, TYPE in corresponding Profile Id from column 8 or 9.

Capture

Click the Save Profile button.  Be sure to close the .csv file as it cannot be loaded while open in another app.

Import the New Data Import Templates

First, download the new template files from Facebook.  Then in Options, select Import Templates.  Click on the Import Template button and select one of the downloaded files.

Capture

Next Click Save Template button.

Capture

Repeat for the second downloaded data import template.

Set Import Options

In Options -> App Settings, set the Import Criteria.  This criteria is used to determine what data to load and what information to place in the log file during import.  If you make changes, be sure to click the Save button.

xxx
Options -> App Settings

Import 23andMe Files

To load GMP Relative Records:  At the top of the GMP, select Import Data -> From DNAGedcom -> In GMP select Import Data -> From DNAGedcom -> 1: New 23andMe:  Load RF file.  Select the RF file downloaded with the DNAGedcom Client. Import progress will be shown at the bottom of the GMP page.  When complete an Import Summary window will be displayed.  Relatives without names will show up as blank name errors.

To load GMP Chromosome Browser data:  In GMP select Import Data -> From DNAGedcom -> 2: New 23andMe:  Load FIA file.  Select the FIA file downloaded with the DNAGedcom Client. Import progress will be shown at the bottom of the GMP page. If you used the Gather All FIA button to download data then triangulation data will be included.

Note that numerous errors are reported.  This corresponds to the import criteria set in Options, as a result of bad data or even the wrong import file being selected.

To see more detail, click on the Show Log File button. The corresponding error will be displayed in column 1 followed by the record causing the error.

Click on the Chromosomes Tab and the chromosome browser data will be displayed.  New matches are highlighted in yellow.  If no data is displayed, verify that the check boxes are set as follows.

GMP Chromosomes
GMP Chromosomes

Problems?

  1. Did you TYPE rather than copy the 23andMe name and profile id into the GMP profile record?
  2. Do you have the filters on the Chromosome Browser page checked correctly?
  3. Did you verify that the Import Criteria in Options is set correctly?
  4. Did you examine the log file for clues?
  5. Did you use the correct Import Data template?
  6. Are you on the latest GMP release?

Load FTDNA files from DNAGedcom

Here we will go through the steps necessary to load the Family Tree DNA files that are downloaded via DNAGedcom.  I highly recommend using DNAGedcom to download FTDNA data because they contain an internal kit # that makes each relative unique.  This avoids loading data into two different people with the same name and duplication of data when someone changes their name on FTDNA.

Set up the Profile Record in GMP

First, click on the profiles tab and add or edit the profile for the person corresponding to the FTDNA kit to be loaded.

xxx
New Profile

On profile line numbered 5, TYPE the name in column 1 of the Chromosome Browser file.

Capture
DNAGedcom FTDNA Chromosome Browser File

If you only have one kit, skip this step.  Otherwise, open the ICW file of a second kit and find the name of the profile person then TYPE their kit number on the line numbered 6 in the profile.

Capture2
DNAGedcom FTDNA ICW File

Click the Save Profile button.  Be sure to close the .csv files as they cannot be loaded while open in another app.

Activate the Data Import Templates

In Options, select Import Templates.  Look for the FTDNA templates from Provider DNAGedcom.  Click and activate each one.

Import Data Templates
Import Data Templates

This will add those templates to the Import Data menu list on the menu bar at the top of the page.

Set Import Options

In Options -> App Settings, set the Import Criteria.  This criteria is used to determine what data to load and what information to place in the log file during import.  If you make changes, be sure to click the Save button.

xxx
Options -> App Settings

Import FTDNA Files

To load GMP Relative Records:  At the top of the GMP, select Import Data -> From DNAGedcom -> 1: FTDNA Family finder file.  Select the Family Finder file downloaded from DNAGedcom. Import progress will be shown at the bottom of the GMP page.  When complete an Import Summary window will be displayed.

Capture3
Import Summary Window

To load GMP Chromosome Browser data:  In GMP select Import Data -> From DNAGedcom -> 1: FTDNA Chromosome Browser File.  Select the Chromosome Browser file downloaded from DNAGedcom. Import progress will be shown at the bottom of the GMP page. When complete an Import Summary window will be displayed.

Capture6
Import Summary Window

Note that numerous errors are reported.  This is typical of FTDNA imports and correspond to the import criteria set in Options, as a result of bad data or even the wrong import file being selected.  I hit the ESC key to abort the load and that is reported here as well.

To see more detail, click on the Show Log File button. The corresponding error will be displayed in column 1 followed by the record causing the error.

Import Log File
Import Log File

Click on the Chromosomes Tab and the chromosome browser data will be displayed.  New matches are highlighted in yellow.  If no data is displayed, verify that the check boxes are set as follows.

GMP Chromosomes
GMP Chromosomes

To load ICWs:  In GMP select Import Data -> From DNAGedcom -> 1: FTDNA ICW File. Select the ICW file downloaded from DNAGedcom. Import progress will be shown at the bottom of the GMP page. When complete an Import Summary window will be displayed.

Import Summary Window
Import Summary Window

ICWs are only loaded when both relatives are found in the database.  So it is necessary to load these three files sequentially in the order listed.

Problems?

  1. Did you TYPE rather than copy the FTDNA name and kit number into the GMP profile record?
  2. Is there data in the notes field of the Family Finder file?  Sometimes bad data in the notes or surnames field can cause data to fail to load correctly.
  3. Do you have the filters on the Chromosome Browser page checked correctly?
  4. Did you verify that the Import Criteria in Options is set correctly?
  5. Did you examine the log file for clues?
  6. Did you use the correct Import Data template?
  7. Are you on the latest GMP release?

Using DNArboretum in GMP

In this post, I am going to describe an optional feature that can be used to get surnames and ancestors from FTDNA trees into Genome Mate Pro.

To start, get the DNArboretum Addon for Chrome.  See the Kitty Cooper Blog for details on how to use.

On the FTDNA Family Finder page, select the individual’s tree.

Capture

On the myFamilyTree page, select Ancestor View.

Capture

Click on the DNArboretum icon in the top right corner.

Capture

This will open a new tab in the Chrome browser with the Ahnentafel of the FTDNA individual.  The first time I click on the icon, I get a blank page but on the second time it works:

Capture

Starting with the very first person, highlight the Ahnentafel list and copy it.

Capture

Paste the data into the GMP Relative’s Ancestor list:

Capture

If the very first character in the box is a “1.” then there will be a + in the upper right corner.  Pressing + will reformat the data and add surnames to the Relative’s surname list.  Be sure to press the Save Relative button to retain the data.  Selecting one of the Surnames in Common will scroll the Ancestor List to that name on the list.

Capture

As mentioned above, this is an optional feature.  It is not perfect but can come in handy for those FTDNA relatives that have trees but no surnames listed.

~ Becky ~

 

Adding Additional GedMatch Triangulations

The GedMatch Tier 1 Triangulation data is limited by the closest 500 matches to the profile person.  As such, this data is incomplete and must be supplemented with 1-to-1 comparisons on kits of interest.

A new feature in Release 2016r01 will allow importation of a Relative’s Tier 1 Matching Segment Search data into GMP’s triangulation data, eliminating most of the 1-to-1 comparisons.

On GedMatch, run the Tier 1 Matching Segment Search for a Relative and wait for the results.

Capture

When complete then copy the segments of interest from the GedMatch page.

Capture

On the GMP Relative page, select DNA Comparison and note the + symbol in the Possible Triangulated Segments box.

Capture1
Select the + in the upper right corner to import Relative GedMatch data

Click on the + and a conformation box will appear.

Capture

Copy the data on the GedMatch page and paste into GMP.

Capture

Click Okay to load.  Just the data for Relatives in your database will be loaded.

Capture

As you see, there are now more triangulations.

Capture
After Import

While you may not want to do this for every GedMatch kit, it can be helpful for selected kits.

~ Becky ~

p.s.  This is not on the Import Data menu because it is Relative specific and the Relative record has to be up for it to be loaded correctly.

Performance

I know performance is a point of frustration with Genome Mate Pro (GMP) for some.  I can see a significant difference when loading data into my husband’s computer versus mine.

There is no doubt that GMP is much slower importing data than the old app.  This is because the old app kept all data in memory and saved to disk every 5 minutes. Memory access is much faster than disk access but due to a Silverlight constraint, the in-memory database was limited in size.

The new app is virtually unlimited in size as it keeps the data in an SQLite database on disk.  However, the time it takes to load is a function of disk access speed and this varies from user to user.  I have been told that a solid state drive (SSD) is very efficient.

One way to improve performance without replacing your hardware is to set the import criteria in Options as highlighted below.  This will reduce the number of writes to the log file.

Capture

Starting with the 2016r01 release, it will be possible to enable in-memory journaling.  What this does is to write the SQLite journal file to memory rather than disk.  This is not recommended because there is a risk that if GMP is shutdown abnormally, the database may become corrupted and unusable.  If you do choose to use this option, BACKUP (File -> Backup) first.

CaptureThose are the two things that may help performance.  I have been investigating the utilization of an in-memory database for a future release but am going slow as both XOJO and SQLite are new to me.

~ Becky ~

P.S.  Tested loading data on my husband’s slow computer with these two options and the import time was amazing (197k records in less than 30 mins).

Managing Expectations

Let me start by saying that I developed Genome Mate Pro (GMP) to help me with my autosomal DNA research.  In this post, I want to talk about what to expect from GMP.

GMP is complicated because the subject matter is complicated and because there are so many different sources of data in different formats. If you are having difficulty, try reading Jim Sipe’s Users Guide first.

I developed GMP on a PC with the Windows operating system.  My hardware knowledge is limited and I have no experience with either the Mac or Linux. The app was developed using XOJO and it has compile options for the Mac and Linux but they do not always translate to what I see on the PC.  Fortunately, we have a great group of people on the Facebook Users Group who can fill in the gaps.  If you have problems, ask nicely.

My time is limited and I try to work on the items that will provide the most benefit given my time constraints.

Here is a high level overview of GMP:

Types of Data

There are four main types of data retained in GMP.

1.  Relative data – Surnames, halpogroups, research notes, etc. about a person

2. DNA segments – Chromosome comparisons between profile person and a relative

3. Triangulations – Chromosome comparisons between relatives

4. In Common With (ICW) – Relatives who are related to each other but not necessarily on the same segments that they share with the profile person.  If they triangulate then they will not show up as ICW.

GMP Sections

Capture

Profiles – Information about the DNA kits you manage.  If this is not setup correctly then data may not load.

Chromosomes – Displays DNA Segments that have been imported.  It is profile specific.

Relatives – This section brings together all that is known about a specific relative.

Ancestors – The ancestors associated with each profile are loaded here from a gedcom.

Segment Map – This is a map of all of the DNA segments that have been assigned to a most recent common ancestor (MRCA); marked as a maternal/paternal segment and given a group name.  Note that MRCAs cannot be assigned if ancestors for the profile person have not been loaded.

Options – This is where import data templates are activated and app settings can be changed as well as miscellaneous other functions are available.

Importing Data

Go to Options and activate an existing import data template (or make one of your own).  This will make that template available on GMP’s menu bar under Import Data.  I recommend only activating the ones you use.

Larger data files take a long time to load.  I’ve experimented with some ways to speed it up but the risk of database corruption has made me wary. Regardless, it is wise to Backup (File -> Backup) before imports.

The number of records processed and number of errors encountered are displayed at the bottom of the page.  If the error count seems high the press ESC to abort.

To get more information on errors, click on the Show Log File button in the Import Summary window to see the error associate with each individual line. Click here to see more about Import Data issues.

 In Summary

Be aware that the learning curve is high.  Read the user’s guide and ask questions in the Facebook user’s group.

Data Imports Hints

I have DNA tested at 23andMe, FTDNA and Ancestry plus have uploaded data to GedMatch.  These are my recommendations on what tools to use for data collection and input to Genome Mate Pro.

23andMe

My favorite tool for 23andMe data collection is a Chrome extension called 529andYou because it will collect not only DNA segments associated with each profile but will collection triangulation data as well.  Getting started with it can be very time consuming but well worth the effort in the long run plus it is easy to update as new relatives share.  Please see the GMP User Guide for information on how to use it.

If not 529andYou then to avoid corruption of data with the same relative name by using imports that contain the 23andMe profile id.

  • FIA aggregation file from 23andMe (contains the profile id)
  • RF (Relative Finder) file from DNAGedcom (contains the profile id)
  • AF (Ancestry Finder) file from DNAGedcom (no profile id)

FTDNA – Family Tree DNA

For FTDNA imports, use DNAGedcom downloads as they contain unique keys.

Do NOT mix DNAGedcom imports and FTDNA imports or you may duplicate data in the database.

GedMatch

I recommend subscribing to GedMatch’s Tier 1.  It is a great time saver and well worth the subscription.

  • Tier 1 Matching Segment Search
  • Tier 1 Triangulation (Top 400 matches only)
  • Then use One-to-One compares for segments of interest

Non subscribers can use the chromosome browser feature of the One-to-Many function.  This is quite tedious as each match to be compared has to be checked. See the chrome tool CheckChrome for an assist.

Ancestry

I recommend subscribing and using the DNAGedcom client to download Ancestry data as it is more trouble free than AncestryDNA.

Comments

Large files can take a long time to import into GMP so to break those files into smaller pieces use Free Huge CSV Splitter.

As a final note, I use a PC with Windows 7 so not all of these tools may work on other platforms.

~ Becky ~

Import Issues?

Having issues importing data?  Data not there or rejected?  Formats occasionally change but most of the time the issue can be resolved by following these steps:

  • Verify that the data is not being hidden because of a filter selection on the Chromosome Browser page.  These should be set as follows to view all segments.

Capture

  • Verify that the correct import option was selected.  There are many different import sources and sometimes multiple providers of the same data so ensure the import option reflects both the correct source and provider.

Capture

  • Verify that the data in question is contained in the import file and if using copy/paste that the data is copied from the Chrome browser.

Capture

  • Verify that the keys (name, kit, etc.) in the Profile record are entered correctly with the names and keys exactly as they are listed in the source data file.  Otherwise, the app will not be able to identify the correct profile.

Capture

  • Review the error generated in the log file.  Jim Sipe’s User Guide contains a list of error messages and possible resolutions.  No log file? Verify that the log file is not open in another program such as Excel.

Capture

  • Verify that the settings are correct in Options -> App Settings

Capture


When steps above have been verified and the issue is still not resolved then post the following screen shots on Facebook.  Be sure to include the version of the app and type of computer being utilized.

  1. Snip of the profile page in question
  2. Snip of the error in the log file
  3. Snip of the App Settings in Options
  4. Snip of GMP page (Relative, Chr Browser) showing problem

Please note that in options you can select Privatize the Display for Sharing before taking a screen shot.

If we cannot identify the issue from the screen shots, you may be asked to email your import file to me for further analysis on my test database.

~ Becky ~

Triangulation and ICW in Genome Mate Pro

The purpose of this post is to explain how triangulation and “in common with” (ICW) work in Genome Mate Pro.

Chromosome Browser

From the Chromosome Browser page, right click on a Relative to see what segments triangulate with that person and the profile person.

Capture

The first thing to note is that the app only displays what has been imported. It has no means to automatically identify either of these so if it is missing in the display then look to the import log to see if there is an error message associated with the import.

The second thing to note is that triangulation and ICW data are only retained for relatives found in the database. This is different from the old version of the app and it is important to load relative data before loading chromosome data.

Triangulation

In Genome Mate Pro, triangulation is based on two people being related to the profile person on the same chromosome segment so triangulation data has a chromosome number, base pair start and end points associated with it.

For example, profile person P matches relative A and matches relative B on the same segment (P==A, P==B). If relative A also matches relative B on the same segment (A==B) then P, A and B are triangulated. Otherwise, either A or B is likely related to profile person P on their maternal side while the other is related on their paternal side.

Data is loaded via various chromosome browser files such as the 529andYou and FTDNA Chromosome browser .csv files as well as specific triangulation imports such as GedMatch’s Tier 1 Triangulation.  Triangulations loaded from chromosome browser data have the import limitations listed on the Options page imposed on them as well.

Capture

Some caution should be exercised when a displayed triangulation is based on a very small segment overlap between the Relatives A and B as the endpoints from different sources tend to break at different intervals and endogamy can also cause havoc with the results.

In Common With (ICW)

ICW relatives do not have the chromosome, start point and end point data associated between them (typically from FTDNA) and as a result, the profile person P can have an ancestor in common with both relatives A and B but even if A and B have an ancestor in common, it is not necessarily the same one that is in common with profile person P.

Genome Mate Pro will display the people who are in common with the selected relative on the same segment.

Relative Page

Capture

Triangulation and ICW data are displayed on the Relatives page for the segment selected in the DNA Match Segments table on the left.  Since a Relative that triangulates with the selected person on that segment is already considered ICW, there may not be an ICW displayed in both the Triangulation and the ICW lists.

In summary, triangulation data has a segment specific information while ICW just says two people are related but gives no information on what segment that relationship occurs.

~ Becky ~

Genealogy: The Paper Trail

DNA genealogy research is painstaking, detailed analysis of both DNA matches and the paper trail to confirm shared ancestors.  Excluding adoptees, about 90% of the matches with whom I share DNA have not put together a basic family tree so trying to find a common ancestor can be difficult.  This blog is dedicated to the paper trail.

Getting Started

First, get a genealogy program like Roots Magic to record your findings.  Starting with yourself and moving backwards generation by generation, try to answer these questions:

  • Who was this person’s parents and why do I think so?
  • Who did this person marry and why do I think so?
  • Who were this person’s children and why do I think so?

Exact dates are nice to have but not necessary.  If I don’t have a date for a marriage, I assume they were married before their first child was born.  If I don’t have a birth date, I assume that males were at least age 21 and females age 18 before they married.  While this can vary somewhat, it puts you in the right decade for evaluating the reasonableness of your research.

Source Data

It is important to answer the question “why do I think so?” with references to source material.  The quality of your sources determines the potential accuracy of your research.  It often takes more than one source to adequately establish the answer to one of these questions.  For example, before a marriage takes place, both families are probably residing in the vicinity so not only do you need a marriage record but proof that the family lived in the area at the time.

Family Search (free) and Ancestry (paid) are great resources for data but be aware of the that not all source data is equally reliable.

Sources can be classified as primary, secondary and questionable.  Most people can document the first few generations of their family with primary sources.

Primary sources while they can contain inaccuracies are often the most reliable information.  In a perfect world, we would document our genealogy with primary sources such as:

  • Birth, marriage & death Certificates
  • Census
  • Wills, probates, etc.
  • Land records – deeds, leases, etc.
  • Bible records
  • Pension records

Secondary sources are less accurate and subject to errors of memory or clerical mistakes. Some of these include:

  • Transcriptions, extracts, and abstracts
  • Bible entries predating date of bible
  • Events recorded at a later time
  • Historical narratives
  • Cemetery Markers
  • Diaries & Letters

Questionable sources are those that give you a clue to where to look but should be confirmed with primary & secondary sources.  These include:

  • Family Histories
  • Family Stories
  • Family Genealogies

In the beginning, I made all the mistakes that someone new to genealogy can make and that are too numerous to list here.  Finally, I settled into this simple but adequate approach for establishing a paper trail of my ancestors.

~ Becky ~