Skip to main content
OCLC Support

Manage MARC imports

Discover ways to manage the data and the process of MARC imports.

►Data Exchange> Import Batch
►Data Exchange> Import Log
►Data Exchange> MARC Load

Reference data validation

Note this feature is an optional component for data imports.

MARC data is in the ISO-2709 file structure and will need to be processed (or “parsed”) before reference data validation can be carried out.

This processing is done by selecting the file and using the Pre-process/Load Data option. This then operates in the background on the data and, when complete, updates the file status to “Parsed”.

Once the file is parsed, the Validate Ref. Data button can be used. The first time this is done (if the incoming data is not “MARC21”) the data needs to be converted to MARC21 using the MARC21 Mappings Rules. This is necessary so that the process knows which tags and subfields hold which reference data.

At this stage, having mapped a subfield to “949 $l” (for location), the Reference Data Mappings may look something like the below:

Tag, Data Association Domain Matched Value Notes
949 $l ARS Import Batch Locations   1 occurrences
949 $l CPL Import Batch Locations   1 occurrences
949 $l CRA Import Batch Locations   6 occurrences
949 $l CRI Import Batch Locations   16 occurrences
949 $l EST Import Batch Locations   1 occurrences
949 $l GEA Import Batch Locations GEA 6 occurrences; Exact LOCLD
949 $l MCP5 Import Batch Locations System Location 1 occurrences; Longest Match LOC: 1

The above results show that “GEA” matched a Location Long Description exactly and that “MCP5” matched the first character of a Location key value – in this case the matched key value is “MAIN”.

Scope of the mapping – the “Association” column

This applies to both OLSTF and MARC Import Batches. A mapping is picked up if it is assigned to either the Import Batch or the Record Source associated with the Import Batch. In this fashion, a mapping can be made more generic so that it will apply to other Import Batches that use the same record source.

The mapping can be made more generic by either removing the Import Batch number from the mapping record manually on its edit layout, or by checking the box to select several mappings and choosing the Make Generic action. Generic mappings are shown at the end of the above list.

Copy Result action

This action is used as a shortcut to simplify the mapping of several values to a single result.

In order to use this action, select one mapping with the desired result and one or more mappings for the same domain that show no result, for example:

Map multiple values to a single result.

Using the Copy Result action with the selection above will map each of the unmapped tags to GEA.

Handle more than one file on the Import Batch

The process effectively handles one file at a time. However, the mappings can remain present and therefore in effect for all future files processed for this Import Batch and/or Record source.

For MARC data, the parsed information for only one file (or object) at a time can be processed as the parsing process starts by clearing out any previously parsed data.

Reference data domains

A list of the tags and the associated reference data domains are provided below. Tags in bold, by default, match on Key IDs as opposed to descriptions.

MARC Tag Domain
949 $o Collections
949 $M Copy Categories
949 $m Copy Categories
949 $l Locations
949 $I Shelves
949 $i Shelves
949 $S Shelves
949 $s Shelves

Check progress

  • Once the data import batch has completed a confirmation message is displayed
  • During the dataload the status of the batch will be: Loading
  • When the batch load has been completed the status of the batch will change to: Loaded
  • If not, the import process displays: Failed

During the import, if you have SQL*Plus installed on the pc and know how to use it, the following SQL may be run for a Titles import. Re-running on a regular basis will give the status of the import.

select mediatp, count(*) from titles
group by mediatp;

When the batch has loaded, you can see a summary of the load:

  1. In the Import Batch record click the link in Files to Import.
  2. This loads the Object record with the details of the file.
  3. Click Related Items and Control Data to view the number of records Processed and the number of Records Loaded.

Import Log

In the Data Exchange> Import Log domain you can search by the Batch No. This is useful for example to check over when you have loaded a small trial batch. You can clear the records out manually prior to running subsequent batches.

If the import fails, entries are made in this domain to indicate the reasons for the failure. Search for import log records by batch number, review the reasons for failure, rectify the problems and repeat the whole process to re-import the records. It is important to check the import log as even batches which load correctly may have some records that have failed.

 Note: data imports pick up where they left off in the event of failure.

MARC Load

Data Exchange> MARC Load allows you to view MARC data after the import has run.

Click Other Searches to search by

  • Batch Number, Title and Tag
  • Batch Number, Tag, Subfield, Start Title No, End Title No, Field Value

The hitlist is a useful display of how the data was imported.

 Note: there is no details view of the data; all values are displayed in the hitlist.

Import Batch domain

This domain lists all record imports regardless of the type of import. You can search for the batch record by description, e.g. 'w' would retrieve WorldCat record download: automated use only. Searches filtered by Batch Type are also useful.

The Import Batch record shows the details of each batch loaded in Files to Import. You can also make amendments for any future batches as required. The second sheet of the Import Batch layout is the MARC Summary. This is useful in identifying how data is being handled in this batch.

Restart a dataload

If the OLIB database is shut down, then the import will pick up from where it left off automatically. If however, the Normalise Server Programme was stopped (e.g. the application server was rebooted), then the import will stop. This could also happen if the database and application server are on the same machine and it is rebooted, because the database might restart the import before the Normalise Server has started up.

When a batch import is set to run and the server closes down overnight, the import may not automatically restart. The File to Import record (the Object) shows how many records were processed. If there are more records to import, but the import process has stopped, you can restart it by simply re-running the Load Data action for the same Import Batch Object record:

  1. Go to the Import Batch domain and search on the Batch No to display the batch record.
  2. Click the check box on the File to Import and in the Actions box select Load Data.

It is possible that a problem is caused by one specific record, rather than a restart of the server. In this situation you can adjust the value in Records Loaded and rerun the Load Data action for the same Import Batch Object record to skip over the problem record(s):

  1. Go to the Import Batch domain and search on the Batch No to display the batch record.
  2. Click on the File to Import to display the Object details.
  3. Go to the Related Items and Control Data sheet. Note the number of records already processed and modify record.
  4. In order to restart the dataload from the right place, enter the number of records it should start off from again in Records Loaded.
  5. Save and Close the Object record and go back to the Import Batch record.
  6. Click the check box on the File to Import and in the Actions box select Load Data.

The import starts again from the point you specified.

Import replace and update

Existing bibliographic records in the catalogue can be updated or replaced by running a MARC import.

Match Flag
Use this drop down to select the key that you want to match the records on.

Modify Flag
MARC; Add + update all incoming records    
MARC: Add holdings and/or new records    
MARC: Add holdings information only    
MARC: Add matched records to auto-delete folder    
MARC: Add only new records    
MARC: Only load matching records    
MARC: Replace just URLs and/or add new records    
MARC: Replace matching records, retaining URLs and/or add new records

Use this drop down to choose whether you want to replace matching records or update matching records.

Replace or update records:

  1. Use the procedure as normal for a MARC data import.
  2. When creating the Import Batch record choose the relevant field to match records on using Match Flag.
  3. Choose whether to replace or update records using the Modify Flag.
  4. Complete the data import as normal.

View imported and matched records

Before running record imports it is possible to configure an Import Batch record so that when records are imported through it, the imported records are automatically added to a Folder to facilitate any review/amendment processing.

The Import Batch record can also be configured so that matched items are automatically added to a different Folder for viewing.
Procedure

The System Manager should add one or both of these attributes to the Import Batch layout using Layout Manager:

  • New Items Folder (mb_new_folder)
  • Matched Items Folder (mb_match_folder)
  1. Create one or both new Folders in advance for viewing the imported records and the matched records.
  2. These should be created by the same System Manager user who goes on to create the Import Batch record and whose name is present in the Import Batch: Data Entered By field.
  3. When setting up the Import Batch record it should include the relevant Folder name(s) in the New Items Folder drop down and the Matched Items Folder drop down.

When the import is run, OLIB will populate the relevant Folder(s) with imported MARC records.

Matches on Author

During the MARC import, OLIB attempts to match on the author records. If the import process produces too many author duplicate records, you can run the de-duplication process on the relevant records.

The author matching process helps to achieve accurate author matching during imports and to prevent duplicate author records if possible.

During normal usage, you do not normally need to visit or maintain this information. However, it could be useful to ascertain why an imported record has mismatched an existing author. You can then amend the match value to avoid future mismatches.

Author initials generation

During the import, Author initials are not generated if:

  • your OLIB system does not include the initials field on any authors layout
  • the OLIB Defaults> Prevent Generation of Author Initials = Yes

Match values

During imports, OLIB generates and maintains a list of match values against each author. You can view these on the Names layout: Controls sheet - Import Matching.

Author match process

  1. The author matching process first checks to see if ORCid, VIAF or ISNI are present on the incoming record. If one of these match an existing author, any further processing is ignored.
  2. If not present, the import will generate a Match Value from the incoming data and attempt to match this with a Match Value for an existing author.
  3. If no match is found by Match Value, the import will attempt to match on the surname, forename, initials etc... in the author record itself.
  • If a match is found, the process adds Match Values to the author it has matched with
  • If no match is found, the process creates a new author record in OLIB, with the generated Match Value
  1. If this causes a duplicate record, you can edit the Match Values in the Authority Control Details: Import Matching field, for your preferred author record. You can then run de-duplication (the match values will be copied over to ensure that future imports with the same incoming data will match accordingly).

 View a match value

In Import Matching, you can view any existing match values - if there has been a previous import since OLIB 9 Service Pack 12:

  1. Go to Cataloguing> Names and search for the author.
  2. Go to Authority Control Details> Import Matching. For the current author, this will show information such as:

Match Value, Source, First Use, Last Use, Modified By

For any future imports, the Match Value is the value that will be used in preference for matching. The import processes will first attempt to match exactly and then attempt a case insensitive match on just the letters, numbers and spaces in the field.

You can see who the new value was generated by and when. In Source where an OCLC#, Import RCN or Batch number is shown, this match value was generated by an import.

Value generated by existing author record

If the import author matching does not find a successful match for an author, then OLIB matches on a value generated from the existing author record. To view what this would be:

  1. Go to Cataloguing> Names and search for the author.
  2. In Authority Control Details> Import Matching, click New. OLIB generates a Match Value based on the author record.
  3. You can manually edit the Match Value in the list box, or it can be removed if you check the box and click the Delete link below the list. You can also click New to create an additional match value. (The import process will try to match with the first value and if not an exact match, then try the next value).

 Note: for MARC imports, the match value will be generated from all but the Relator Term and Relator Code subfields. The subfield codes other than $a will be included in the match value.

 Note: for OLSTF imports, the match value will be generated as Forenames Initials Surname for personal names, as seen when generated by New above. In addition, there is an OLSTF import setting that permits the dates associated with the author to be ignored as part of the matching. If this is then enabled, the search will be carried out as a secondary match after the generation of the match value with dates, and a match value including dates will be added to the new or existing author record.