Dataset Retention - What Datasets to Keep and for How Long?

AGRIS datasets can grow quite large and when a 'live' dataset is copied at every month end, the amount of space needed to maintain all of the dataset copies can exceed the storage capacity of the database server that the datasets reside on.  While it is recommended to copy the 'live' dataset at month end, users may be unfamiliar with guidelines for retaining various datasets.

NOTE: This article deals with various types of datasets and ensuring that multiple copies of the datasets aren’t using more space on the hard drive than necessary. It is recommended that organizations consult a licensed CPA as well as state and federal agencies (IRS and INS) to determine an appropriate data record retention schedule depending on the type, audits may require records from 3 to 7 to 10 years. In fact, some records are required to be permanently maintained.

The following guidelines may be helpful when determining which datasets to keep when cleaning up or removing datasets in order to free up drive space on the database server.  For many organizations, the following datasets should be kept as a minimum:
 

• LIVE DATASET - This is the dataset where daily transactions are recorded.

• PRACTICE DATASET - This is a dataset that users can practice various processes and procedures without affecting the 'live' data.

• ARCHIVE DATASET – Data transferred from a live dataset to an archive dataset then the data is purged from the live dataset so it is a smaller size and generally runs faster.  Always transfer data before purging from the live. Multiple archives will exist when reusing document numbers.

• MONTH END COPIES FROM CURRENT FISCAL YEAR - Primarily used in the event of an audit to show the balances at the end of each month of the current fiscal year.

• PREVIOUS FISCAL YEAR END COPY - Primarily used in the event of an audit to show the balances at the end of the last fiscal year.

Data Reconciliation Tips

• Always transfer data before purging

• Update Live and Achives when doing the following:

  • Name ID Change

  • Combine Name ID

  • Change of item numbers

  • Change of product category on item numbers

  • Change to Location Codes

  • Change Variety/Class Codes

  • Change to Commodity Codes

  • Change/combine ledger codes

  • Change of any Document Number Setup